The 34-year-old was targeted for having backed his sibling during the racism row which saw Chelsea's John Terry retire from international duty last year.
And, after anti-racism group FARE reported Three Lions supporters to Fifa over the incident, Ferdinand has condemned those involved.
"You expect and accept banter from fans on the terraces as its part of what makes the game great, but racism is not banter and from your own fans? WOW!" he wrote on Twitter.
"Let’s not jump to and assume though as it might just have been banter. We'll see after the investigation.
"Always a small minority who ruin it for others."
The 34-year-old withdrew from the England squad for the recent World Cup qualifying clashes with San Marino and Montenegro due to his need to follow a strict training regime.