The Professional Footballers Association chairman also criticised Manchester City defender Joleon Lescott, the trio among the most high-profile of players who refused to participate in the campaign but did not give the recently retired centre-back good enough reasons for doing so.
"Rio and Anton Ferdinand were among a group of high-profile abstainers, along with Joleon Lescott, who has not worn one since Newcastle's Turkish midfielder Emre was cleared of racially abusing his then-Everton team-mate Joseph Yobo, and Jason Roberts," Carlisle wrote in an extract from his autobiography, published by The Daily Mail.
|PREMIER LEAGUE LATEST
|4/1||Manchester United are 4/1 with BetVictor to win the Premier League
Reading striker Roberts decided not to wear the T-shirt due to feeling "let down" by the campaign after what he perceived were lenient sanctions handed by the FA to Liverpool' Luis Suarez and Chelsea's John Terry, who were found guilty of racially abusing Patrice Evra and Anton Ferdinand, respectively.
"Jason apart, the Premier League 'big' players happily throw in a grenade but do nothing when it explodes, nothing other than appear to hide behind their advisers," Carlisle continued.
"That's because they are s***houses. They have made their 'statement' for all to see but how many are prepared to stand in front of the cameras and be counted?
"The impression conveyed is that I have failed [as PFA chairman] to get people to unite and support the cause. Not only that, I feel under attack from a group of players questioning my position and representation of them on the issue. This has really hurt me."
Carlisle, who announced his retirement from football in May after a 16-year career, is a firm supporter of the campaign but acknowledges that there is discontent with its level of success.
"Kick It Out is one of few bodies doing something positive about the issue," he argued, "but there are some who evidently feel it is not doing enough."