The 34-year-old has criticised the anti-racism group, insisting he will only give his support to an organisation "totally committed to kicking racism out of football"
Premier League players will be asked to wear the shirts prior to this weekend's matches in an effort to raise awareness of the campaign during their annual week of action.
However, following a year in which Liverpool striker Luis Suarez and Chelsea defender John Terry have both been banned by the FA for racially abusing opponents, Roberts has voiced his dissatisfaction with Kick it Out's attempts to combat racism.
"I find it hard to wear a T-shirt after what has happened in the last year. I won't wear one," the 34-year-old told BBC Sport.
"I'm totally committed to kicking racism out of football but when there's a movement I feel represents the issue in the way that speaks for me and my colleagues, then I will happily support it.
"I think people feel let down by what used to be called 'Let's Kick Racism Out of Football'. People don't feel like they have been strong enough."Roberts went on to insist that the sanctions imposed upon Terry for racially abusing QPR centre-back Anton Ferdinand should have been far harsher.
He added: "The four-match ban was, for me, not a heavy enough sanction for what happened."
Kick it Out chairman Lord Ouseley later acknowledged Roberts' concerns and criticised Terry in particular for his failure to apologise for his actions earlier.
"I can understand his frustrations. If Kick It Out had the power to sort this out then we would have done. Sorry Jason, we don't have that kind of power," he said.
"A personal apology last October would have clinched it and saved everybody the pain they have gone through.
"It has been intolerable for Anton Ferdinand and his family, with the hurtful and abusive messages they have been subjected to. I would like to think that a personal apology would be forthcoming."
The 67-year-old also expressed regret over the racism controversy which has tainted English football in the past 12 months.
"We in English football have to get our house back in order," he added.