Ex-official Clive Wilkes suggested that the action could take place following the recent Premier League game between the Blues and Manchester United, after which the Londoners filed a complaint against Mark Clattenburg for allegedly making a racist comment towards John Obi Mikel.
Wilkes declared that Clattenburg has the backing of all his fellow referees over the issue.
He told The Sun: "I keep in touch with a lot of the lads and there is such a strength of feeling about what is happening to Mark.
"I know a few referees who are even talking about boycotting Chelsea games because of all this.
"It's no exaggeration to say the refs in this country have never felt lower. They are so disillusioned. They feel vulnerable, feel that they are getting no backing.
"They want to speak out themselves but are too frightened, they fear they will be sacked if they go public with their grievances. There have even been murmurings about strike action, but that is very much a last resort."