The current club captain accepted a four-match ban and £220,000 fine after being found guilty by an independent hearing.
And Di Matteo says the ordeal - which started almost a year ago – has put “a cloud over the image of the club”.
“Certainly the image of the club has suffered from this issue. There’s no denying that,” Di Matteo is quoted as saying by The Sun.
“It’s put a bit of a cloud over the image of the club. It’s not for me to judge the whole package. I’m the manager, not a judge of people’s lives.
“I judge on what I see on the training ground and the pitch. That’s what my job has to be about.
“I think he realises he let himself down and the club in that day and moment.
“It’s a shame that we can’t talk about football, the main priority. I understand you have to report current news.
“I’ve got a big week coming up, one of the most important weeks of the season and I don’t get one question about football.”
The club announced on Saturday that Terry will continue as club captain despite accepting the charge, and Di Matteo was quick to underline his support for the defender - highlighting his service to the club during his career.
He continued: “John’s been a great servant over 555 games. He’s made a mistake, let himself and the club down. He’s done a lot of good work as well. I do not regret supporting him.
“I believe the evidence he gave to the magistrates court is exactly what happened on the day. He was cleared in court and we take that as a result of it.
“He realises his language was not appropriate. He realises he let himself, the club and the supporters down and will get punished for that.”