A late Pablo Hernandez strike cancelled out Victor Moses' opener as the Blues dropped points at the Liberty Stadium, also losing ground in the title race as Manchester United leapfrogged them into first place with a win over Arsenal.
Di Matteo has had to field a number of questions surrounding the well-documented Mark Clattenburg saga leaving some to suggest that the events may have had an adverse effect on his players.
However, the former West Brom coach stressed that the investigation was not distracting his team, instead indicating away wins are hard to come by while he pointed out the west London outfit are normally "fair" with officials.
"I wouldn't read anything into what has happened in the last week and the result today," the Switzerland-born boss told reporters. "It is very difficult to win away from home in the Premier League.
|MANCHESTER UNITED LATEST
|11/5||Shakhtar are 11/5 with William Hill to beat Chelsea at Stamford Bridge
"It's a long season and it is going to stay very tight at the top. There are a few teams fighting for that. We are in touch and we are going to show how strong we are in the future.
"Talking about the football is how it's supposed to be week in, week out. Everyone would like it to be this way, including me. I've got to say the officials were very good. Kevin Friend did very well in the game to keep both teams calm and let the game flow.
"He didn't have any influence on the game. I congratulated him after the game because they were very good. I didn't speak to the players before the game about the referee. Generally our team is very good [with officials]. There has to be something really out of the ordinary for them to react. In general, we are a fair team."
The Blues have now failed to win their last two Premier League fixtures, falling to a dramatic and controversial 3-2 loss against the Red Devils as well as the stalemate with Michael Laudrup's men.
That loss, as well as competing in another action-packed tie with Sir Alex Ferguson's men in the Capital One Cup where they edged a win in extra time, is something Di Matteo believes may have hindered his team, revealing his troops are suffering from fatigue.
"We have certainly felt (tired) this week after the two games with Manchester United, including extra-time," he added. "It would have been great to finish off the week with a 1-0 victory. The disappointment is we have conceded a late equaliser.
"I thought we had more of the game and clear chances than Swansea. When the opposition equalises so late on then you tend to think it's a bit lucky. To be honest, they didn't have any real clear chances, but they scored a quality goal."