Watford recovered from Pedro's early goal to go 2-1 up at Stamford Bridge, only for Michy Batshuayi to score twice and Cesar Azpilicueta to add another for the Blues.
The win saw Antonio Conte's men move above the visitors in the Premier League, but Silva felt the three points should have gone to his side.
He told reporters: "It’s a really unfair result. I think we didn’t deserve the result even if we took only one point I think it is not fair to us. It is football.
"We played more in the second half but even how we played first half we started very well. Of course, Chelsea scored a fantastic goal but we reacted well when we got to 1-1. We deserved that goal.
"After the first 25 minutes in the second half we played really good, we controlled the match. We didn’t give chances for Chelsea to counter. We scored the second goal after a fantastic chance for Richarlison and after the goal, [we had] two or three chances and we need to score to kill the match.
"A third goal would have killed the match and given us more chance to control in a different way. When you start to lose big chances to score it’s normal Chelsea with fantastic players start to believe. Then the fresh players in the pitch like Willian, Batshuayi - in one moment they can make the difference. It was a really unfair result."
Silva felt disappointed in the officials for awarding the corner that led to Pedro's opening goal instead of a goal kick.
"Of course everyone knows it was not a corner. Everyone knows it is a goal-kick not a corner," he said.
"Even in the second half I felt after the second goal all the corners and throw-ins were too easy, they fell to the offensive team in every moment.
"That’s football, it’s not fair. I don’t want to lose my time talking about these situations."