Branislav Ivanovic appeared to elbow Christian Benteke two minutes before netting the winner in the closely fought 2-1 defeat, with the Villans also denied a late penalty
Branislav Ivanovic's 73rd-minute header denied the visitors a draw at Stamford Bridge after Christian Benteke had brought them level by cancelling out an Antonio Luna own goal with an excellent finish.
However, the Serbian should arguably not have been on the pitch to score having only received a caution for an apparent elbow on Benteke - while Villa were also denied a penalty for a late John Terry handball - and Lambert feels his side were robbed of a deserved share of the spoils.
"It's easy to come on and have sour grapes because you've lost but it's there for everyone to see," Lambert said.
"It's a clear elbow on Christian, it's a sending-off and I think the referee is in the vicinity to do it.
"The penalty is a penalty. Everyone saw that. These are major decisions. You're asked to officiate and you miss two huge moments.
"It was a flying elbow. I saw the penalty, everyone did, how can you miss that magnitude of a decision? It's beyond me."
Despite his frustration, the Scot went on to praise his team's performance, which came just four days after they earned a surprise 3-1 win over Arsenal at the Emirates Stadium.
"I am so proud of my team. I thought they were absolutely outstanding. Petr Cech has kept them in the game," Lambert said.
"There have been some big, big European teams come here over the years but we've given them a right fright, there's no two ways about it.
"The lads are together. They're a really strong group, they are young lads with energy and pace. The character is always there but some of the football? I was really proud of them.
"Now we need to take this form on. We have had two games at Arsenal and Chelsea away. We should have taken more out of them. Then we have Liverpool.
"These are three incredibly hard, tough games but I think everyone who has seen Villa over the last two games will say they're a good side."
Villa face Liverpool in their first Premier League home game of the season on Saturday.