Marko Arnautovic's early goal proved decisive for the Hammers as the forward netted his first for the club at the end of a fine move inside the opening six minutes.
After that early breakthrough David Moyes' side were content to sit back and soak up the pressure, with Chelsea toiling fruitlessly at London Stadium.
And the lack of tempo and cutting edge to the visitors' play left Cahill deflated.
"We were miles off [our best] from the start," the defender told Sky Sports .
"They got the early goal, which was a poor one on our behalf. We still had plenty time, but that laboured start carried on throughout the game.
"We didn't play our stuff. They got the goal, sat behind the ball and tried to hit on the break.
"It was a bit frustrating with their time wasting, which was clever on their part, but we were just not at the races. It was very disappointing."
Cahill was not about to concede defeat in the title race, though he did suggest that Sunday's Manchester derby could be crucial to that, with Manchester City able to go 14 points ahead of them if they beat United at Old Trafford.
"We have to wait and see," added Cahill. "It was a huge game for us and we needed to win.
"We have to sit back and wait for the results tomorrow. We have to freshen up and get the energy levels back up. We've been in good form recently which is why it's so disappointing."