After taking the lead at the Britannia Stadium on Saturday through Andy Carroll in the fifth minute, West Ham were undone as Peter Odemwingie's double and one from Marko Arnautovic sealed all three points for the hosts.
It is the fifth time this season that West Ham have fallen to defeat after taking the lead in the Premier League, more than any other side, but Allardyce felt his players deserved at least a point.
Marc Muniesa and Marc Wilson both handled in the area, but the incidents went unpunished by Pawson late on and the West Ham boss was left confused that neither had been spotted.
"I think that in the cold light of day we have to blame ourselves, there is a big part to blame the referee for," Allardyce told reporters.
"Of that there is no doubt in my mind. I have reviewed the situation and watched a blatant handball [from Muniesa] that stops Carroll chesting a ball down and trying to volley it.
"The ball is diverted away by his left arm. It's arm to ball, not ball to arm. The referee is in a fantastic position to see it. I'm totally and utterly baffled."
Referee Pawson was not the only person Allardyce had harsh words for, with his defenders also feeling the wrath of the 59-year-old.
"It's our fault for the poor defending has cost us," he added. "On two occasions we just have to defend correctly.
"We were in the ascendancy, in control looking for a goal to get in front, then we fall asleep and I don't know where our defending went."
Mark Hughes, meanwhile, was thrilled with his side's display and feels the Potters could have won by a bigger margin.
"I was really pleased with what we produced, apart from the first few minutes," he said.
"We talked about it [West Ham's threat from set-pieces] so to concede from the first ball in was disappointing, but apart from that I am pleased with what we did.
"We got back on level terms, which was important, then showed real composure when we broke quickly to take the game away from West Ham. In the end, we had a number of chances and could have won more comprehensively."