The Toffees gave a decent account of themselves in the first half and created enough to go into the break level after Son Heung-min's opener in the 26th minute gave Spurs the lead.
Harry Kane then netted Spurs' second right after the interval and Allardyce believed that to be "the killer", before the England star added another just prior to the hour mark and Christian Eriksen wrapped things up late on.
Although Spurs produced a commanding performance at Wembley in the Premier League clash, Allardyce felt that Everton aided their victory by giving up, suggesting that only goalkeeper Jordan Pickford played to his usual level.
Speaking to BT Sport , Allardyce said: "I think we thought the second goal was offside, I think the same with Wayne's [Rooney, who had a goal disallowed for offside] – just.
"When the second went in, that was the killer, but it doesn't allow us to throw the towel in for me.
"Too many of our players just decided not to defend anymore when Tottenham had the ball.
"But for Jordan Pickford it would have been more. I'm hugely disappointed with the professionalism of my players.
"It was probably what was apparent before I got here and I've got to cut it out, and cut it out starting tomorrow.
"I think we're lacking in our physical abilities. We haven't got a lot of pace in the side.
"Tottenham are capable of scoring a lot of goals against anybody, but certainly they should have had to work a lot harder than they did."
One player who would certainly add pace to the team is Arsenal forward Theo Walcott and Everton are interested in signing him, but Allardyce is awaiting updates from the club
"I don't know at the moment," Allardyce said. "I will have to see tonight or tomorrow [Sunday] what the situation is."
Everton sit ninth in the league and return to action against West Brom next Saturday.