The north London club have gone through nine managers under the 52-year-old since taking the role in 2001, and the former coach has questioned his knowledge of the game
Levy parted company with head coach Tim Sherwood earlier this week, having only appointed the former midfielder last December following the sacking of Andre Villas-Boas.
|11/4||Tottenham are 11/4 with Paddy Power to finish in the top four next season|
He told BBC Radio 5 Live: "He's a businessman, an incredible businessman. But no, I don't think he is a football man in all honesty. Those decisions have come back time and time again to haunt him.
"It's a very expensive squad and someone has to be answerable for that,
"It's so bitterly disappointing the squad has finished sixth this season and the squad of players in all honesty, are not competitive with the top four teams.
"A decision was made that a new direction had to be found and I just wonder what that direction is."
Southampton manager Mauricio Pochettino is the favourite to replace Sherwood, with Ajax coach Frank de Boer also reported to be a contender.