The future of the venue has long been in question with the Hammers initially denied tenancy last October following a legal dispute with top-flight rivals Tottenham and League One side Leyton Orient.
The requirement for a home for British Athletics was the cornerstone to the successful London 2012 bid, but the former Blackburn boss insisted the stadium and surrounding areas will fall into decline without the lure of regular Premier League football.
"If they keep it for athletics, the only time it will ever get a big crowd again after the Olympics is when they hold the World Championships there in five years' time," he told The Times.
"You only have to look around the world at some of the Olympic Stadiums that have been built. They're white elephants now.
"There are weeds growing there. Look at the 'Bird's Nest' in Beijing. Look at Athens."
|HAMMERS' OLYMPIC SPIRIT|
|5/1||West Ham are 5/1 with bet365 to finish in the top half of the Premier League|
The Olympic Stadium was built at a cost of £486 million and with an 80,000 all-seater capacity is the third largest stadium in the country behind Wembley and Twickenham.
The venue was the centre of a glorious fortnight of British sport, which culminated in Sunday night's closing ceremony but in spite of a move being mooted, many Hammers fans are keen to remain at the Boleyn Ground which has been their home since 1904.
"The Olympic Park is a fantastic place," added Allardyce.
"But it can't be left to rack and ruin. If a club with the history and fanbase and potential of West Ham don't go there, the concern would be that the park is left empty the vast majority of the time.
"I can understand some West Ham fans feeling that they don't want to leave the history and tradition of Upton Park, but it's an incredible opportunity for the club to move forward."