Lewandowski has long been linked with a move to the Santiago Bernabeu and his change of agent to Pini Zahavi earlier this year only intensified the rumours.
Though Bayern have insisted that Lewandowski will be staying at the Allianz Arena, former Madrid president Calderon feels a move could be inevitable.
“There is no doubt that Real wants him. Everyone would want Lewandowski in their team,” Calderon told Bild.
“If Robert wants to leave, Bayern does not have a chance. In my opinion, it makes no sense to keep someone bent and broken."
Despite Bayern chairman Karl-Heinz Rummenigge’s insistence that there is no chance Lewandowski will leave this summer, Calderon has seen it all before.
He added: "Of course, he wants Robert to stay. But I've heard that so often during my time as president.
“With Cristiano Ronaldo, Manchester United and Sir Alex Ferguson always said there was no chance. It was the same with David Beckham. But, in the end, the player will always decide on his future.”
Calderon was in charge of Madrid when they broke the transfer record twice for Kaka and Cristiano Ronaldo, and he believes that Lewandowski will be worth whatever his old club have to pay for him given how much has been spent on the likes of Neymar and Philippe Coutinho in recent months.
"When I bought Cristiano for 94 million euros, everyone said that would be completely crazy,” he said. “Meanwhile, there is no limit.
“After the last transfers, it will be difficult to get a top player for under 150 million euros. Robert also belongs to this category. "