The 39-year-old tookover the role at the European finalists in early 2010 before being replaced by Matthias Sammer in summer 2012.
And Nerlinger admits he knew what was coming after losing the Europe's premier competition to Chelsea.
“To be perfectly honest, my end at Bayern was not unexpected and I was not surprised. In the end I knew that moment when we lost the Champions League final against Chelsea, it was over for me,” Nerlinger told Bild.
The decision to remove the retired Germany international from his position came after an unsuccessful season by the German side’s standards.
The Bundesliga outfit lost out not only in the aforementioned Champions League final but also felt defeat to rivals Borussia Dortmund in both the Bundesliga and DFB Pokal.
Nerlinger added: “In the history of Bayern, it was always the case: after a title less year a change in direction always followed.
“Bayern is past for me now. It was a very intense time for me. We must not forget: with the total combination of being a trainer, president and chief executive it was not always easy."