A Thomas Muller double along with a goal for Niklas Sule had handed the home side a 3-1 lead at the Allianz Arena heading into the final quarter-hour, but a subsequent brace from Dodi Lukebakio – including an injury time strike – to complete a remarkable hat-trick rescued a point for the visitors
Despite their two-goal advantage, Bayern threw bodies forward in the closing stages, twice allowing Lukebakio plenty of space to run into and coolly finish past an exposed Manuel Neuer.
Kovac, who took over at the club ahead of the 2018-19 campaign, was less than impressed with his side's defending and acknowledged they deserved to be punished in his post-match press conference.
“You can imagine how I feel internally,” the former Frankfurt boss said.
“If you have the game under control and lead, then I am anything but happy – I am very angry.
“I just cannot understand that we did not realise the position we were in.
“But far worse is the way we defended all three goals conceded. It does not matter if you are playing against a struggling side or Champions League team, these mistakes will be punished.”
The draw, coupled with Borussia Dortmund's 2-1 win over Mainz, means Bayern are now nine points behind the league leaders and extends their winless streak to three games, following a draw with Freiburg and defeat to Lucien Favre’s side before the international break.
Goalkeeper Neuer, who has endured a torrid few months as captain of Bayern and the Germany national team, insists his side's under-performing players will leave no stone unturned in a bid to bridge that gap.
"That should not happen to us," he said.
"We are bitterly disappointed with ourselves, but we have to live with that. We always question ourselves and always want to do our best. We are our harshest critics."
Bayern next host Benfica in the Champions League group stages on November 27 where Kovac’s side can at least secure progression to the knockout rounds with a draw.