Robert Lewandowski twice gave Bayern the lead against his old club, but captain Marco Reus hit back on both occasions for Dortmund at Signal Iduna Park on Saturday.
Paco Alcacer came off the bench to score a dramatic winner for Lucien Favre's men, racing on to Axel Witsel's pass to beat Manuel Neuer with a brilliant dinked finish.
Dortmund moved seven points clear of champions Bayern, whose six-game unbeaten run ended at the hands of their most likely rivals for the Bundesliga title.
But despite Bayern twice losing the lead and conceding numerous other clear chances for Dortmund to score, Kovac refused to criticise his team's display too much.
"We saw a fantastic football match from both sides today. It was an open game," Kovac said in quotes reported on the Bayern website. "In the first half we had the game in hand, in the second half Dortmund [did].
"After 1-1 we managed to come back again. Then we conceded twice; that must not happen to us. We closed the midfield very well in the first half and drove the opponent out. In the second, we did not succeed, there were then the chances.
"Unfortunately, we lost the game, even though it could have been a draw. You can be satisfied with the performance. That was the best game since Schalke: playful, combative and tactical, especially in the first half. We will have to build on that and deliver more games in the future."
Favre hailed Dortmund's comeback but accepted his side were fortunate at times, with Bayern having been on top throughout the first half, while Lewandowski saw a stoppage-time equaliser rightly ruled out for offside.
"It was a crazy game. Bayern were very strong in the beginning," Favre said. "They clearly dominated us, had plenty of possession and we could not keep the ball. If we got it, we lost it again.
"That was very hard. But I knew they could not keep up the pace. The pace was very high. 'Gegenpressing', pressing, ball-winning, everything was perfect.
"We could not hold any balls. It was like a boomerang. We had to wait, we waited and luckily it was only 1-0 at half-time.
"After Reus' [second] goal the crowd was there, and we felt we could win, and we did it. The morale was fantastic."