Leon Goretzka gave Bayern a half-time lead, but goals after the interval from Leon Bailey, Kevin Volland and substitute Lucas Alario means the champions were unable to capitalise on rivals Dortmund being held to a 1-1 draw at Eintracht Frankfurt.
Speaking to Sky after the match, Kovac was left to lament a VAR offside call going against Robert Lewandowski when the striker thought he had doubled Bayern's advantage during first-half stoppage time.
"The result does not show what was happened on the field," Kovac said.
"We played well in the first half and the decisive scene against us was certainly the offside decision [for] 2-0.
"[If that goal stood] we would not have to lose this match. In my opinion, a draw would have been a fair result. Leverkusen shot four times and scored three goals."
The call against Lewandowski was marginal, while a similarly tight decision fell in Leverkusen's favour when replays of the build-up to Alario's 87th-minute goal were reviewed.
"It's difficult. If we make the 2-0, there's nothing else happening," Kovac added. "Things ran against us with the referee's decisions.
"The only positive is that Dortmund did not score three points, so not much has changed, even though seven points are a lot."
Dortmund had their former boss Peter Bosz to thank after Leverkusen produced a vibrant display.
"I'm proud of this team," he told reporters. "This was a very important win, also for our confidence, but it's only three points.
"We're on a good path, but we still have a long way to go. We still need more points.
"Maybe we didn't play our best match but we still got the win. I think it was a great match for the fans. Both teams wanted to win and played attacking football."