Ashley Barnes put Burnley into a seventh-minute lead, but the Reds responded through Georginio Wijnaldum's goal in first-half injury time.
Emre Can turned the match on its head with Liverpool's second just after the hour, as the home side scored with their first two shots on target and that proved enough for Klopp's men to close within a point of third-placed Manchester City, who they play at the Etihad Stadium next week.
Liverpool boss Klopp described the win as a "strange" feeling, but was delighted to see his team scrap out a win.
"The result is massive for us," Klopp told Sky Sports. "I think it's the first ugly game we've won. Normally when we're not that good, we lose.
"At the end, I like it because it's these type of games that we haven't won until now. We are not used to it. It feels strange.
"I'm happy about the three points and that's the most important thing."
Wijnaldum also acknowledged an element of luck in the win, but insisted the level of performance is not important at this stage of the season.
"It was difficult in the beginning, especially because they scored a goal in the first couple of minutes," he said.
"We just tried to fight and do everything to win the game. We tried to do it in the first half and we did it better in the second half.
"I must say, we were also a little bit lucky, because previous games that we played like this we lost. In those games we didn't have a fighting spirit, but [this game] was good.
"We tried everything to keep them away from our goal and stop conceding goals. I think we did it well in the second half and it was a good victory.
"At this stage [it does not matter how you play], because we have had a few bad games. If you have a lot of points and you win a lot of games playing good football, you can talk about the football.
"But at this stage of the season, we have to collect points and it doesn't matter how. We just have to win the games."