The Reds were brilliant in attack, but defensive errors and goalkeeping lapses ensured Klopp and his side had to endure a nervy finish after twice leading by a two-goal margin.
This has been the tale for many of Liverpool's games this season - something which did not escape the attention of their manager.
"It was exactly how it has been so far in the season," Klopp told BT Sport . "There were very, very good moments and then we make it a little bit exciting."
Klopp felt the victory was proof that his side are moving in the right direction.
"I really thought, in a moment like this, so many things are positive," he added to BBC Sport.
"The three points are the proof for the public that we are still here, everything is good. Let's make the next step together."
Leicester manager Craig Shakespeare, meanwhile, struggled to come to terms with a barmy game in which the Foxes consistently threatened.
"I am struggling to digest it myself," he said. "You come in and contribute to another fabulous game of football but ultimately came away with nothing."