Liverpool – who fielded a side showing eight changes from the 1-1 draw with Burnley – conceded twice in the second half at the King Power Stadium to go out of the competition in the third round.
Klopp's men initially cleared a corner but then failed to deal with a return ball into the box, allowing Shinji Okazaki to seize on the scraps and angle a low shot beyond goalkeeper Danny Ward.
Leicester put the tie beyond their opponents with 12 minutes remaining, working the ball in field from a throw in before Islam Slimani unleashed a left-footed effort into the top corner.
"We played really good football in the first half; we should have decided the game early. We had big chances," an irate Klopp told Sky Sports.
"But that's football – it's a 90-minute game.
"It was a second ball after a corner I think, then a throw in. I'm sick of goals like this, to be honest. It's unbelievable.
"You need to stay concentrated to the end, they scored too easy. The first goal, we don't push up together, I can't say it one time or a thousand times, or whatever, it's not nice to concede these goals and today it's again because of this.”
Liverpool had dominated the first half of the all-Premier League tie but struggled for fluency after the break without Philippe Coutinho, who was replaced by Ben Woodburn at half-time.
However, Klopp revealed it had always been the plan for the Brazilian – who only made his first club appearance of the season last week – to be withdrawn at the midway stage.
"Forty-five minutes was the limit. We cannot push him through," the German said.
Klopp will hope for better fortunes when he takes his side back to the same venue in Premier League action on Saturday.