The Reds looked set to ease to victory after Sadio Mane marked his 100th league appearance for the club with his 50th goal five minutes before half-time, but Jurgen Klopp's men missed chance after chance to extend their lead.
And Leicester, having scarcely threatened, struck through James Maddison with 10 minutes remaining as Foxes boss Brendan Rodgers threatened to end the Reds' streak on his first return to Merseyside since his departure in 2015.
However, there was time for late drama as Mane won a penalty having been clipped by Marc Albrighton moments after Mohamed Salah hobbled off with an ankle issue, allowing Milner to net a last-gasp winner.
Liverpool have now triumphed in 17 consecutive top-flight matches - one short of the all-time record - and lead Manchester City by eight points in the table ahead of the champions' Sunday meeting with Wolves.
MASSIVE moment. COOL finish.— Liverpool FC (@LFC) October 5, 2019
JAMES. MILNER. pic.twitter.com/L6uSSmnrMU
Trent Alexander-Arnold was involved in much of Liverpool's best attacking play from right-back and crafted early opportunities for Milner and Roberto Firmino, yet both were squandered.
It was a ball from Milner on the opposite flank that led to the breakthrough, though, as Jonny Evans failed to cut out a pass in behind and was punished ruthlessly by Mane, who might have had a second before the break but shot straight at Kasper Schmeichel.
Salah was denied by Schmeichel shortly after the restart and Evans deflected the ball narrowly wide off Mane, while both Firmino and Andy Robertson went close.
This profligacy set up a tense finish and Maddison looked to have snatched an improbable point as he fired under Adrian after good work from Ayoze Perez.
Salah then departed after a rugged challenge from Hamza Choudhury, but Liverpool did not give up and Albrighton tangled with Mane in the area, with a VAR check siding with referee Chris Kavanagh on the award of a spot-kick, which Milner rifled home.
The full-time whistle was greeted with scenes of jubilation from Klopp and the Liverpool supporters, while Perez appeared to clash with Robertson, prompting a melee that granted a gripping contest a suitably chaotic ending.
What does it mean? Huge win for early leaders
Leicester were the last Premier League opponents to avoid defeat at Anfield, back in January, and this had the feel of a big match from the outset as Liverpool raced out of the blocks.
In truth, the Foxes struggled to match the league leaders for long periods, but Maddison's goal meant the finale was played out amid a cacophony of noise as the Reds were roared on to a vital win.
Mane the man again
This milestone match epitomised everything that makes Mane so key for Liverpool.
A left winger by trade, he turned up in the right-back position to hunt down Ben Chilwell early in the first half but was soon back in his usual berth to seize on Evans' error and slot coolly past Schmeichel. It was no surprise that he should then have a role in the winner.
50 - Sadio Mane has scored his 50th Premier League goal for Liverpool. He's the 10th player to reach the milestone for the club in the competition, more than any other side. Nifty. pic.twitter.com/XvpojW5a9v— OptaJoe (@OptaJoe) October 5, 2019
Defensive errors costly
Leicester might well argue Mane should not have been awarded a penalty, but Albrighton's intervention was messy and cost his side. Evans had early suffered a lapse in concentration that put Liverpool in complete control.
Rodgers' men look like challenging the big boys this season but must learn errors will not go unpunished at this level.
Key Opta Facts
- Milner's penalty was the 34th time Liverpool have scored a 90th-minute winning goal in a Premier League match – at least nine more than any other side.
- Since the start of last season, Leicester have conceded four 90th-minute winning goals in the Premier League, at least twice as many as any other club.
- Leicester boss Rodgers has lost four of his last nine Premier League games at Anfield (W4 D1 L4), as many as his previous 40 such games at the home of former side Liverpool beforehand.
- Maddison's equaliser came from Leicester's first and only shot on target of the match against Liverpool; Maddison has scored in back-to-back Premier League games for the first time since December 2018.
There is an international break before the next round of Premier League fixtures, but Liverpool return to another mammoth clash against misfiring bitter rivals Manchester United. Leicester play Burnley at home.