The former Juventus defender soared to head home his first West Ham goal in the 121st minute from a Dimitri Payet free-kick, after Lucas Leiva stained an otherwise exemplary performance at centre-back by fouling Enner Valencia.
The visitors' main villain was Christian Benteke, who undermined an industrious showing by wasting a string of chances.
Earlier, Joey O'Brien and Payet struck the post for West Ham either side of Philippe Coutinho doing likewise and Michail Antonio crowned an eventful opening 45 minutes by dispatching a fine opener for the hosts.
Coutinho, starting as one of 10 Liverpool changes after a month out with a hamstring injury, netted a deserved free-kick equaliser three minutes after the restart but West Ham – in their final season at Upton Park – were roared on to dramatically book a fifth-round trip to Blackburn Rovers.
Injury-ravaged striker Daniel Sturridge took his place on the Liverpool bench, with Benteke leading the attack and he twice troubled West Ham at the far post from corners as the quarter-hour approached.
Jurgen Klopp, back on the touchline following a bout of appendicitis, saw Lucas deflect a low strike from right-back O'Brien against the base of the near post in West Ham's first attack of note.
It was Coutinho's turn to hit the post five minutes later from Benteke's cut-back, the chance coming after Winston Reid coughed up possession, and the West Ham defender again lost the Belgium international from a corner following that attack, with Darren Randolph producing the save.
Payet then tested the upright from a 38th-minute free-kick that Simon Mignolet got a faint touch on before making a reaction save from Antonio's follow-up header.
But the former Nottingham Forest man got the better of Mignolet in the final minute of the half – expertly steering home a volley from Valencia's deflected cross.
Liverpool's players were sent out early for the second half by Klopp and they found a swift response, Coutinho cleverly clipping his 18-yard free-kick under the defensive wall after Benteke was bundled over by Pedro Obiang.
As chances arrived with regularity at either end, Klopp grasped the nettle and introduced Sturridge and Divock Origi to lend further weight to his attack, but their defence was looking increasingly flimsy and Mignolet unconvincingly clawed away Antonio's lofted centre.
West Ham's goalscorer skipped past Smith with a sumptuous piece of 66th-minute skill and Valencia was bundled over in the box by Tiago Ilori as he tried to connect with the cross, although referee Roger East was unmoved.
Home boss Slaven Bilic made a trio of enforced changes as Reid appeared to tweak a hamstring and Kouyate suffered a sickening clash of heads with Liverpool defender Jon Flanagan before O'Brien added to West Ham's injury woes.
Valencia blazed a free header over, Randolph parried Jordon Ibe's drilled effort and Andy Carroll – on for Kouyate – clipped a shot wide as eight-minutes of stoppage time could not produce a winner.
Benteke twice should have put Liverpool ahead in the first half of extra time but lashed a volley wide and came off second-best to Randolph in a one-on-one.
Sturridge looked to illuminate a quieter outing by sending a vicious, rising drive just over in the second period, before Benteke erred again and the beleaguered Belgian was left to rue his wastefulness when Ogbonna struck.