The Red Devils led at the interval but imploded in the second half against Nigel Pearson's men, who came back from two goals down to claim a famous win
Two goals in three minutes from Robin van Persie and Angel di Maria put United in a commanding position early on but Ulloa's header ensured Leicester were still alive at the interval.
Ander Herrera appeared to have settled matters when his flick wrong-footed Kasper Schmeichel to put United 3-1 up, but David Nugent struck from the penalty spot on 62 minutes before summer signing Esteban Cambiasso brought the hosts level moments later with a low drive.
|VIEW FROM KING POWER STADIUM
|By Peter Staunton
As a spectacle, that will take some beating this season. Leicester City, at 3-1 down, looked dead and buried. But they were handed a life-line through the penalty kick earned by Jamie Vardy and never looked back.
Radamel Falcao made his expected United debut, replacing the in-form Juan Mata, with captain Wayne Rooney eased back into a slightly deeper role.
Vardy was one of four changes for Leicester alongside former United trainee Danny Drinkwater, Cambiasso and son of Old Trafford legend Peter Schmeichel, goalkeeper Kasper.
Schmeichel was called into early work after Van Persie was fed by Di Maria, but he could do nothing about the Dutchman's deflected header from a superb Falcao cross which put the visitors ahead on 13 minutes.
The lead was doubled in sublime fashion three minutes later when Di Maria linked up with Rooney before the Argentina star lifted a delightful effort over the helpless keeper.
However, United were almost immediately caught cold at the other end when Vardy got to the byline and crossed for Ulloa to head home powerfully to David de Gea's left.
The visitors suffered another setback when Jonny Evans limped off and was replaced by Chris Smalling with less than half an hour gone, but both defences just about managed to hold firm for the rest of a fast and furious first period.
Falcao went close to restoring United's two-goal cusion at the start of the second half when controlling a Rooney pass and firing a half-volley against the crossbar, but it proved a temporary respite for Leicester.
It was Herrera who did claim the third when flicking a Di Maria strike from the edge of the area past a stranded Schmeichel on 57 minutes.
However, Leicester again showed great heart to rally through a Nugent spot kick after Rafael da Silva shoved Vardy in the box and Cambiasso connected sweetly with a shot from just inside the area to level the scores just after the hour.
As United poured forward in search of a winner, they were caught woefully short in defence.
Mata, on as a substitute, was robbed by Ritchie De Laet, whose throughball released Vardy - the former Fleetwood Town man keeping his nerve to slot past De Gea one-on-one.
A mesmerising comeback was complete just four minutes later, more defensive dithering allowed Vardy in on goal again and Blackett could only chop him down in the penalty area, earning the youngster a straight red card and providing Ulloa the chance to cap a stunning outing for Leicester and a first home win in the top flight since May 2004.