Lloris miscued a clearance, allowing Toivonen to seal victory in spectacular fashion after the game appeared destined to end in a draw.
Arsenal striker Olivier Giroud had earlier followed up his hat-trick against Paraguay last time out with a wonderful volley to put France in front in Stockholm.
But Jimmy Durmaz equalised for Sweden just before half-time with his first international goal since October 2014.
And the amazing late drama means Sweden move above France at the top of Group A as the race for Russia continues.
In a competitive group, Netherlands are now only three points off top spot courtesy of their win against Luxembourg. Bulgaria are fourth, just four points behind Sweden - who sit at the summit on goal difference from Didier Deschamps' men - despite suffering a surprise loss in Belarus.
A quiet first half-hour saw Dimitri Payet and Mikael Lustig miss the target from half chances at either end, but the match burst into life prior to half-time.
First, Giroud squandered a golden chance by heading wide from inside the six-yard box, Sweden captain Andreas Granqvist doing just enough to put him off after great work from Benjamin Mendy down the left to create the chance.
But the striker quickly made up for that miss, putting Les Bleus ahead in impressive fashion after 37 minutes.
Payet's cross from the right was only half-cleared by Jakob Johansson to the corner of Sweden's box, where Giroud took one touch to set himself up before smashing a volley into the top corner, giving home goalkeeper Robin Olsen no chance.
Sweden hit back two minutes before the break with a fine goal of their own. Ludwig Augustinsson's cross from the left found its way to Durmaz and the winger smashed a first-time left-footed finish past Lloris into the bottom corner.
Antoine Griezmann swooped to head a Moussa Sissoko cross straight at Olsen after the break, before Paul Pogba sent a curling effort narrowly wide from the edge of the box.
France were looking the more likely side to find a winner and Olsen had to repel an ambitious Griezmann free-kick, but Sweden still possessed some threat, with Johansson sending a 30-yard drive just off-target and a dipping Emil Forsberg set-piece forcing Lloris into action.
Deschamps brought on Monaco stars Kylian Mbappe and Thomas Lemar for their competitive debuts in the closing stages, but there were no major chances for either side until a stunning twist in the tail.
Lloris – who was under pressure from substitute Sebastian Larsson - scuffed a dreadful clearance to Toivonen on the halfway line and the Toulouse forward punished him with an extraordinary strike to win it for the hosts.