Having squandered a chance to put the Ligue 1 strugglers ahead from the spot, Thuram later held his nerve to convert the winner from the spot in the 93rd minute.
PSG, the tournament's champions in each of the previous five campaigns, had earlier gone ahead in the 62nd minute, Neymar breaking the deadlock just over a minute after Thuram's initial miss.
But a second Guingamp spot-kick was dispatched by substitute Yeni N'Gbakoto with nine minutes left and Thomas Tuchel's men were then sensationally dumped out by Thuram's decisive strike in the final moments.
The first half of Wednesday's quarter-final provided little indication as to the see-sawing events that followed as Guingamp's deep defence encountered few scares outside of the inventive, looped volley from Kylian Mbappe that shaved the post in the third minute.
Thomas Tuchel turned to Marco Verratti and Edinson Cavani at half-time and his moves almost paid immediate dividends, the latter heading Neymar's cross straight at Karl-Johan Johnsson.
The decisive period in the match appeared to come when Thomas Meunier - making amends for the trip on Ludovic Blas that sent Thuram to the spot - crossed superbly for Neymar to head in and compound the Guingamp forward's miss that had seen him fire over from 12 yards.
But Cavani's lazy connection with Moussa Diaby's low delivery prevented the hosts from going two ahead and PSG were their own enemies once more in the 81st minute, Juan Bernat's tangle with Marcus Coco allowing N'Gbakoto to restore parity by sending Alphonse Areola the wrong way.
Thuram then surprisingly stepped up to complete both an incredible comeback and his own personal redemption, dusting himself off after being fouled by Thilo Kehrer and condemning PSG to just their second competitive loss of the season, Areola getting an arm to his effort but unable to keep it out.
What does it mean? PSG must correct course
A month is time enough for the French champions to restore order before the first leg of their Champions League last-16 tie with a revitalised Manchester United and recent weeks suggest they might need all of it.
PSG drew twice against inferior opposition in December, were not entirely convincing in overcoming minnows Pontivy GSI in the Coupe de France at the weekend and, here, were vulnerable to worrying lapses in concentration that must be eradicated sooner rather than later.
40% - Thilo Kehrer has conceded 4 of PSG' 10 penalties in all comps this season. Howler. pic.twitter.com/muvwYK4d10— OptaJean (@OptaJean) January 9, 2019
Kerbrat comes to the fore
An unexpected result called for an unlikely hero and Christophe Kerbrat played the part for Guingamp.
The 32-year-old was a rock in defence for Jocelyn Gourvennec's side, contributing 10 clearances and five interceptions to help quell PSG's star-studded attack.
Questions over Cavani
He passed up two fine opportunities after appearing off the bench at half-time, the second a miscued finish that should have confirmed PSG's progression to the semi-finals.