Second-half substitute Michu had given the hosts the lead mere minutes after his introduction before Lukas Podolski and Kieran Gibbs looked to have won the tie for the Gunners
After a lacklustre opening 45 minutes, second-half substitute Michu took mere minutes to break the deadlock and give the hosts the lead.
However, two goals in three minutes from Lukas Podolski and Kieran Gibbs turned the tie on its head, only for Graham to lash home from inside the area after Arsenal failed to clear a late corner.
Arsene Wenger made three changes to the side that drew 1-1 with Southampton, with Olivier Giroud, Per Mertesacker and Aaron Ramsey coming in for Podolski, Thomas Vermaelen and Alex Oxlade-Chamberlain respectively.
Michu - who scored two at the Emirates Stadium earlier in the season - had to make do with a place on the Swansea bench after failing to fully overcome a back injury.
The opening 10 minutes saw the hosts more than happy to soak up pressure as Arsenal dominated possession to an extent that they have rarely enjoyed against Swansea in recent league encounters without showing any cutting edge in attack.
But, threatening on the counter, the Welsh side carved open the first real chance when Graham stung the palms of Wojciech Szczesny with a fine half-volley from the edge of the area.
While the hosts were frustrating Arsenal with a solid defensive display, Chico Flores in particular impressing, the Gunners were not doing themselves any favours with aimless runs down the flanks and shots from distance.
And, again, Swansea nearly sprung a surprise when Kyle Bartley, in only his second appearance for the Swans since his summer move from north London, outjumped Mertesacker and guided a header against the bar, with Szczesny beaten.
Arsenal came out in the second half with more dynamism about their attacking play, with Giroud going close three times inside the opening seven minutes, only to be thwarted by Michel Vorm on each occasion.
Loud shouts for a penalty to the visitors then followed when the Frenchman appeared to be felled by Bartley, but replays showed he tripped over his own heels.
As an instant reaction to Arsenal’s rediscovered flair, Michu was introduced by Michael Laudrup and rewarded his manager within seconds by controlling Graham’s flick on, gliding past Mertesacker with ease and holding off Laurent Koscielny to slot home his 15th goal of the season.
Arsenal could, and perhaps should, have restored parity soon after when Ramsey found Giroud seven yards out, but the striker failed to hit the target with a header, before Theo Walcott dragged a shot past the far post with Vorm beaten.
As the match entered the final 10 minutes, Swansea were visibly being pushed further and further back by their opponents, and Arsenal’s own second-half saviour Podolski spun smartly inside the area to convert Koscielny’s flick and level the scores.
And the visitors were not done there. Just two minutes later Gibbs played a neat one-two with Mikel Arteta to dart into the area and smash a close-range volley into the roof of Vorm’s net to turn the tie on its head.
But Swansea, despite Arsenal's control, refused to be beaten. With under four minutes left to play, Arsenal failed to deal with a Swansea corner and allowed Graham to beat Szczesny from a tight angle and give his side a second bite of the apple.