The German scored the goal that took the tie with Wigan to extra-time before watching his teammates convert all four penalties to seal a place in the final.
The Gunners were strong favorites for the tie with Uwe Rosler's Championship side but fell behind to a Jordi Gomez penalty - conceded following a foul on Callum McManaman by Mertesacker.
The Germany international made amends though, equalizing late in the second half with a deft header to push the game to extra time.
After a goalless additional half an hour, Arsenal converted all four of their spot kicks to book a place in the Wembley final.
And Mertesacker, when asked by ITV if the win was down to the team's mental strength, said: "Yeah I think so because of the second half. First half we were a bit timid, couldn't get through enough, but second half I think we hit the post twice and had a couple of good chances so it was a good comeback.
"I think we have great character. A lot of people have questioned that recently but our comeback today, we proved a lot. Now we want to win it."
Arsene Wenger and his side now await the winner of Hull and Sheffield United, who play on Sunday, for the Wembley final on May 17.