The Nerazzurri's Argentine striker coolly chipped home a spot kick after the Wales winger had opened the scoring for Los Blancos with a stunning long-range strike.
Bale gave Madrid the lead in the ninth minute in sensational fashion, surging through midfield with the ball at his feet before hitting a 30-yard strike, which swerved past Inter keeper Samir Handanovic into the top corner.
But Inter equalized midway through the second half of the International Champions Cup match after Nemanja Vidic was pushed in the area by Ruben Sobrino and Icardi cheekily dinked his spot kick over Jesus Fernandez.
In the shootout, Juan Pablo Carrizo gave Inter an immediate boost, saving Isco's first penalty for Madrid and then denying Asier Illarramendi with a trailing leg, before Icardi ensured victory with the final spot kick.
Madrid started the brightest at California Memorial Stadium in Berkeley with Bale looping a header onto the bar from Lucas Vasquez's third-minute cross.
Vidic - Inter's new central defender - made just his second appearance for the Italian side since joining from Manchester United and the veteran looked rusty in the first half.
The Serbian's backpass in the sixth minute went straight to Bale at the top of the box but the Welshman could not make the most of the opportunity, although Vidic made up for that error with a fine block to deny Lucas soon after.
It did not take much longer for Madrid to hit the front, however, with Illarramendi's pressure allowing Bale to pick up a loose ball near the centre circle, burst forward and score a brilliant goal.
Vidic was lucky to escape sanction just after the half-hour mark, as the 32-year-old collected Lucas on the chin with a slightly-raised elbow but the referee kept his cards in his pocket, as Ancelotti's men saw out the rest of the first half of an increasingly spiteful encounter.
Unrest flared again just after the break when Raul de Tomas of Real cut down Inter's Dodo from behind, sparking plenty of push-and-shove between the two teams.
Madrid seemed to lose their focus as niggly fouls began to permeate the match with Inter substitute Yann M'Vila forcing a fine save before Mazzarri's side equalised in the 68th minute thanks to Icardi's penalty.
Inter could have hit the front with 10 minutes left but Madrid's substitute keeper Fernandez produced two brilliant stops in the space of five seconds to deny Vidic and Marco Andreolli.
But Inter would not be denied in the shootout, winning despite Yuto Nagatomo having his effort saved and M'Vila hitting the post.