The Villans secured their third quarter-final spot in four seasons despite losing a 2-0 lead with 20 minutes remaining, as the Belgian's second goal secured progression
Paolo Di Canio's men looked far from out of their depth and fought back from Christian Benteke and Gabriel Agbonlahor strikes with Nicky Storey's double before the Belgian broke their hearts late on.
The home side made four changes from their last game, former Villa man James Collins, who also scored a hat-trick in the win over Stoke City in the previous round, starting for the Robins. Paul Lambert also brought in several new faces, naming Shay Given, Stephen Ireland and Benteke in a strong line-up with Darren Bent on the bench.
The tie started brightly, with Benteke wasting a decent chance for the visitors very early on when he failed to make proper contact with Agbonlahor's cross.
However, it was the League One side who edged the early phases. Though they couldn't conjure up any clear opportunities, Di Canio's men boasted the better share of possession.
Back came Lambert's men, though, as the likes of Andreas Weimann began to settle, but the hosts mustered the first real opportunity of the night, Paul Benson passing Given before dispatching a low shot across goal. Matthew Lowton was on hand to clear it form the line, though.
Di Canio was left rueing that miss just five minutes later when Benteke nodded past a helpless Wes Foderingham. Ireland made the goal with some neat work on the left side before providing a pinpoint cross for the striker.
Though some resistance came courtesy of Matt Ritchie, who fired a number of dangerous balls into the box, Agbonlahor doubled the away side's lead with just over five minutes left in the half - smashing his effort past the keeper after more good play from Weimann and Ireland.
And it could have been even worse when Weimann sent an effort goalwards with the last action of the first period but Foderingham pulled off a superb save to keep the hosts in the tie.
In similar fashion to the first, the Robins started the second period fighting. Benson and Gary Roberts were both guilty of failing to connect with good crosses while Villa, on the other hand, looked off the pace.
As the half continued, though, Swindon's tempo again died down as the visitors successfully disrupted the flow of the game.
However, almost out of nowhere, Roberts had a golden chance to hand the League One side a glimmer of hope, the winger somehow heading wide from another fine Ritchie delivery.
Then, with the game looking all but settled, the hosts struck with around 20 minutes remaining. Storey, on as a substitute just moments earlier, beat Given with a smart finish. before netting a second under five minutes later.
Yet another cross flew into the Villa box and Storey finished the move off with a brilliantly improvised backheel, which sent Di Canio into raptures.
The drama was not over, though, with Lambert having the last laugh. After a poor second half from the Premier League outfit, Benteke salvaged the win very late on after bringing down Eric Lichaj's ball and keeping his cool to beat the keeper.