The Brazilian scored a screamer early on but Sebastian Kehl's goal was enough to seal an aggregate win for the Germans despite Rondon's winner on the night
The Russians grabbed the lead through a stunning strike from Hulk early in the first half before Sebastian Kehl equalised before the break.
Rondon restored the visitors' advantage in the second period but Jurgen Klopp's side were able to hold on to a 5-4 aggregate advantage to mark their 200th match in Europe's premier club competition with progression to the last eight.
Sergey Semak, taking charge of his final Zenit match before Andre Villas-Boas is officially installed as coach on Thursday, at least saw his reign close with a victory at Signal Iduna Park, although his side failed to turn around their first-leg deficit.
After five minutes of the first leg, Dortmund were two goals to the good thanks to efforts from Henrikh Mkhitaryan and Marco Reus, and Pierre-Emerick Aubameyang almost got the hosts off to another flying start in Dortmund as he headed Marcel Schmelzer's cross wide.
Axel Witsel had the game's first effort on target shortly afterwards as the Belgian tested Dortmund goalkeeper Roman Weidenfeller with a long-range drive.
Malafeev was then forced into a relatively routine save from Robert Lewandowski before Hulk produced a stunning effort to open the scoring.
Having cut in from the right, the Brazilian hit a wonderful, swerving left-footed strike that gave Weidenfeller no chance.
As the Germans responded, Kevin Grosskreutz saw his effort well saved by Malafeev before Sokratis Papastathopoulos headed over from a corner, but for all their possession and forward momentum Dortmund struggled to break down Zenit's resolute back line.
That was the case until Kehl rose highest to head past Malafeev - breaking his Champions League duck in the process at the 30th attempt.
Hulk threatened a repeat of his first-half effort shortly after the break when he sent a free-kick over the crossbar before Viktor Fayzulin's shot drifted wide of Weidenfeller's post.
Rondon's winner may have spoiled the evening for Dortmund somewhat - the Venezuelan arriving to meet a cross from the left - but it changed little as Jurgen Klopp's men followed Bundesliga rivals and European champions Bayern Munich into the last eight.