Dortmund captain Schmelzer has been out of action since December with an ankle injury and wasted no time in making an impact on his comeback as a half-time replacement for Jeremy Toljan, reacting first to Etrit Berisha's fumble to lash home and give the Bundesliga side a 4-3 aggregate victory.
Peter Stoger's side were second best all night and found themselves behind after just 11 minutes.
Goalkeeper Roman Burki, whose season has been plagued by high-profile errors, flapped at a corner and Rafael Toloi stole in at the back post to prod into an empty net.
The hosts squandered a number of golden opportunities to put the tie beyond Dortmund, most notably through Bryan Cristante, Josip Ilicic and Alejandro Gomez.
And their wastefulness was punished seven minutes from time as Schmelzer's timely intervention secured progression to the next round for the visitors.
Atalanta started at a blistering pace and could have been ahead inside four minutes, but Marten De Roon blazed his effort from the edge of the penalty area unceremoniously over the crossbar.
The Dutchman's profligacy was forgotten seven minutes later, though. Toloi took full advantage of Burki's inability to deal with Gomez's corner to prod home at the back post ahead of the despairing Sokratis Papastathopoulos.
Gomez was the provider again midway through the first-half when his clipped ball over the top was met by the unmarked Cristante, who inexplicably headed wide with the goal at his mercy.
A strangely subdued Dortmund finally burst into life after the half-hour mark with Andre Schurrle testing Berisha from distance and Michy Batshuayi seeing a deflected effort creep agonisingly past the post.
The hosts started the second period with the same verve that characterised their first-half display, Remo Freuler dragging a shot wide within a minute of the restart.
Ilicic then drilled beyond the post, before Mattia Caldara saw his goal-bound effort deflected just over.
Dortmund threw on Marco Reus in a bid to inject some life into their stuttering attack, but it was the Serie A side who came closest to scoring next.
Ilicic stormed towards the visitors' backline and curled a left-footed effort past Burki's post, while Gomez spurned a glorious opportunity with 11 minutes remaining.
On-loan Chelsea striker Batshuayi had seen a low drive thwarted by Berisha, but the keeper spilled a shot from Reus and Schmelzer pounced to seal Dortmund's place in the last 16.