Kenny Jacket's side booked their place in the fifth round with a deserved win over the Villans, who were poor throughout despite taking an early lead through Darren Bent
Darren Bent gave the visitors an undeserved lead with a scuffed effort from a rebound but Danny Shittu forced the ball home from a corner not long after to get the Lions back on terms.
The tie looked to be heading for a replay before Marquis found space in the box to head home at the second time of asking after his initial effort hit the bar.
Kenny Jacket restored Liam Feeney and Karleigh Osborne to the his starting line-up while John Marquis made his first appearance of the season for the hosts.
Paul Lambert handed Darren Bent a start up front ahead of Christian Benteke, with Ashley Westwood also returning to the side after being cup tied in the Capital One Cup.
The game got off to a scrappy start with neither side able to create a clear goal scoring opportunity. Charles N'Zogbia sliced a shot wide of the post in the opening exchanges before Liam Trotter had an effort easily gathered by Shay Given.
The breakthrough eventually came in the 20th minute when N'Zogbia drove through the Millwall midfield before testing Forde with a low effort. The Lions' keeper could only parry allowing Bent to scuff his shot into an unguarded net.
The hosts reacted well, though, as Keogh forced Given into a good save which led to a succession of corners but the visitors managed to weather the storm.
But Millwall weren't to be denied for long as Shittu rose highest from a corner to force James Henry's cross over the line with the Villa defence half asleep in the 27th minute.
The hosts dominated the rest of the half with Villa remaining a break on the counterattack but neither side were able to fashion a chance before the break.
It was Millwall who started the second period on top and nearly took the lead after the ball ricocheted just past the post from a corner kick.
The Lions continued to pile the pressure on the visitors, who were dropping deeper and deeper in trying to deal with the bombardment of aerial balls.
Villa had a chance on the break when N'Zogbia found Bent over the top but Forde was alert enough to come rushing off his line to smother Bent's shot.
Jackett's side continued pressurise the Villa defence late on but didn't seem to have the guile needed to unlock the visitor's defence for the second time.
But The Den exploded when Marquis rose highest to head Henry's cross on to the bar before nodding the rebound into an empty net.