The Championship side saw off their London neighbours at Kingsmeadow courtesy of a fifth-minute header from Murray Wallace.
Wimbledon stunned West Ham with a 4-2 victory in round four, but another shock appeared unlikely once Wallace turned the ball in from six yards out after a fine run and cross from Ryan Leonard.
The attacking flair that caused the Hammers so many problems seemed to desert the home side, who failed to register a shot on target until Michael Folivi's 61st-minute effort was saved by Jordan Archer.
@AFCWimbledon’s Scott Wagstaff promised to dye his beard if they beat @WestHamUtd.— The Emirates FA Cup (@EmiratesFACup) February 16, 2019
He kept to his word and dyed it in the club's colours for today's game against @MillwallFC! pic.twitter.com/NPL0ohNC6d
Aaron Ramsdale kept the advantage limited to one goal with a good save from Leonard, while Lee Gregory headed wide from a promising position.