The Bundesliga side took an early lead and, despite an encouraging rally after half-time from the Hammers that saw Joe Cole score, added to their tally in the second half
Sam Allardyce's team had won all four of their pre-season fixtures going into the game but were without Andy Carroll and George McCartney, who were ruled out with heel and knee injuries respectively.
And the Premier League outfit were behind after just nine minutes, when Dani Schahin took advantage of poor West Ham defending to latch on to Nicolai Mueller's delivery before sliding the ball past Jussi Jaaskelainen.
Just 12 minutes later, Mainz were 2-0 to the good, with Mueller again playing a pivotal role as he drove through the heart of West Ham's midfield; with Winston Reid standing off, he guided an effort into the bottom left-hand corner to double his side's lead.
Niki Zimling then hit the post from a free kick with 31 minutes on the clock before Benedikt Saller almost made it three just before half-time.
Two minutes after the break West Ham pulled a goal back through Joe Cole, after Ricardo Vaz Te nodded down a Mark Noble cross for the midfielder to sweep home - and the visitors then set about piling on the pressure.
Modibo Maiga and Cole both went close to equalising before Shawn Parker struck from the spot after 74 minutes for Mainz to kill off any hopes of a comeback.
Four minutes later, Sebastian Polter made it 4-1 with a volley from eight yards out to ensure a deserved win for the hosts.