The hosts took the lead through Lazio striker Miroslav Klose's first-half strike, the 68th of his international career, as the veteran equalled Gerd Muller's long-standing goalscoring record for the Mannschaft.
Austria came out for the second half in a determined mood and they almost pulled level only for Toni Kroos to effectively end their hopes just six minutes after half-time, while Thomas Muller added one for good measure in the final moments.
The visitors' last win over Germany came in 1986 and that drought looked like continuing from the start, with the hosts unlucky not to go two goals ahead within the opening 10 minutes as Sami Khedira curled a right-footed shot just wide from the left before Klose hit the side netting from a narrow angle after a Christian Fuchs error.
Muller and Philipp Lahm - who was earning his 100th cap for Germany - were proving a troublesome partnership down the right, but after the 15th minute Low's side were finding it more difficult to break through the Austrian defence as their final ball generally disappointed.
After wasting two more chances just before the half-hour mark, Germany took the lead in the 33rd minute as Muller produced a dangerous cross from the right and Klose jumped in front of Aleksandar Dragovic to poke the ball into the net.
The visitors hardly attacked in the first half but they should have drawn level within one minute of the restart as David Alaba broke forward with purpose before playing a through-ball into the area for Andreas Weimann, but he could only fire over.
Austria's excellent start to the second half was encouraging, but the ruthless Germans were not in a forgiving mood and they doubled their lead in the 51st minute as Kroos hammered a powerful shot past Robert Almer from range after Marco Reus' lay-off.
Muller almost added a third for Low's team soon after, but after reaching Mesut Ozil's inviting delivery to the back post he could only hit his shot straight at Almer.
The hosts looked good for their lead, but Austria were not going to let their opponents run all over them and they nearly pulled one back just before the 70th minute when Martin Harnik volleyed a corner back into the area only for Manuel Neuer to deny Dragovic.
Germany, who looked relatively comfortable throughout, added a third in the final moments of regulation time as Benedikt Howedes squared for Muller to tap in from close range.