Low's side took a 2-0 lead in the first half, only to see it dismantled by second-half goals from Mattias de Ligt and Memphis Depay before Nico Shulz netted a 90th-minute winner to claim a 3-2 victory.
The result ensured the 2014 World Cup winners got their Euro 2020 Qualifying campaign off to a winning start and gave them some much-needed momentum following a tricky period over the last nine months.
Germany's 2018 World Cup ended in disgrace when a 2-0 defeat to Mexico saw them knocked out in the group stages after they had lost their opening match to Mexico, and they failed to win a game as they finished bottom of their Nations League group behind France and the Dutch.
Low then announced earlier this month he was axing Thomas Muller, Jerome Boateng and Mats Hummels from his future plans, and the Germans were held to a 1-1 draw by Serbia in their first game since that decision last Wednesday having gone behind 11 minutes in.
Speaking after the victory on Sunday evening, Low told RTL: "The first half was great, we had the game under complete control.
"The goal in the second half came at a very bad time. It is a process that our young team have to go through with. With a good attack, we made the decision.
"We have not lost the faith, today we had the gambling luck that we lacked in the past games."
Hoffenheim left-back Schulz was an unlikely hero for Low's team, scoring his second international goal in just his sixth cap.
Schulz said: "That feels very good. The victory is very good for us as a team, I put everything in my right foot and the ball went in.
"At the beginning of the second half we were too defensive. We will get a lot of confidence from this game."
Leroy Sane netted the opening goal for Germany and Serge Gnabry doubled their advantage with a stunning strike from the edge of the box in the 36th minute.
And the Bayern winger was ecstatic with his side in the early exchanges.
He said after the game: "The first half was sensational from us. Then we wobbled a bit and the Dutch came on. Our opponent then was brutal on the trigger. But we held out until the end and then deservedly made the goal."