Giacomo Bonaventura scored either side of Simone Verdi's 23rd-minute goal to guide Milan past Bologna 2-1 in Serie A on Sunday.
It marked a first win at the helm for club great Gattuso, who replaced Vincenzo Montella late last month.
The former defensive midfielder said there was still a lot of work to do for his seventh-placed team as he explained the tension he felt on the bench.
"I'm always tense, even when I play football with my son, but certainly on this bench it's even more exciting," Gattuso told Mediaset Premium.
"Milan is my second skin."
He added: "We still have to improve, especially on the defensive aspect, because Bologna have put us in trouble on more than one occasion.
"But I have to thank the guys for the victory, for the mentality and the sacrifice they made on the field.
"I keep the performance and the result, but we must bear in mind the road is still long."
The final whistle has blown! Back to winning ways at San Siro! That's the way to make a home debut count, boss!— AC Milan (@acmilan) December 10, 2017
Fischio finale! Torniamo alla vittoria a San Siro! Esordio a San Siro con i punti per Mister Gattuso!
11' 76' @giacomobona #MilanBologna 2-1 pic.twitter.com/IwPx7VttS9
Bonaventura's 76th-minute goal was decisive for Milan, who ended a three-game winless run in the league.
Gattuso hailed the Italy international, labelling the 28-year-old one of the best midfielders in Serie A.
"He can play everywhere. He has great quality, he knows how to score," he said.
"I think he is one of the best midfield players in Italy. He still has to improve a little bit from a physical point of view because he had a lot of injuries at the start of the season."