Nikola Kalinic scored his first goals for the Rossoneri either side of Kevin Lasagna's strike for Udinese to get Montella's side back to winning ways in the league after last weekend's 4-1 thumping by Lazio.
It could have been a very different story, though, if Lasagna's effort at 0-0 had not been ruled out for offside by the video assistant referee system.
Montella was critical of his side's reaction to that disallowed goal, and their inability to kill the game off after they had taken the lead.
"I liked the performance, but not so much the slip in concentration after the Udinese goal was disallowed, as that could've been costly," he told Mediaset Premium.
"Overall, it was a step forward in various different ways, including the attitude. We could've finished it off earlier and must always remain concentrated.
"This is an important win, and a few weeks ago we probably wouldn't have won it. We are getting used to playing matches like this."
Montella believes that last weekend's humbling to Lazio could be the best thing to happen to his side this season, and is excited for what the long-term benefits might be.
"I have no regrets over the defeat to Lazio, because I think it helped us," he added.
"It was a slap in the face that did us good. This team has character, but I am convinced it'll improve further."
Udinese boss Luigi Delneri, meanwhile, was left to lament his side's defending for Milan's winner.
Kalinic shrugged off the attentions of his marker to latch onto a Franck Kessie header and prod home from close range to grab his second of the game.
He said: "It was a shame about the second goal we conceded. It came from a situation we could have dealt with.
"We stayed in the game until the end and missed the chance right at the end to draw through [Rijad] Bajic."