Gonzalo Higuain gave Milan a second-minute lead and Giacomo Bonaventura restored the advantage after Alejandro Gomez's equaliser.
But with Milan seemingly on track for victory at San Siro, substitute Emiliano Rigoni scored in the first minute of added time to earn Atalanta a draw.
Milan have won one of their last four Serie A matches to sit in the bottom half of the table and Gattuso could not contain his frustration after seeing his side drop points again.
"There is great regret, we are a two-faced team," Gattuso told Sky Sport Italia. "We played a great first half, in the last half hour we struggled and we suffered against our opponents.
"It has been happening for four games this time; sometimes we play good football, but that's not enough, we have to do more.
"We have to improve because if you do not close the games it is normal that they can then become a problem. This is the aspect to be improved.
"Another aspect not to be underestimated is that we struggle to change the type of game and take a few second balls. Even in these aspects we must grow.
"Playing well is not enough and is not synonymous with victory. We last too little, we give 30-40 minutes and then leave the opponents to the field."
Milan missed their target of Champions League qualification last season and they are already well off the pace, although they have a game in hand.
"I know that we played well," added Gattuso. "But now I would prefer to play badly and have more points in the standings.
"We are giving good performances, but we need continuity of play and mentality. We cannot suffer and keep the field well for 90 minutes.
"There are no alibis. This is a young team, with room for improvement, but we cannot lose ourselves on the way. It's a fact that what we do is not enough, it's a team problem.
"Today we had a great first half, but the last half hour worries me."