The 108th-ranked team are competing in the tournament for the first time and were unfancied against the three-time AFCON champions.
But goals from Lalaina Nomenjanahary and Carolus Andriamahitsinoro settled the clash in Alexandria as Madagascar surprisingly finished on top of Group B.
Heading into the fixture on the back of a win against Burundi - the nation's first of any kind since October 2018 - Madagascar battled well and were a threat on the break against a Nigeria side who appeared to have the next stage of the competition on their mind.
Nigeria advance in second place, and this was a wake-up call for them.
The opening goal came almost from nothing, John Ogu's 12th-minute pass back to defender Leon Balogun intercepted with ease.
Balogun called for the ball but hesitated as it approached and as he dithered, he also lost his footing.
That gave Nomenjanahary the time to nip in, steal the ball and show composure to round Nigeria goalkeeper Ikechukwu Ezenwa and tap into an empty net.
Samuel Kalu curled an effort wide as Nigeria sought a quick response but even though they controlled possession, they struggled to create clear-cut opportunities.
And Nigeria were short on luck, too, with Kalu's appeal for a penalty turned down despite it looking like he was clipped as he tried to round Madagascar goalkeeper Adrien Melvin.
Madagascar doubled their lead in the 53rd minute as Andriamahitsinoro's free-kick took a wicked deflection off half-time substitute Wilfred Ndidi.
The deflection wrong-footed Ezenwa before bouncing in off the right post.
And the match fizzled out from there, Nigeria never really getting close to pulling a goal back, with both sides now turning their attention to the round-of-16 stage.