The Super Falcons struggled to contain the Cameroonians, who were keen to upstage them, with captain Onome Ebi, Osinachi Ohale and Josephine Chukwunonye managing to foil their early threats from attacks led by Gaelle Enganamouit.
The eight-time African champions, who had defeated the Indomitable Lionesses at the last two finals, and were set to defend their title again but an Annette Ngo Ndom-led defence stood in their way, the goalkeeper stopping efforts from Francesca Ordega, Ngozi Okobi and enterprising Asisat Oshoala.
Despite taking the holders little time to make their intent known, the central Africans made threatened through Feudjio Raissa, Ajara Nchout and Onguene Gabriella but could not find a way past the Falcons' rearguard.
15 minutes from half time, Oshoala found space and pull through a brilliant cross, locating Ordega inside the area, who attempted setting up Desire Oparanozie but Cameroon keeper Ndom hurriedly rushed in to make a quick save.
In the 41st minute, Onome smartly launched a long pass into the area from the center line, which came through nicely for Ordega in what looked the best chance of the half for the Falcons, but her volley sailed inches wide.
Two minutes later, Ohale long throw-in fell rightly in front of striker Oparanozie, who set up Oshoala to make another scoring attempt. However, keeper Ndom was in the right position to ensure another fine save to deny the Dalian Quanjian striker to end the first half goalless.
Nigeria came out stronger after the break but Oparanozie's attempt from a setpiece sailed over the crossbar a minute after Okobi's long-range shot was saved by keeper Ndom in the 49th minute.
The Cameroonians had their fair share of scoring moments to savour, but could not match Nigeria's defence, who held out strongly during a tense closing and extra time to force the game to be decided from 12-yard out.
The Lionesses missed two of their penalty kicks with left-back Ngozi Ebere converting the winning kick for the Awcon record winners to stand a chance to claim their ninth title.
Africa Cup of Nations (@CAF_AWCON) November 27, 2018
Thomas Dennerby's side, also through to the France 2019 World Cup, will play the winner between South Africa and Mali while Cameroon battle the loser of the tie for a ticket to the quadrennial showpiece.