Forest dealt Wenger his first third-round elimination, with Eric Lichaj scoring twice before penalties from Ben Brereton and Kieran Dowell.
Per Mertesacker had levelled matters at 1-1 and Danny Welbeck's 79th-minute effort to make it 3-2 failed to spark a comeback from the defending champions.
"I believe that overall we played against a Nottingham team who were very sharp, more dangerous than us and dominated us at the front," Wenger told BT Sport. "After that it became a difficult game.
"I think we were a bit unlucky as well but overall they were sharper than us, more incisive and more dominant in the challenges. We lost too many challenges to win a game like that."
1996 - Arsenal have lost an FA Cup third round tie for the first time ever under Arsene Wenger, last falling at this stage in 1996 vs Sheffield United. Dumped.— OptaJoe (@OptaJoe) January 7, 2018
Wenger made nine changes to the side that drew 2-2 with Chelsea in midweek and had very little experience to introduce from the bench.
Asked whether he had misjudged his team selection, the Frenchman said: "That's the inevitable question that comes always in a cup game.
"We have many players on the edge. We have to play on Wednesday again. We have to play Sunday-Wednesday-Saturday and many times we have got away with it. Not today.
"Is it a selection problem? Maybe. Is it an individual performance problem? Maybe as well.
"You never put disappointments behind you but the only thing that can help is to focus on the next game.
"It's true that we won the FA Cup three times in the last four years, so it hurts even more because it's a competition we love and that's why we are so disappointed that we weren't at the level today."
Wenger served the first match of a touchline ban for his treatment of Mike Dean after the 1-1 draw with West Brom last weekend and wisely swerved questions relating to Forest's two penalties, the second of which was allowed to stand following suspicions of a double touch from Dowell.
Arsenal visit Chelsea in the first leg of their EFL Cup semi-final on Wednesday.