Wenger has lifted the famous trophy on seven occasions – including three of the last four years – but saw his much-changed side undone by two Eric Lichaj goals along with penalties from Ben Brereton and Kieran Dowell in a classic cup tie at the City Ground.
The likes of Alexis Sanchez, Mesut Ozil and Alexandre Lacazette were all rested as Theo Walcott and Danny Welbeck were afforded rare starts in an unfamiliar Arsenal XI.
But Wenger's plan backfired against a Forest side who are still without a permanent manager following the sacking of Mark Warburton last month.
Lichaj's headed opener was swiftly cancelled out by Per Mertesacker but the American restored the hosts' advantage with a stunning volley just before half-time.
Forest had a cushion when 18-year-old Brereton showed composure beyond his years to beat David Ospina from the spot following a Rob Holding foul on Matty Cash, but a late Welbeck strike threatened to make it a tense last 10 minutes for the home side.
However, Gary Brazil's men were awarded a second penalty when Mathieu Debuchy brought down Armand Traore from behind, allowing Dowell to convert from the spot, and Joe Worrall's red card mattered little as Forest held on for a famous win.
The result represents Arsenal's first third-round defeat since they lost in a replay to Sheffield United eight months prior to Wenger's appointment in 1996.
Wenger's decision to make nine changes appeared ill-judged when Lichaj ghosted into the six-yard box beyond a napping visiting defence to power home a header from Dowell's free-kick after 20 minutes.
But Wenger – serving the first match of a touchline ban following his treatment of referee Mike Dean after last weekend's 1-1 draw with West Brom – had something to celebrate in the stands three minutes later when his side levelled.
Reiss Nelson was awarded a soft free-kick on the left side of the penalty area and, after Holding headed Walcott's delivery onto the left-hand post, Mertesacker was on hand to prod home with the outside of his left boot.
Ospina had touched wide a Cash chance prior to the opener and was forced into two one-handed diving saves by the excellent Brereton before Lichaj produced a moment of magic that left the Colombian helpless.
Lichaj controlled Holding's clearing header on his chest on the edge of the box a minute before half-time and lifted a sublime volley beyond Ospina into the top-right corner.
Arsenal offered little by way of a response early in the second half and Holding made a vital interception to prevent Brereton getting on the end of a dangerous Tyler Walker cross.
But the teenage frontman netted the goal his performance merited in the 64th minute, with Holding rightly penalised by referee Jon Moss.
The previously ineffective Welbeck made the most of a Jordan Smith mistake to side-foot home as Arsenal set up a tense finish, but Debuchy's foul on former Gunner Traore led to the Dowell penalty that wrapped up the win.
Dowell slipped as he struck the ball, but suspicions of a double touch were eventually dismissed by the officials to ratify Forest's two-goal advantage.
And, while Worrall was sent off for a tackle on Chuba Akpom and Walcott came close to pulling another one back, a poor Arsenal were beaten by the better team on the day in a result that will be an unwelcome addition to Wenger's record.