The Gunners cruised to a 3-0 win in the first leg in Sweden last week but found themselves 2-0 down at half-time of the return fixture at the Emirates Stadium.
A strike from Sead Kolasinac early in the second half eased some of the nerves in north London, though, and ensured their place in Friday's draw for the last 16.
Ostersunds, who were only founded in October 1996, after Arsene Wenger had been appointed Arsenal manager, were fearless in their approach and sent the visiting fans into raptures when Calum Chambers' own goal and Ken Sema's strike put them 2-0 up in the space of 70 seconds in the first half.
A Gunners team including Henrikh Mkhitaryan, Jack Wilshere and Danny Welbeck offered little by way of a response until the second minute of the second half, when Kolasinac lashed home from close range.
Graham Potter's visitors could not find the two goals they needed to complete a shock turnaround, but the performance will have given Wenger plenty to consider ahead of the EFL Cup final against Manchester City in three days' time.
6 - Arsenal have conceded in each of their last six games at the Emirates in all competitions; their longest run without a clean sheet there since April 2014 (seven consecutive games). Woes. pic.twitter.com/ESv5NmGM97 — OptaJoe (@OptaJoe) February 22, 2018
Arsenal, no doubt with one eye on Sunday's showdown at Wembley, made little effort to add to their aggregate lead during the early stages in front of a sparse home crowd.
Ostersunds sensed the hosts' complacency and Hosam Aiesh stunned the Emirates 22 minutes in, latching onto Saman Ghoddos' long ball in behind Kolasinac and firing across David Ospina, with Chambers deflecting the effort into the net.
Before the visiting fans had even finished celebrating, Sema capitalised on more slack defending from Arsenal, leaving Chambers for dead with a clever drag-back before rifling the ball into the bottom-right corner from the left of the penalty area.
Arsenal responded with a spell of possession but offered very little going forward, with a flicked header from Danny Welbeck, which was comfortably held by goalkeeper Aly Keita, the closest they came to a goal back in the first half.
However, 90 seconds after the break, they eased some of the anxiety around the Emirates. Full-back Ronald Mukiibi swiped at Hector Bellerin's cross and missed, allowing Kolasinac to take a touch before smashing the ball into the net from 12 yards out.
Arsenal began to look a little more relaxed, with Mohamed Elneny and Danny Welbeck each denied by good saves from Keita, but Ostersunds looked a threat until the end and the Gunners will have to improve significantly if they are to claim the first major trophy of the English season this weekend.
Key Opta stats:
- Despite progressing into the next round, Arsenal have now lost each of their last six home games in the knockout rounds of European competition.
- Graham Potter is just the sixth English manager to win at the Emirates in all competitions (after Alan Curbishley, Phil Brown, Harry Redknapp, Michael Appleton and Garry Monk), but the first to do so in European competition.
- Arsenal's defenders have contributed 20 goals in all competitions this season, the most of any Premier League side (Chelsea 16, Man City 10).
- The Gunners have conceded in each of their last six at the Emirates in all competitions; their longest run without a clean sheet there since May 2014 (seven consecutive games).
- Ostersunds are the first Swedish side to win an away European match in England since Osters IF beat Tottenham in the 1995 Intertoto Cup.
- Arsenal have scored nine own goals since Calum Chambers debuted for them in August 2014, while he has been responsible for 33 per cent of them (3 of 9).