Club captain Per Mertesacker celebrated his first Premier League appearance in 532 days with a thumping header to give the visitors a first-half lead.
However, after Alex Iwobi and Mesut Ozil failed to convert glorious opportunities to double Arsenal's lead, Swarbrick judged Hector Bellerin had fouled Richarlison inside the area, enabling substitute Troy Deeney to score an equaliser from the spot in the 71st minute.
Watford went on to grab an injury-time winner too, as Tom Cleverley popped up in the right place to slam home a loose ball and hand Arsenal their first defeat in eight games.
"It was a scandalous decision to give a penalty like that," Wenger told BBC Sport.
"But we have to deal with it. We had many times the chance to score the second goal. When one-on-one with their goalkeeper, we missed them. In the end, we were punished for it.
"The referee was much closer than me [for Watford's penalty], so you trust his decision. Having seen it again on replay, however, it's difficult to understand.
"We missed a few final balls as well, where we should have been better. But as long as you don't score that second goal, any decision can hurt you."
Deeney's impact off the bench made a real difference to the game, with Watford switching from three at the back to a 4-4-2 system following the introduction of their club captain.
Wenger, though, felt Arsenal's attempts to hold on for a point were dealt a blow when Laurent Koscielny - who was playing for the first time since the 2-0 win at West Brom on September 25 - was forced off through injury in the 85th minute.
"We didn't know if Koscielnly would survive for 90 minutes," he added.
"When I looked to change Iwobi with [Jack] Wilshere, Koscielny was not right. In the end, we suffered a bit centrally."