The Terriers suffered a 2-1 loss in Saturday’s Premier League game at the John Smith's Stadium despite starting the encounter brightly.
After Mathias Jorgensen put Huddersfield in front, the Benin international was sent off for a serious foul play on Yves Bissouma and the Seagulls capitalised on the advantage to rally from behind and claim victory, much to the displeasure of the 47-year-old gaffer.
In his post-match conference, when asked if the decision cost his side defeat Wagner simply responded: "Yes".
The loss saw them drop to the 17th spot after gathering 10 points from nine games, one point above the bottom three.
Huddersfield will hope to bounce back from the setback when they take on Bournemouth on Tuesday at the Vitality Stadium.