The winger capitalised on Tim Krul's poor clearance to secure victory for the Red Devils as Jonas Gutierrez made his return following treatment for cancerAshley Young struck late on as Manchester United maintained their push for the a Champions League place with victory over Newcastle United.
The home side felt they should have had a penalty early on after a challenge by Chris Smalling on Emmanuel Riviere, but referee Anthony Taylor waved their appeals away.
Wayne Rooney wasted the best chance of the first half as he pulled a shot wide from 12 yards before Tim Krul made a fine double save after the break to keep out Marouane Fellaini and Young.
Papiss Cisse wasted a chance to win the game for the home side, before Young took advantage of a poor Krul clearance to fire into an empty net.
Despite being hauled off halfway through United's 2-0 win over Sunderland on Saturday, Angel di Maria retained his place in the starting line-up, with Fellaini replacing the dropped Radamel Falcao as Van Gaal's only change.
And the Argentinian put United into trouble after just eight minutes, as Gabriel Obertan cut out his cross-field pass and fed Riviere, who blazed over from the edge of the area.
The visitors enjoyed much of the early possession, although Newcastle should have been given a penalty in the 10th minute, Smalling escaping unpunished following a blatant trip on Riviere.
Newcastle were then lucky to remain level 16 minutes later, as Young knocked Di Maria's cross into the area for Rooney, who brought the ball down on his chest and stabbed wide from about 12 yards.
With the first half drawing to a close, Johnny Evans and Cisse appeared to spit at each other following a collision, before Krul palmed away a Fellaini header on the stroke of half-time.
Proceedings became considerably more frantic in the second half and both sides saw glorious chances go begging.
Newcastle were the first to go close in the 49th minute and they found United goalkeeper David de Gea at his best, as the Spaniard denied Riviere from point-blank range.
Krul was forced to match his counterpart a few moments later, rushing out to thwart Fellaini and then also blocking Young's effort from the rebound.
John Carver introduced Jonas Gutierrez for Ryan Taylor with 25 minutes left and his entrance initially boosted Newcastle, who almost instantly crafted a fine chance, Cisse inexplicably missing the target after being fed by Ayoze Perez.
And United capitalised on that let-off with just a minute to go, as Krul panicked and played his clearance straight to Young, who coolly slotted into the bottom-left corner to secure a dramatic late win for the visitors.