Michy Batshuayi's equaliser had seemingly done enough to force extra-time against a Watford side who were poor in the second half, but Gray found the net 11 minutes from the end to send the hosts to Wembley.
Watford generally controlled proceedings in the first half and were deserving of their 1-0 lead at the break courtesy of Etienne Capoue, following a goalkeeping error from Vicente Guaita.
Palace rallied after the interval and equalised through Batshuayi, but Gray – who scored the winners in each of Watford's last two home matches – prodded in shortly after his introduction to knock Roy Hodgson's men out of the FA Cup.
After a low-key start, Watford made the breakthrough in the 27th minute as Guaita failed to claim a corner, Craig Cathcart subsequently setting up Capoue to finish into an empty net.
Guaita went some way towards making amends a few moments later, producing a brilliant save to keep Gerard Deulofeu's fierce strike out of the top-right corner.
Palace looked re-energised early in the second half, Max Meyer forcing Heurelho Gomes into a fine save with a header early on, before an Andros Townsend free-kick soon after was tipped over the crossbar.
Batshuayi increasingly became a threat and, a few moments after seeing a stinging drive go just wide, he levelled, charging down Adrian Mariappa's clearance and clinically finding the bottom-right corner.
But Watford had the last laugh, confirming their trip to Wembley as Gray latched on to Roberto Pereyra's inch-perfect pass in behind Palace's defence and poked beyond the helpless Guaita.
What does it mean? Hodgson suffers again
Palace boss Hodgson had lost both of his previous FA Cup quarter-finals as a manager, and he fell short again on Saturday, a semi-final eluding him despite his team's improved display after the break.
As for Watford, they are into the last four for the second time in last four seasons, having lost 2-1 to Palace in the 2015-16 semis.
That feeling when you're going to Wembley!— Watford Football Club (@WatfordFC) March 16, 2019
COME ON!!! pic.twitter.com/7bZkVNTX1Q
Pereyra provides some spark
It may not have been his best game in a Watford shirt, but Pereyra was certainly effective. He provided three key passes – the joint most of anyone on the pitch – and played that fine ball for Gray's winner.
He was neat and tidy on the ball, but Cheikhou Kouyate offered precious little when Palace were not in possession. He made no tackles or interceptions and just a single clearance – not an affective performance.
Key Opta Facts
- Watford have reached the FA Cup semi-finals for the seventh time in their history, although they've eventually progressed to the final just once (1984 under Graham Taylor).
- Against no team have Watford won more FA Cup matches than they have versus Crystal Palace (W4 D2 L2).
- Gray's last three goals for Watford in all competitions have been as a substitute - only Pierre-Emerick Aubameyang and Lucas Perez (4 each) have scored more from the bench among Premier League players this season.
- Three of Capoue's four goals for Watford this season have arrived in cup competitions, including scoring two in the FA Cup.
- Palace striker Batshuayi has scored in all four of his appearances against Watford in all competitions, scoring five goals in the process.