The Belgium international was trying to retrieve the ball from 17-year-old Charlie Morgan, who was lying prone on the ball in an apparent attempt to waste time, but referee Chris Foy sent the attacker off for his aggressive efforts.
While Hazard has apologized, Nevin - who played 193 times for Chelsea between 1983-88 - says the actions of the ball boy were "disgraceful", and feels the incident has soured what should have been a celebration for Swansea.
"I am probably going out on a limb here, but I would have done the same as Hazard," he told Goal.com. "I've seen the footage from every angle it is clear he just kicks the ball out from under the ball boy. It is annoying that some of the news broadcasts have shown the worst angle where it looks like he has kicked the ball boy when he clearly has not.
"It is the actions of the ball boy that upsets me the most. His one job is to get the ball back in play as quickly as he can and yet he was just time wasting. That is disgraceful. Given the high-octane situation, it is no wonder that Hazard was going to do everything within reason to get it off him."
Nevin also suggested clubs may have to think twice about the use of ball boys in future, and attempted to qualify his comments by also suggesting Demba Ba should have been cautioned for an apparent dive early in the game.
"The really sad thing is that this could spell the end of ball boys. This is not the first time we've had problems with ball boys this season," he said. "It is a shame that the actions of one or two ball boys could spoil it for the 99 percent of ball boys that behave impeccably. I understand that the ball boy is 17 years old. He isn't even a "boy" for goodness sake.
"People will say that I'm biased towards Chelsea, but that's nonsense. Demba Ba cheated to try and get a penalty last night and should have been booked."
Despite his views on where the blame should lie over the controversial incident, Nevin believes it would be best for all concerned if the footballing world moves on - but fears the FA may punish the winger further.
"I think both clubs and the FA want to draw a line under it now. It is very embarrassing for all concerned," he said. "The player and the ball boy have both apologized so we should move on, but I fear that the authorities will throw the book at Hazard, and that would be unfair.
"It is a shame that this will take away from Swansea's achievement. They have been a delight to watch all season and were the better side last night. However, when the dust settles they will have their day at Wembley and I'm sure they will win overcome Bradford to win their first major trophy."