Di Matteo netted after just 42 seconds for the Blues as they ran out 2-0 victors over the Teesside outfit in the Wembley showpiece.
And Downing, then part of the club's youth academy, admitted he still harbours bad memories of the trip to London thanks to the Italian.
"I've hated him ever since!" Downing told the Daily Mail.
"We had got tickets off the club, who took us and our families down on the coach. It was a good day out but with bad memories of the final.
"What did I call him at the time? Nothing I could really repeat!"
|DOWNING TO DELIVER?
|10/1||Stewart Downing is 10/1 to score anytime in the FA Cup final with Paddy Power|
But the former Aston Villa man insists that the scrutiny does not get to him and believes it is just a matter of time until he settles.
"Others might take [criticism] hard, but it doesn't bother me," he added.
"It's easy to look and think just because it's not gone well in the league it's [down to] the new players. There are more than four or five players in a squad and we have all not done it, not just the new ones.
'You're expected to come in and set the world alight but sometimes it doesn't happen. You have seen players at other big clubs take time to settle but then become big players. That's how it is for us at the minute.
"You have to get your head down and work hard - that's what I am trying to do. I've shown glimpses but I need to be more consistent.
'Kenny [Dalglish] paid a lot and worked hard to get me here. Every time you play you want to do your best for him."