The 27-year-old, then a youngster with Middlesbrough, was at Wembley to see the Italian score within just 42 seconds and ruin the Teesside club's FA Cup hopes
Di Matteo netted after just 42 seconds for the Blues as they ran out 2-0 victors over the Teesside outfit in the Wembley showpiece.
And Downing, then part of the club's youth academy, admitted he still harbours bad memories of the trip to London thanks to the Italian.
"I've hated him ever since!" Downing told the Daily Mail.
"We had got tickets off the club, who took us and our families down on the coach. It was a good day out but with bad memories of the final.
"What did I call him at the time? Nothing I could really repeat!"
The 27-year-old joined the Reds last summer for a fee of £20 million but has so far failed to repay his hefty pricetag, failing to net in 29 Premier League appearances.
But the former Aston Villa man insists that the scrutiny does not get to him and believes it is just a matter of time until he settles.
"Others might take [criticism] hard, but it doesn't bother me," he added.
"It's easy to look and think just because it's not gone well in the league it's [down to] the new players. There are more than four or five players in a squad and we have all not done it, not just the new ones.
'You're expected to come in and set the world alight but sometimes it doesn't happen. You have seen players at other big clubs take time to settle but then become big players. That's how it is for us at the minute.
"You have to get your head down and work hard - that's what I am trying to do. I've shown glimpses but I need to be more consistent.
"Kenny [Dalglish] paid a lot and worked hard to get me here. Every time you play you want to do your best for him."