After seven seasons at Blackburn Rovers, Duff was signed by Chelsea, who paid £17 million to activate the release clause in his contract. Manchester United were also interested in the ex-Republic of Ireland international, but failed to meet Blackburn's required fee.
Duff won two Premier League titles with Chelsea but has since revealed that he had a meeting with United boss Alex Ferguson the season before he moved to Stamford Bridge.
"I met him [Ferguson] a long time ago," Duff told Second Captains. "I was just a nervous wreck, they were my club.
"Obviously before I was at Chelsea, I was at Blackburn and I wanted to go there but it didn't materialise, back then I think my fee was £15m and I ended up going for £17m, it's a lot of money and it didn't quite happen. I never really found out, I just said I want to go, but it didn't materialise. I stayed at Blackburn for another year.
"It's only kind of years later, I meet Roy [Keane] when he joined back up with the Irish team and he was like 'why didn't you come?', I said I wanted to come and then you realise right, well something has gone missing somewhere along the line, but I'm not one to sit at home mulling over things."
Duff turned out multiple times for the away team at Old Trafford, but only realised his love for the club when he featured in Michael Carrick's testimonial game in early June.
The 38-year-old also admitted that the only reason he agreed to play in the match was so his children could watch him play at one of the world's most famous football grounds.
"It’s only kind of when I retired I’ve realised, my God I just love Manchester United," Duff said.
"Driving there with my kids [to Old Trafford for Michael Carrick’s testimonial], the only reason I kind of played in it to be fair my kids never really saw me play in a game in a big stadium, but I brought them over and I see the Man United sign at Old Trafford, and I was like, that’s ingrained in me.
"I used to watch my Man United videos when I was four, five, and it just kind of all came flooding back. It was Carrick’s testimonial but it felt like my day!"