The Argentine went onto claim legendary status while playing for Los Blancos during the 1950s and 1960s and is regarded by many as the club's greatest-ever player.
Speaking in an interview with Marca, Di Stefano told of how he was planning to go back to Buenos Aires before he was lured to the Spanish capital.
"I was really fed up of flying everywhere. I had a contract with Millonarios until December 1954, and from the start of 1955, I intended to be at River again," the former striker said.
"Barca approached River first, and Real took advantage of that, jumping into negotiations with Millonarios. I didn't know anything about the club. I only knew about South American football. It was only the second time I'd been to Europe."
Though he ended up in Madrid, the 86-year-old spoke of the few months he spent with the Catalan giants between July and August 1953.
"I hadn't signed anything with them, everything was up in the air," Di Stefano added. "I went there because they told me I had to travel.
"I was there two or three months, but one day they told me to come to Madrid and I came. I didn't mind either way; what I did want was the question to be resolved one way or another and to start playing in one place or the other.
"At first we weren't that good, we were lacking understanding, but we got better and made progress with a lot of enthusiasm."
Before departing for Spain, Di Stefano made a name for himself at Colmbian side Millonarios, where he was known as 'La Saeta', meaning 'the arrow'."They [Millonarios] are one of my three teams, together with Real Madrid and River Plate. I was there during a great period. We had a terrific team and won a lot of things," he concluded.