The South Africa international underwent a physical test in Johannesburg after the two clubs had agreed a deal believed to be worth around £800,000 but a problem was discovered in his right foot.
As a result, Celtic have decided not to sign off on the transfer and Coetzee, 20, will return to Cape Town.
"Ajax Cape Town can confirm that Rivaldo Coetzee will return to the club from Johannesburg where the player was undergoing a medical examination ahead of a propsed transfer to Scottish champions Glasgow Celtic," a statement from the South African club reads.
"Following a routine medical examination carried out by the Celtic medical department they discovered an underlying problem with his right foot.
"The clubs were therefore unable to conclude the deal based on the medical information available. The player will return to Ajax Cape Town for further evaluation.
"The injury has not hampered Coetzee to date having not missed out on any competitive action either for his club or the national team.
"Ajax Cape Town can confirm that they were not aware of any condition that would have potentially hampered a transfer to Celtic.
"While disappointed that the deal could not be completed at this point the club are fully committed to giving the player the care and attention required in order to make a full recovery.
"Stuart Baxter, the Bafana Bafana head coach, has been informed as Rivaldo had been called up for the upcoming matches against Cape Verde."
Coetzee became the youngest player to ever represent South Africa when he made his debut at age 17 in October of 2014, though Fagrie Lakay would later break his record.
He has won 23 caps for his country and made more than 90 appearances for his club at the senior level.