The Dutch outfit put in a strong display against the Liga leaders, but saw their Champions League ambitions come to an end following some controversial refereeing decisions
Jose Callejon gave the Blancos the lead early on with a calm finish after being set up by Kaka. Ajax then thought they had equalised twice, but goals from Nicolas Lodeiro and Miralem Sulejmani were wrongly disallowed.
Gonzalo Higuain showed his class as he made it two for the guests in the closing stages of the first half. Callejon then beat the offside trap well into stoppage time to add a third for Madrid, as the home side crashed out on goal difference courtesy of Lyon's incredible 7-1 triumph over Dinamo Zagreb.
The Spaniards enjoyed the majority of possession in the opening stages of the game, and deservedly opened the scoring in the 14th minute. Kaka set up Callejon with a brilliant pass over the defence and the former Espanyol star kept his nerve to slot the ball past the Ajax goalkeeper Kenneth Vermeer from inside the area.
Karim Benzema then came close to doubling Madrid's lead after a good individual action, but the Frenchman's low shot was kept out by Vermeer.
Ajax then took over the initiative and thought they had restored parity in the 31st minute. Antonio Adan failed to hold on to Gregory van der Wiel's shot and Lodeiro was alert to head home the rebound. Nevertheless, the assistant referee waved his flag and the goal was wrongly disallowed.
The match officials got it all wrong a few minutes later when Ajax again found the net, this time via Sulejmani. The goal was again wrongly disallowed as the Serbian was adjudged to be in an offside position when he deflected Vurnon Anita's shot past Adan.
The Eredivisie giants were dealt yet another huge blow in the 41st minute. Higuain escaped the attention of Daley Blind after a superb Benzema pass and the Argentine showed good control before doubling Madrid's lead with a calm finish.
Ajax continued to push forward after the interval and really should have pegged one back via Jan Vertonghen, but the Belgian centre-back headed just wide from six yards after a corner kick from the right.
Minutes later, the Dutch champions again came close to making it 2-1, but Sulejmani's delicately lofted shot came back off the crossbar, before the Madrid defence cleared the ball to safety.
Callejon poured more salt in Ajax's wounds well into stoppage time as he beat the offside trap before rounding Vermeer and doubling his personal tally. As the news of Lyon's triumph filtered through, there was a atmosphere of disbelief inside the stadium.
Real Madrid have won Group D with a record 18 points from six games, while Ajax will have to settle for Europa League action as the section's third-placed side, on goal difference.
The Spaniards resume domestic action on Saturday when they take on Barcelona at the Santiago Bernabeu, while Ajax's next match is the Eredivisie game away against RKC on Sunday.