Zinedine Zidane's side saw their 40-match unbeaten run ended by Stevan Jovetic's stoppage-time winner at Ramon Sanchez Pizjuan.
The malaise continued on Wednesday and second-half goals from Iago Aspas and Jonny at the Santiago Bernabeu made it back-to-back losses for the La Liga leaders.
Madrid will have the chance to turn the quarter-final around at Balaidos next week and Casemiro - whose compatriot Marcelo briefly had the hosts level as all the goals arrived during a frantic six-minute spell - conceded they were short of their best on this occasion.
"We didn't play well," the Brazilian told beIN Sports. "We had a good first half but didn't score. There is still a lot to play for. They are dangerous but so are we.
"We were on a very good run. The game at Sevilla hurt us, although we played well.
"Today, we did not play well."
In line with such an assessment, Celta forward Aspas felt his side thoroughly merited the victory after netting his 15th of the season.
"Over 90 minutes we were a better team than Real Madrid," said the former Liverpool man.
"We pressured them well, stopped them bringing the ball out and scored twice through that."
Madrid will seek their first win in four attempts when they host Malaga in LaLiga on Saturday, while in-form Celta – who have claimed five victories from as many matches in 2017 – travel to Real Sociedad the following day.