The 46-year-old Spaniard takes on the role at the club where he made 416 appearances on the way to winning five Liga trophies and three Champions Leagues.
Madrid issued a statement on Thursday that announced that the Spaniard would become a key part of Ancelotti's management team ahead of next season.
"The Real Madrid CF announce that Fernando Hierro has joined the sports structure as first team coach, replacing Zinedine Zidane," the club wrote on its official website.
Zidane, 42, was a part of Los Blancos' coaching staff last season, helping guide the side to a 10th Champions League final, and its first since he scored the winner in a 2-1 win over Bayer Leverkusen in 2002.
The Frenchman was appointed Real Madrid Castilla boss in June, tasked with reinvigorating the reserve side after it was relegated to Segunda Division B, Spanish soccer's third tier, last season.