The 46-year-old Spaniard takes on the role at the club where he made 416 appearances on the way to winning five Liga trophies and three Champions Leagues.
Madrid issued a statement on Thursday that announced that the former Bolton Wanderers defender would become a key part of Ancelotti's management team ahead of next season.
"The Real Madrid CF announce that Fernando Hierro has joined the sports structure as first team coach, replacing Zinedine Zidane," the club wrote on their official website.
Zidane, 42, was part of Los Blancos' coaching staff last season, helping guide his side to a tenth Champions League final, their first since he scored the winner in their 2-1 win over Bayer Leverkusen in 2002.
The Frenchman was appointed Real Madrid Castilla boss in June, tasked with reinvigorating the reserve side after they were relegated to Segunda Division B, Spanish football's third tier, last term.