The winger was substituted at half-time of Sunday's game after being hurt by a challenge from Joe Bennett.
Scans on Monday showed the extent of his injury, although City are reluctant to estimate the length of Sane's absence.
"Manchester City can confirm that winger Leroy Sane sustained ankle ligament damage during the FA Cup win over Cardiff," the club said in a statement.
"The Germany international was substituted off at half-time following a challenge from Bluebirds player Joe Bennett.
"Sane has since undergone scans that confirm the 22-year-old is set for a spell on the sidelines - although no timeline has yet been set for his return."
Manager Pep Guardiola said he expected Sane to be "out for a while" and suggested he could be missing for a month or possibly longer.
Bennett was booked for his harsh lunge on the City star near half-time and was sent off with a second yellow card in second-half injury time for a foul on Brahim Diaz.
However, he took to Twitter after the game to apologise for the damage done to Sane.
"Just want to apologise to Leroy Sane for my tackle today," he tweeted. "I tried to stop the counter-attack and completely mistimed my attempt.
"I didn't mean to catch him like I did. I hope it's nothing serious and he recovers quickly."
Sane has been in spectacular form for the Premier League leaders this season, scoring 11 goals in all competitions.