The 32-year-old was allegedly a target of home fans as Pescara suffered a 1-0 away defeat to Cagliari at the Stadio Sant'Elia.
Having had enough, the Black Star walked to the fans to register his displeasure, but was then brandished a yellow card for dissent, after which he angrily walked off the pitch.
“They were chanting against me from the start, then in the first half I saw in the group there were some children and the parents said nothing,” Muntari explained to Sky Sport Italia.
“So I turned to the parents and gave them my jersey, to set the example. In the Curva, the issue continued with another group of fans.
"I talked to the referee. He told me I did not have to talk to the fans. I asked him 'but didn't you hear that?'
"I insisted that he must have the courage to stop the match.
"The referee does not just have to be on the field and whistle; he has to do it all, even feel these things and be an example."
Pescara were left to finish the game with 10 men after Muntari abandoned the game.
“I was trying to reason with the fans, but the referee told me to leave them alone," Muntari continued.
"That’s when I got angry, because rather than stop the game, he decided to punish me?
“The fans were wrong, but the referee had to act differently; not accuse me of causing trouble. I am the victim here.
"If officials actually begin stopping games when this happens, I am convinced it won’t happen again.”
In 2013, compatriot Kevin-Prince Boateng, then on the books of AC Milan, similarly walked off the pitch after being racially abused by fans during a friendly game with lower tier side Pro Patria.