Carlo Ancelotti claimed Napoli asked for the match at San Siro to be suspended on three separate occasions due to the chanting, but no action was taken.
Koulibaly was sent off after picking up two yellow cards before Lautaro Martinez scored a stoppage-time winner for Inter , and Lorenzo Insigne also saw red late on for Napoli.
After the match the defender took to social media to send a defiant message , saying: "I'm disappointed by the defeat, but above all at leaving my brothers.
"But I am proud of the colour of my skin. Proud to be French, Senegalese, Neapolitan: Man."
And Inter boss Spalletti took the chance to say that change was needed in Italian football, adding there was no place for racism.
"Racist chanting should not be taking place," he told a news conference.
"If 65,000 people come and watch the match on Christmas, they want to see something else. This is the football that Italians and the people around the world want to enjoy.
"People want to find football a pleasant place to be together, this is the change of mentality we have to make.
"This is the right thing to do because our objective is to bring our football up to the top in Europe."
Inter's win saw them move to within five points of second-placed Napoli and the manager was pleased with his team's performance.
"I saw a strong Inter tonight, from the beginning to the end," he said.
"In many ways, we deserved to win, but we could've also lost if Asamoah hadn't made that block.
"This Inter-Napoli was a good match. Both teams were passing the ball with speed and deserved to win."