Goals from Samuel Umtiti, Antoine Griezmann and Ousmane Dembele earned Didier Deschamps' men a 3-1 victory in Nice on Friday.
But Pogba, who had a quiet game, was whistled by the home fans at the Allianz Riviera.
Tolisso hit out at supporters for targeting the Manchester United midfielder, telling supporters the nation needs to be united.
"The fans came to support the French team. Personally, when you witness this kind of whistling – and I also noticed some for Florian Thauvin – it's exasperating, it annoys me.
"We're France, we are all France players, we're all fighting for our country and for the fans there and we produced a very good performance.
"Paul had a good game, he worked hard. It hurts because I wouldn't want to be whistled, even if I went to Saint-Etienne as an ex-Lyon player.
"Fans need to realise this. We're all on the same boat, and we have to all stick together."
France will go into the World Cup as one of the favourites, facing Australia, Peru and Denmark in Group C.