The 19-year-old midfielder joined the Eredivisie club on loan for the 2015-16 season, after signing for Chelsea from Partizan Belgrade.
But his chances for first-team action have been severely limited, having made just 10 appearances in the league, and Pantic finds it hard to explain why.
"I want to come back to Chelsea, just to train. Why am I in Vitesse if I am not good enough?" the youngster lamented to Blic.
"I am crazy! I've made no mistake. If I had a chance and blew it, I would keep quiet.
"Maybe in Vitesse they don't like us Serbs: Nemanja Matic did not get a chance here either and Uros Djurdjevic played 22 matches in a year and a half."
READ MORE | Chelsea needed a bit more rugby - Hiddink
Not only is Pantic unhappy in his loan spell, but he admitted he is starting to think leaving first club Partizan was a mistake.
"There are five of us here at Vitesse from Chelsea and none of us are playing, that means something is wrong," he said.
"For the first time I am saying: 'I shouldn't have left Partizan for Chelsea'. But not yet!
"It is a privilege to be a Chelsea player and I can't expect to instantly play with the likes of Willian and Eden Hazard... It will be hard too for Marko Grujic in Liverpool, with Philippe Coutinho in his position.
"We are still kids, you cannot expect to play in such a big club before you are 23 or 24."