Baxter said he was in Sweden doing his assignments on African teams that participated at the Fifa World Cup finals when he started hearing reports that Chiefs wanted him back.
However, he admitted that there was 'life' in the rumours, although Chiefs were not bothering him every single day.
"Yes‚ I'm sure there was life in it," Baxter told the media.
“I was bombarded with reports about how interested Chiefs to bring me there. Now, I am not blowing my own trumpet when I say that I would imagine everytime Chiefs need a coach my name will come up there somewhere rightly or wrongly, “ said Baxter.
"And I'm sure‚ because you get to know through different sources that there was a scenario where people were hoping that that would happen. Does that mean that I was sitting down in negotiations with Chiefs? No, it doesn't, and it doesn't mean that they were disturbing me every day, pestering me," added the former England U21 manager.
Chiefs enjoyed huge success with Baxter at the helm between 2012 and 2015, and their trophy drought in the last three seasons put the management under immense pressure to find the right man for the coaching job.
"But you guys like a good story. So, there was a bit of life in that one. So, you flogged it a little bit, and I'm sure that Safa were not sure how they would deal with it, but we were in contact and I was doing my job," he explained.
"I was reporting on things like the African teams and how they were doing at the World Cup. So‚ yes I'm sure there was life in it. But was it an ongoing saga as we saw in the press? No, it wasn't," added the Bafana mentor.
"So, I think very quickly we could understand that no matter what ambitions other people might have, but sometimes it doesn't work," concluded Baxter.