The 26-year-old former Moroka Swallows captain put pen to paper on a three-year deal with Veselin Jelusic's side.
Baloyi parted ways with Amakhosi at the end of last season after finding game time hard to come by under Steve Komphela over the last two campaigns.
The Limpopo-born player was initially linked with a possible move to Chippa United, but Phunya Sele Sele offered him a deal he couldn't turn down.
Meanwhile, the Free State outfit also announced the signing of Given Mashikinya from Cape Town City.
Mashikinya joins Siwelele as part of the deal that sees Lyle Lakay becomes a City player.
Celtic CEO Khumbulani Konco expressed his delight at finally findiing an amiccable solution to the Lakay matter with City.
"We are glad that the matter between ourselves and City concerning Lakay has been solved harmoniously. Lakay was still contracted to the team and even though he wanted to leave it was prudent for him to follow appropriate procedures," Konco told the Celtic website.
"City approached us in a professional way trying to solve the matter and after deliberations and also consultation with our technical team, we agreed to sign Given and allow Lyle to join them," he concluded.