The AC Milan man had put Alberto Zaccheroni's men into the lead but two nodded goals in the second half sealed a comeback, inspired by Didier Drogba's substitute arrival
Keisuke Honda put Alberto Zaccheroni's men into the lead with a wonderful finish, controlling the ball in the box with his first touch then hammering into the top corner at the near post with his second touch.
Sabri Lamouchi's team were profligate in the first half but Didier Drogba's arrival in the second half sparked the Elephants into life and quickfire headers midway through the second half turned the scoreline on its head.
Full-back Serge Aurier created both goals with impressive crosses from the right with Bony and Gervinho taking advantage, although the latter benefited from poor goalkeeping from Japan's Eiji Kawashima.
Saturday's game was the first time Cote d'Ivoire have won their opening fixture at a World Cup as the African nation look to claim a maiden berth in the knockout stages in their third consecutive appearance at the global tournament.
The shock in the two starting line-ups was the absence of captain Drogba from Cote d'Ivoire's XI, after he failed to recover from a thigh strain suffered at training on Wednesday.
Bony took Drogba's spot up front, while coach Sabri Lamouchi made two other changes to the XI that began Cote d'Ivoire's 2-1 win over El Salvador with Kolo Toure - who had struggled with malaria before the tournament - and Max Gradel being replaced by Didier Zokora and Yaya Toure.
Japan coach Alberto Zaccheroni recalled Makoto Hasebe after the captain recovered from a knee injury to make just his third appearance for club or country this year.
Hasebe started as one of two holding midfielders alongside Hotaru Yamaguchi, while Yuya Osako got the nod in Japan's most troublesome position: centre forward.
Of the four strikers in Zaccheroni's squad, Osako was the only one yet to have scored for Japan in 2014 but started at the point of their attack ahead of Shinji Okazaki, Yoshito Okubo and Yoichiro Kakitani.
Okazaki started on the right wing, while Okubo and Kakitani were named on the bench.
Cote d'Ivoire had 59.7 per cent possession in the first half with the Africans starting stronger, as Bony received the ball in the box before his shot was deflected past the post in the sixth minute.
But Japan edged their way into the game and in the 16th minute they hit the front.
Yuto Nagatomo collected a throw-in and fed Honda on the edge of the box, and the Milan man slipped away from his marker and fired a thunderous strike into the top corner.
Atsuto Uchida could have made it 2-0 five minutes later but the full-back's powerful drive was parried by Boubacar Barry, before Cote d'Ivoire's goalkeeper was almost chipped by Honda.
While the Elephants had more of the ball, they generally struggled to break down Japan's well-organised defence and their best opportunities came on the counter-attack.
After Yaya Toure and Arthur Boka just missed the target with free kicks, Gervinho got free on the break in the 34th minute only for a scrambling Maya Yoshida to block his shot with a sliding challenge.
Honda slalomed through the opposition defence soon after only to watch his deflected shot loop over the bar, while Cote d'Ivoire had a couple of half chances to Salomon Kalou and Boka but were unable to beat Kawashima.
After a frantic first half, the second started slightly slower but in the 57th minute Cote d'Ivoire had a penalty claim ignored after Yaya Toure's drive into the box ended with a stumble following Yoshida's sliding challenge.
Drogba was brought on in the 62nd minute for Serey Die and almost created a goal immediately with a back-heel to Gervinho before Japan's defence scrambled the ball clear.
But Cote d'Ivoire fans did not need to wait long for an equaliser with Bony getting between Yoshida and Masato Morishige to nod Aurier's fine cross past Kawashima in the 64th minute.
A cross from Aurier set up Cote d'Ivoire's second goal soon after but Kawashima would have been unhappy with his effort, failing to stop Gervinho's header despite getting two hands to the ball at the near post.
Kawashima did better in the 82nd minute, parrying Drogba's free-kick away, while the striker had a shot deflected past the post soon after.
But it mattered little as Cote d'Ivoire moved into second in Group C behind Colombia ahead of their clash on Thursday, while Japan face Greece on the same day.