The former West Bromwich Albion player who joined the Killie until the end of the season in November, broke his duck after six league games.
After a goalless first half, Mulumbu volleyed in a Jordan Jones cross in the 70th minute to hand Steve Clarke’s side a famous victory over the league leaders at the Rugby Park.
The 30-year-old’s strike condemned Celtic to just their second league defeat of the season after Hearts ended their 69-game unbeaten run in December.
69' 1-0 | GOOOOAAAALLLL @mulumbuofficial has his first Killie goal after Jordan Jones' cross from the left.— Kilmarnock FC (@OfficialKillie) February 3, 2018
Killie lead at Rugby Park
Cote d'Ivoire's Kouassi Eboue and Moussa Dembele were on parade for the visitors but could not help the Hoops from defeat.
Kilmarnock are placed sixth in the league table with 29 points from 23 games, with three outstanding games at hand.