The hosts opened the scoring with Muniain but Diego Simeone's men replied twice either side of half-time to secure all three points at San Mames
With Barcelona beating Espanyol earlier on Saturday, Diego Simeone's men needed to win at San Mames to return to the summit, and goals from the Spanish pair duly provided the maximum haul.
Ernesto Valverde's side threatened to end Atleti's run of four straight league victories in just the sixth minute as Iker Muniain punished slack Rojiblancos defending with a neat lob.
However, Costa restored parity with a cool finish for his sixth goal in five games across all competitions as the teams reached half-time at 1-1.
Costa had claims for a penalty turned away early in the second half when he was tripped by Aymeric Laporte, who was later sent off, but it failed to derail the visitors who moved ahead 10 minutes after the break through Koke's close-range header.
The result means Atleti hold a one-point advantage over Barcelona, with seven games left to play, though Real Madrid also remain in the hunt, while Bilbao remain in the last Champions League spot.
The hosts made an electric start with the opening goal in just the sixth minute.
Mikel San Jose's long ball caught the Atleti defence off guard and Muniain reacted quickest before delicately lofting over Thibaut Courtois from the edge of the box.
Atleti went in search of the leveller and Costa almost provided when he sprung the offside trap to collect Koke's intricate throughball, but was foiled by a smart save from Gorka Iraizoz.
Costa was proving difficult for the home defence to handle and he levelled the scores after 22 minutes.
Ander Iturraspe's poor pass was picked off by the Spain striker who easily outpaced San Jose before applying a cool left-foot finish across Iraizoz.
Raul Garcia then wasted a great chance to put Atleti ahead when he shot wide at the back post following good work from Filipe Luis.
Bilbao proved equally wasteful at the other end. Markel Susaeta ghosted between the Atleti defence to meet Artiz Aduriz's flick but headed past the onrushing Courtois and agonisingly wide of the post.
Atleti continued to create the better openings in the second half and Costa dragged an effort wide from the edge of the box shortly after the restart.
The visitors then had claims for a penalty in the 50th minute when Laporte tripped Costa in the area, but referee Jose Teixeira was unmoved.
That decision almost proved more controversial as at the other end Oscar de Marcos robbed Diego Godin of possession on the edge of the box but he fired his effort wide.
Atleti punished that miss in the 55th minute, however, as Koke played in Filipe before he continued his run into the box to head home the deflected cross.
The visitors threatened late on, Gaizka Toquero seeing penalty appeals waved away before Courtois produced a stunning save to deny Aduriz.
There was still time for controversy as Laporte was sent off for a second bookable offence after tripping Costa, though the visitors were unhappy as Cristian Rodriguez was through on goal and could have added a third.
However, it did not cost Atleti, who remain on course for a first title since 1996.