Diego Simeone's side continued its fine start to the season to draw level on points with Barcelona at the top of La Liga.
Manager Diego Simeone's side dominated the opening 45 minutes, and the visitors had goalkeeper Yoel Rodriguez to thank for saving a Costa penalty on 25 minutes and pulling off a number of other saves to keep the home side at bay early on.
Atleti's dominance continued, though, and on 42 minutes it took the lead when Costa bundled home from close range to put them 1-0 in front at the halftime interval. The striker then doubled his and Los Rojiblancos' lead just past the hour mark with a well-taken finish. Even though Nolito pulled a goal back for Celta with 20 minutes remaining, the hosts held on to pick up a deserved victory.
The win sees Atleti draw level with Barcelona - who beat Valladolid 4-1 on Saturday - at the top of La Liga, with both sides continuing their perfect starts to the season.
The Copa del Rey holders are next in league action when they take on Espanyol on Oct. 19 following the international break.