Perea's sending off in the first-half changed the night for Atletico and Toquero took advantage with his first and second goal of the season....
A goal in either half for Gaizka Toquero gave the visitors three points in their search for European football, although they were helped on their way when they were a rewarded with a penalty and a red card for Atletico defender Luis Perea. Fernando Llorente missed from 12 yards but in the end it mattered little.
For large parts, even with ten men, the hosts looked the better side but couldn't find a route past Gorka Iraizoz in the Bilbao goal; star strikers Diego Forlan and Sergio Aguero both missing chances that they would normally convert with ease.
Atletico Madrid began the better of the two teams working the ball around nicely but in-form Javi Martinez and Fernando Llorente both tested David De Gea in the opening minutes. The Manchester United target to replace Edwin Van der Sar was up to the task twice to keep the score level.
Los Leones were beginning to find their feet in the game, looking comfortable on the ball but wasted their opportunities. David Lopez rather lashed at a shot which De Gea watched comfortably and Llorente couldn't quite pick out the run of Iker Munain.
Sergio Aguero was being frustrated by the tight defending of the Basque club and the Argentine's frustrations were doubled by keeper Gorka Iraizoz flying off his line to prevent the striker a clear sight of goal.
A wonderful build-up from Atleti allowed Tomas Ujfalusi time to pick out a cross to the back post but Aguero couldn't keep his header down in space. Jose Antonio Reyes then squandered a good chance, cutting back inside his marker but hitting a tame effort; Diego Forlan was the least impressed standing free in the box.
A clear chance then dropped to Bilbao, a pass from David Lopez was flicked on by Llorente, catching the Atleti defence ball watching, but Gaizka Toquero couldn't force his header beyond De Gea.
The game then turned on it's head in the 39th minute when Llorente was brought to the ground Luis Perea in the box. The referee gave a penalty and showed the Atletico defender a red card as last man. The decision may have been harsh to give Perea his marching orders, but as Lopez and Llorente argued who would step up, Llorente then missed the spot-kick.
The ten men of los Rojiblancos should have taken the lead two minutes before half-time, Diego Forlan finding himself one-on-one with Iraizoz but missed the target.
The deadlock was finally broken in stoppage-time and it was the eleven men of Athletic who had the lead. Andoni Iraola's cross found Toquero in acres of space in the area and he guided his volley past De Gea.
Bilbao could have doubled their lead within three minutes of the re-start. A nice cross from KoiKili from the left found Llorente, who muscled past two defenders, but poked the shot wide of the post when it looked easier to score.
Despite being down to ten, Madrid continued to look just as threatening as their visitors. Reyes was pulling the strings from the right wing, the former Arsenal man testing Iraizoz with a crisply struck shot.
The same combination that unlocked the home side in the first-half did the same again in the second, Iraola this time with a low cross behind Diego Godin and Toquero slid the ball home from the edge of the six-yard box.
The hosts made their changes introducing Juanfran and Diego Costa in search of a goal that would give them hope of a draw but it just wasn't happening for Enrique Sanchez Flores and his men.
The three points for Bilbao lifts them above Atletico Madrid in La Liga into sixth and leaves the capital side questioning their UEFA Champions League potential for another season.