A second half Jan Vertonghen goal was enough to secure the three points against the Swans and although Tottenham now find themselves in pole position to finish in the top four, Defoe concedes that there is plenty of football left to come this season.
“It's going to be difficult [to get fourth] with the other teams around us,” the England striker told Sky Sports.
“It's a strong league with good teams but it's important to look at ourselves and keep winning games. That's all we can do."
Spurs looked set to leapfrog Everton and climb into fourth place last Sunday when they led 1-0 in the closing stages at Goodison Park, only for the Toffees to strike twice in injury time to stun the White Hart Lane side.
Andre Villas-Boas’ side have also conceded late goals in several other Premier League games this season, and Defoe admits the team were nervous in the dying stages, as the visitors piled on the pressure in the hope of snatching a late equaliser.
|WINDOW SHOPPING: TOTTENHAM
|11/1||Tottenham are 11/1 with bet365 to beat Stoke 3-0
“It's worrying that we've been conceding late goals but I thought it was a fantastic performance and it's important that you concentrate even towards the end, as they put us under a lot of pressure towards the end,” the former West Ham striker added.
“We're not negative. Even though we've conceded late goals I was confident we'd get the three points today. It was a great finish from Jan and we're delighted.”
The winning goal from Vertonghen was his first in the Premier League for the club following his £9.5 million move from Ajax in the summer, however the Belgian defender is counting it as his second, after his solo effort in the 3-2 win at Manchester United was ruled to be a Jonny Evans own goal.
“I count two goals in my head, I don't know why they took it [the goal against Manchester United] away from me,” the 25-year-old said.
“Swansea did play good football but we didn’t give too many chances away and we were strong at the end and we concentrated towards the end because we know what happened in the other games.”