Ashley Williams excelled for the Swans as he kept Emmanuel Adebayor and Defoe at close quarters, before Jan Vertonghen struck the winning goal with 15 minutes remaining.
The Belgium international was on hand to calmly stroke a finish past Swansea goalkeeper Gerhard Tremmel after Kyle Walker’s free-kick had left the visitors’ defence in disarray.
“It’s going to be difficult with other teams around us,” Defoe told Sky Sports.
“It’s a strong league and good teams but it’s important to look after ourselves and keeping winning games.”
Spurs were understandably nervy in the closing moments of the game after Everton’s late comeback at Goodison Park last weekend, and Defoe revealed that Villas-Boas had worked with his players during the week on improving defensively.
“It’s well known that we’ve conceded a lot of late goals, but it was a good performance from us today and it was important we concentrated until the end as they put us under pressure,” added Defoe.
“It’s something we worked on during the week.
“Even though we’ve conceded a lot of late goals, we’re quite confident we were going to the three points today and it was a great finish by Jan [Vertonghen].”