Ryan Shawcross fouled Wayne Routledge in the area to give Scott Sinclair the opportunity to put the hosts ahead from the penalty spot inside the first 10 minutes.
The Potters were chasing the game from then on and summer signing Danny Graham netted his first Swans goal to seal the win late on.
"We gave them a good start," Pulis said after the match.
"Ryan's challenge was a poor challenge which he did not have to make as the kid was going away from goal, and they get the start every team in the Premier League wants.
"It lifted them, they were nice and bright in that first 15 minutes. After that time lapsed until they got the second I thought we were the team looking to get the next goal and net an equaliser."
Pulis also believes that Stoke's Europa League game against Besiktas in midweek may have affected his side's fitness.
The Potters have now lost their last two Premier League matches that have followed a Europa League tie.He explained: "Psychologically three games at this level in a week is tough - we did look a little leggy in the first 15 minutes.
"But we grew into the game and if Walters' effort goes in or Whelan's doesn't hit the post, then it could have been different, but those are the fine lines."