The Swans lost 3-0 at the weekend to an Everton side containing Neville and the 35-year-old believes the club are missing the centre-back after he returned to Tottenham following a year-long loan spell in Wales.
"They've lost a couple of really good players - [Scott] Sinclair [to Manchester City] and Joe Allen [to Liverpool]," Neville told reporters.
"Caulker - I know people talk about Joe Allen - but I think Caulker was a massive rock at the back for Swansea last year.
|7/1||Everton are 7/1 with Bet365 to beat Leeds 2-1 in the League Cup third round
"I've had the pleasure of working with him for a short space of time [as coach of] the England Under-21 team. He really impressed me - [an] out and out defender. He was technically good on the ball, but he did like to defend.
"And I think out of all the players that Swansea have lost this season, maybe Caulker was the one that they probably miss more than anything because in the Premier League forwards are getting quicker, better technically and goal scoring this season has been frightening.
"And I think if you keep clean sheets you're going to win most games and at the moment Swansea are probably finding that difficult."
Michael Laudrup’s side started the season in blistering fashion - taking seven points from their first three matches but they have lost their last two league encounters, and Neville believes the Toffees' physicality was key to their win.
He added: "I wished we played down at Swansea every week. Our performances there have been some of the best that we've produced.
"Last season we went there and it was at the time when Brendan Rodgers' side, you know, they were making 500-600 passes a game and they were totally dominating.
"And we went down there and we gambled, in a way the tactics were just to flood the Swansea half of the field and just stopped them playing out of the back and to try and win the ball back as early as we possibly could.
"And I think that had a little bit of a mental thing in Saturdays' game because I think they were a little bit wary of us and we carried off where we left off last season.
"We did the same last season and we're a physical team and Swansea are a technical team. They probably don't have the physicality that we've got."