The London outfit have secured just one point in eight games involving the top-four sides, and the 45-year-old believes he now needs to start "assessing" his side
Sunday's 4-0 defeat at Liverpool has left the club eight points behind fourth-placed Arsenal with six league games remaining.
Tottenham have secured just one point in eight games involving Liverpool, Chelsea, Manchester City and Arsenal, while conceding 27 goals in those matches.
|40/1||Tottenham are 40/1 with William Hill to finish in the top four this season|
"I want to instil belief into the players and reassure them they're not inferior to the teams they're stepping onto the field against, the likes of Liverpool, Chelsea, Man City, but the stats don't lie and the return of points against top teams has not been good enough," Sherwood told reporters.
"It's too late now because we've got six games left against none of them above us. We certainly need to address it. If we want to aspire to be one of the top teams in the division then we certainly need to pick up more points than what we've done."
An own goal from Younes Kaboul after two minutes gave Liverpool an early lead before goals from Luis Suarez, Philippe Coutinho and Jordan Herdenson fired Brendan Rodgers' men to the top of the table, two points clear of Chelsea.
Sherwood spent the entire match in the stands and the Spurs boss insists he can "assess" the club's future away from the touchline.
"[The fans] know me and know I'm not shirking any responsibility. I'm in it with the team and it's my responsibility as well as the guys who are on the pitch," Sherwood said.
"At 2-0, with the game going away from us, and at 3-0, I think I'm going to learn more about my players by sitting up there [in the stands] and getting a good look at it rather than getting involved and getting myself in trouble on the touchline.
"We've seen enough of me ranting and raving.
"It's time for me to start assessing, take a step back and looking at what we need to do for the future because our performances against teams in the top four and anyone above us have not been good enough. I learned a lot up there today."