"That falls on us because if you are winning games, that gets taken away," the Chelsea skipper told The Independent.
"If you are losing games, the pressure is on the manager. He has to take responsibility because he is the one who picks the team. So if we are not [winning], naturally with all managers, it falls on his head unfortunately.
The former Liverpool boss, now the Blues interim first-team coach, has been vocally opposed by the team's supporters since his first game in charge against Manchester City in November.
Since his arrival, last season's European champions have slipped to 16 points behind Premier League leaders Manchester United and Terry says that this run of form must end or the criticism from the stands is inevitable.
"[The fans] pay their money so they are entitled to echo their thoughts. All we can do is concentrate on our jobs and that is winning games. If we can do that, it completely takes that away. We won [against Brentford] and I didn't hear much – it takes it away," Terry explained.
"The fans have their opinion and the players and the most important thing is the team winning games."