The retired shot-stopper, who formed part of the World Cup-winning 1966 squad, believes the Manchester City keeper does not face adequate competition at both club and international level.
And the 74-year-old believes that could prove problematic should Hart's form begin to slide, as he is unlikely to fear being replaced.
"Joe [Hart] is the best English goalkeeper by a mile, no question, but what concerns me is that there is nobody challenging him for that position," he told reporters.
|40/1||England are 40/1 to draw 0-0 with San Marino with William Hill
"I don't think he is the best in the world but the age he is, he stands a great chance of being the best. I don't want him to get too over-confident where he doesn't have to worry about his place. Nobody is pushing Joe Hart at all - he is not even being pushed by his club.
"You couldn't have a bad game and then say to yourself that you'd have to be in the squad because there is nobody else, you had to be on your toes."
Sheffield-born Banks also expressed his belief that the influx of foreign goalkeepers to the Premier League is harming English talent.
"We currently have no strength in depth, none whatsoever and the reason is that with the Premier League you have foreign players taking the position of young home-grown lads," he added.
"If you are a goalkeeper your standards can't be raised [by playing alongside foreigners] because you have to be in the side to play."The former Leicester keeper went on to insist that Hart will need to improve his ability to deal with crosses in order to become the best in the world.
"[Hart] is still only a young man and he can still learn a lot and when I see him there is a lot to admire: his positioning and the way he is prepared to come off his line are excellent," added Banks.
"Crosses are the only weakness to his game but it is more difficult for a goalkeeper now because of the lightness of the ball and the way it moves in the air.
"I used to pride myself on my positional play and Hart's is good but positioning sometimes does not make the slightest bit of difference when you see that ball start to swerve."