The Sweden international, who moved to Carrow Road in the summer, missed out on a place in the Canaries' matchday squad in their 1-0 win over Southampton on Saturday
Since his reported £2.5million move from Blackburn Rovers during the close-season, Olsson has played just four minutes in the Premier League as Spaniard Javier Garrido has retained his place at left-back.
"Hart is really not my concern. John Ruddy is and what we have here is an excellent goalkeeper that I think is in the England squad because of his abilities."
Norwich are 22/1 with bet365 to beat Tottenham 2-1
He said: "Martin Olsson is fine. I suppose I'm at the stage that you want to be at where sometimes picking the subs can be as difficult as picking the team.
"If I look at what I had on the bench, I had to make some decisions, and it was a real tough decision [to leave Olsson out] because I've just brought him in, but I decided to go a little bit more offensive on the bench.
"But he's fine, he'll go away with Sweden now, and hopefully he gets some games under his belt."
Meanwhile, Hughton refused to comment on Norwich No.1 John Ruddy's chances of usurping Joe Hart for England, but lauded his in-form goalkeeper.
He added: "About Joe Hart I don't know [whether his form is of international standard], that decision is always for the manager.