Mohamed Salah's 15th goal of the season gave Liverpool a 65th-minute lead against his former club, but Chelsea substitute Willian saw his cross loop over Simon Mignolet and in five minutes from time.
Klopp revealed he had wanted to make a change of his own in the moments leading up to Chelsea's equaliser and claimed Oliver was obstructive.
"I was angry because we wanted to change the system and the ref didn't allow us to," he told BT Sport.
"We wanted to go to five at the back, to change the situation and it's not too cool to have conceded.
"I don't know what he thought in this moment, he told me something [saying Liverpool] took too long. I don't understand it, how can a change take too long?
It ends level at Anfield. pic.twitter.com/0hKHrS9eZ3— Liverpool FC (@LFC) November 25, 2017
"I think it could have helped a lot where Willian was with the ball so that's why I was angry."
Nevertheless, after a humiliating collapse from 3-0 up to draw 3-3 against Sevilla in midweek, Klopp had plenty of praise for how his players acquitted themselves.
"It was two sides with a fantastic attitude after a week like this [playing in the Champions League] – running like this, playing like this," he said.
"The line-up for Chelsea was for a nice counter-attacking game, which makes it hard. Eden Hazard is difficult to defend against.
"We had moments. We scored a fantastic goal but this type of game is not one where you can win three, four or 5-0. You need a little bit of luck and I don't think we had it too much.
"Chelsea fought hard and deserved the point but it doesn't feel perfect."
Salah's form after a false start in the Premier League at Chelsea has become the major positive of Liverpool's campaign to date.
"I love this player," added Klopp. "He is fantastic. He did really well. In this type of game he can play as a nine as well, which is good for us.
"But it was a great team performance in a really difficult game against a Chelsea team defending deep and then playing on the counter-attack. [N'Golo] Kante running so quick both, wing-backs are unbelievable."
The result left Liverpool fifth, one point behind Tottenham in fourth.