The Toffees have won just two of their first 11 matches in the English top-flight this season and sit just three points above the relegation zone.
VAR upset Everton October's 3-2 defeat to Brighton, awarding the home side a penalty in the second half. Video technology then haunted them again last week when a supposed handball from Dele Alli was checked for four minutes, but went unpunished.
VAR was also the subject of criticism from Liverpool boss Jurgen Klopp last week after Roberto Firmino had a goal ruled out because his armpit appeared to be beyond the last defender.
“When we talk about serious moments, very important moments in football, it’s not right to sit here and everyone wants to laugh about it,” Klopp said.
“It is not to laugh about. It is too serious. Managers get sacked for losing football games. They just have to clarify it. I am not saying anyone is doing it on purpose.”
Meanwhile, Chelsea boss Frank Lampard believes VAR has put the league "in a really dangerous place", adding: “We are going to be tossing a coin every week to see what decisions you get.”
And Everton boss Silva echoed the pair's concerns, arguing every manager would agree that the system is not working.
"I'm not talking about myself here, he (Klopp) was talking in general terms and he was completely right," he told Sky Sports.
"It is also my opinion and the opinion of all the managers in the world, not just here in the Premier League.
"It was so clear in the last three or four games and we are talking about it costing us five or six points.
"Where would we be with those points? I don't have any doubts that we would be performing on a different level if we had those points.
"Every team in the world plays with more confidence when it is higher up the table. Now we are not in the position we want. I am not happy with the position we are in, but I know things would be completely different if football was fair with us.
"Now we go into another must-win game [at Southampton], and every game is a must-win game for us when we would have had more freedom and confidence had we been higher in the table."