Netherlands captain Van Dijk was booked in the 13th minute of the Reds' 1-0 victory when his studs caught Mertens' ankle after he swept the ball away in a slide tackle.
Van Dijk claimed he had won the ball cleanly but Ancelotti argued he was fortunate to remain on the pitch.
"I've seen it on the video, I think it was a red card," the Italian said.
"There were a lot of questions about VAR. When VAR comes in to the Champions League, it's too late."
Mohamed Salah's first-half goal sent Liverpool through at the expense of Napoli, who had to settle for a place in the Europa League despite accruing nine points in their Champions League group.
A draw on Tuesday would have been enough to seal Champions League progression for Napoli, who beat Liverpool in the return fixture, but Arkadiusz Milik was denied a late leveller as Alisson stood up tall to deny him from close range.
"We knew that Liverpool would put us under a lot more pressure, they were very aggressive," added Ancelotti.
"It was something we planned for, we knew the intensity was going to be a lot different from the first game we played. That was something we had to try and cope with.
"You can't expect to come to a place like Liverpool and expect to create 10 really good chances, we created one or two and didn't have the luck that we needed."