The defender joined the Toffees from Manchester United in 2005, having won a plethora of trophies at Old Trafford.
Neville spent a total of eight years at Goodison Park, and was even assaulted during a game by a Reds fan, though he says the local clashes were "overhyped".
"I hated Merseyside derbies, I really did. I thought they were overhyped," he said on Sky Sports.
"I think a Manchester derby, we used to talk about from 48 hours before. Merseyside derbies were a week before.
"The masseur would put up on the wall 'five more sleeps'. The next day 'four more sleeps', the next day 'three more sleeps until the Merseyside derby'. It was the biggest game.
"We used to go into derby days full of confidence thinking we could compete but Liverpool always used to play the game.
"We got too involved in trying to smash someone early on, fans saying you've got to die for the badge. It was the only game Everton wanted to win."
David Moyes spent £3 million to sign Neville from United, and he was given the captain's armband almost immediately.
And he admits that Moyes' decision to make him skipper made it more difficult for him to settle in.
"They [the fans] hadn't accepted me because after one month David Moyes made me captain," he added.
"At the time I accepted it but, looking back, I shouldn't have done.
"I should have got my feet under the table and waited six months but he wanted it done and he wanted it done straight away.
"I found it really tough as I wasn't accepted by my team-mates. I was the teacher's pet, I was Moyes' little boy."