The Rossoneri conceded early in the first half and despite playing with a man advantage for the best part of the second, following Yuto Nagatomo's red card, they could not find an equaliser.
Allegri has put much of the blame for his side's fourth loss of the season on the shoulders of Valeri, who he believes made many mistakes on incidents that were "really obvious."
"I never talk about referees. But, I have to now. The referee got it wrong in this game," Allegri was quoted by Corriere della Sera."In the first half he made a technical error. He blew the whistle for half-time 12 seconds early. Then there was a last-man foul on [Urby] Emanuelson which was given against him.
"There was maybe a penalty on Robinho, the Nagatomo red card was obvious as he couldn’t not give him another yellow for that.
"This is not an excuse and perhaps today the referee was unlucky, but the incidents were really obvious."
The former Cagliari tactician expressed his disbelief as to how Riccardo Montolivo's goal was disallowed for an apparent foul on Inter goalkeeper Samir Handanovic.
He added: “I have no idea what the referee whistled for on the Montolivo goal. Let’s just say he was unlucky on the decisive incidents, as the goal would’ve been crucial. There was also a penalty and [Walter] Samuel was last man too.
“It’s not as if the referee did us a favour with Nagatomo’s red card, as he obviously stuck his arm out for an intentional handling offence.
“It’s a shame we are running into some decisions that go against us, but we can’t do anything about that. Maybe the referees are on as bad form as we are."
Despite his tirade at the match official, Allegri did admit that his side need to improve in certain areas, especially in defending set-pieces.
“We have to look at the performance and improve some things. It’s a shame to lose this way, but we have to accept the result and look at where we got it wrong. At the moment we have conceded seven goals from set plays and that should be avoided. We’ll work on it over the break," he continued.
“The season isn’t over, but we have to fight back, use the same intensity and improve on some things.
“I cannot complain about the lads’ performance. We just needed to be a little quicker and a little more precise.
“We tried to spread the play, but it was difficult to break down a team that played in defence for 80 minutes.”
The loss now sees Milan sit in 11th place with only seven points from seven games played.
“Of course it’s odd to see Milan in the bottom half of the table. Unfortunately we didn’t get a result today despite creating many chances and were penalised yet again from a set play," Allegri added.
“Inter never had a shot on goal other than our error straight after the goal and at the end with [Rodrigo] Palacio's effort,” continued Allegri.
“This derby has not changed anything. I even enjoyed it, as the team played very well and created so many opportunities, but we couldn’t do any more than that other than score a goal. To be fair, we did score a goal."