Amakhosi squandered too many goalscoring chances throughout the game and individual mistakes at the back proved costly as the Team of Choice raced to a 2-0 lead after two quick second-half goals from Daylon Claasen.
Samir Nirkovic's goal 12 minutes from full-time set up an exciting finish but Middendorp's side was unable to rescue the game in the closing minutes.
"I don't know, there are other games probably we should have lost but a game like this to lose it's definitely very difficult to process," Middendorp told SuperSportTV.
"First-half, from the beginning we were active, we had five or six minutes where two shots, I don't know, normally with this type of [football], you can repeat it, you should play the Premier League in England or whatever but good, we had these opportunities to make it 1-0, 2-0, 3-0.
"We had chances but we didn't use them and all of a sudden we're 2-0 down. It's something you really have to swallow."
With Saturday's win, the Team of Choice is unbeaten against Chiefs this season, and Middendorp feels his players just can't minimise mistakes whenever they come up against Eric Tinkler's men.
The 61-year-old mentor recalled how Amakhosi brought Maritzburg United back into the game on two separate occasions late last year - in the Telkom Knockout Cup and in the league.
"We had a Telkom [Knockout] Cup game in the semi-final, we brought them [Maritzburg] into the game. At 1-0, everybody knows, a back pass too short. The second game, we brought them back into the game, on the last line, totally unnecessary and today again," he reflected.
"I said it before, minimise your mistakes. We have not done badly so far. The defence... the rest defence was good but this was a casual moment just to give them this opportunity.
"I said it before, this should not happen and then you're in a miserable situation, unfortunately, it happened."