Steven Reid did not seem to make contact with the Spain international before he fell in the box to hand Mikel Arteta the chance to convert the first of two penalties on Saturday
Mikel Arteta scored the first of two penalties that secured three points for Arsene Wenger's team after Cazorla fell in the box despite having evaded Steven Reid's tackle.
Olsson suggested that players, rather than referees, deserve criticism for the prevalence of simulation but admitted that Arsenal merited their victory on the balance of play.
"I don't like people diving," Olsson told reporters. "I'm not very objective either being a defender. Most of the time I like this league because of the fair play it contains.
"The ref doesn't have the privilege of watching replays. It's a difficult job. If it's a dive the focus should be on the person diving as well.
"It went against us. It came at a crucial part of the game. They've had a bad few weeks. It's a goal that keeps them going but they were the better team and deserved to win."
West Brom began the campaign in fine form under Steve Clarke but have now slipped to sixth place following their recent disappointing run.
Swansea and Stoke saw off Clarke's high-flyers prior to their defeat against Arsenal but Olsson is confident they are experiencing only a minor blip.
"All teams have down periods," the Sweden international insisted. "Hopefully ours is now and well get back on track as soon as possible. This is a difficult place to play, Swansea too.
"I don't think we should over-analyse them. Stoke at home is harder to come to terms with. For us it's just important to get back on track."