Chasing the game after Emmanuel Gyamfi gave the hosts a 51st-minute lead, the Blues suffered another setback when Abbey pointed to the spot on the ground of a foul on Obed Owusu by Gideon Ahinkorah.
Leading scorer Abdul Fatawu Safiu stepped up and coolly converted a 69th-minute 'panenka' to make it 2-0 in the Zone A matchday seven fixture at Kumasi's Baba Yara Sports Stadium.
"The penalty was not a penalty," Armah said after the game.
"He [referee] clearly misread the collision to be a foul when it wasn’t.
"He could have given himself ample time to think that decision through. Apart from that, he had a good game."
The loss has seen Chelsea slump to second from bottom on the table.
“We did well, we could have built on from our good start of the game but we didn’t," Armah continued.
"My players lost control of the game, Kotoko took control of the second half and were the better team for most of the period.
"I don’t think Kotoko were better than us overall though. We could have done more for ourselves.
"I congratulate my team for not crumbling even when Kotoko scored two. We have a lot of positives and a few minor mistakes that we will go back and correct."
Kotoko, meanwhile, continue their reign at the top of the table.