The newly-promoted side have lost three and drawn one game thus far, condemning them to the bottom of the table.
In their latest setback, the 'Dade' Boys succumbed to a 3-1 home defeat to Bechem United in an outstanding matchday one fixture played at the Accra Sports Stadium on Thursday.
“The fans should take it easy; we are still learning to improve on our performance," Attram said.
“They should come and see how the players train and the things that they do as professionals and then they can say if I can coach or not.
“I am worried because I always want to win, so since it's not coming, of course I will be worried, but not that Olympics will go down.
"I believe in what I am putting on board, the tactics and the quality players that we have, and I believe we will survive in the league."
Ahmed Toure (2x) and Yaw Arnol struck for Bechem on Wednesday, with Englebert Gray fetching the home side's consolation from the spot.
Olympics host Wafa in their next game on Saturday.