The Knappen deservedly move on to the knock-out stages of the competition, with the Austria international's 25-yard strike proving enough to see off the Greeks
A solitary Christian Fuchs strike was enough for the hosts, who now sit in first place in Group B with one match to go.
Coming out of an unexpected 2-0 defeat to Bayer Leverkusen in Bundesliga on Saturday, the Gelsenkirchen side were eager to bounce back with a win in front of their home crowd and applied the pressure early on.
The hosts had a great chance of taking an early lead in the fourth minute when Austria international Fuchs crossed a ball from the left flank into the danger zone, but Klaas-Jan Huntelaar was unable to control the ball in front of an empty box.
In the 10th minute, Germany Under-21 captain Lewis Holtby was given the time and space to shoot just from outside of the penalty area and took the opportunity, but the youngster saw his effort crash off the crossbar.
Then a superbly-placed free kick by Fuchs in the 23rd minute was well anticipated by former Manchester United goalkeeper Roy Carroll, who prevented it from sneaking inside the right post.
Huntelaar had another chance when he tried to take advantage of some poor positioning in the Greek defence within the penalty area during the 30th minute, but he could not find the left-hand corner of the net.
The visitors were also constantly looking for counterattacking opportunities through an occasionally shaky defence. Indeed, a terrific breakaway in the 35th minute caught Schalke short at the back, but Giannis Maniatis could not quite get on the end of a cross by Djamel Abdoun.
Another promising approach towards the home goal by Paulo Machado was halted by a bodycheck from ex-Olympiakos defender Kyriakos Papadopoulos, who was booked for his troubles.
At the stroke of half-time, Julian Draxler tried to give his side the lead before going into the locker room, dribbling the ball from the halfway line to the edge of the penalty area with fast pace, but his effort went over the bar.
Die Knappen tried to pick up where they left off in the second half, continuously trying to make some good chances pay.
Jefferson Farfan, in particular, was attempting to up the pace coming from the right on several occasions with some good crossing efforts placed closely to Carroll's box, but his team-mates could not make use of them.
The deadlock was finally broken in the 77th minute when a stunning strike from Fuchs found its way into the bottom corner from 25 yards out for the final result.
It was no less than the hosts deserved, having dominated much of the match, and was enough to secure all three points, as well as their place in the last 16. Olympiakos, meanwhile, will now spend the second half of the campaign in the Europa League, with Montpellier having been beaten by Arsenal.