Ally McCoist’s men were forced to settle for a draw in their first fixture in the bottom flight of Scottish football and needed a late strike to snatch a point after falling behind
Rory McAllister and Scott McLaughlin had put the Blue Toon into a second-half lead after Barrie McKay opened the scoring for Rangers.
The first real chance for the Ibrox side came a quarter of an hour in when Dean Shiels received the ball in the middle of the Peterhead half and raced forward before unleashing a low strike from the edge of the area, but unfortunately for the Northern Ireland international, Peterhead keeper Paul Jarvie went down to claim the ball.
Jim McInally’s side were then given a golden opportunity to take a shock lead after slack play by Dorin Goian and Kirk Broadfoot allowed McAllister a chance at goal, but the Blue Toon striker could only hit the ball into the arms of Neil Alexander.
Just after midway through the first half, Rangers got their first goal in the Third Division, as McKay got onto the end of a knockdown header by Lee McCulloch after a long ball from the back to open the scoring.
With 10 minutes left in the half Peterhead should have been level as Rangers switched off at a free-kick, allowing McAllister a one-on-one chance with Alexander, but the former Inverness Caledonian Thistle striker somehow managed to screw his shot wide from six yards out.
The visitors continually found it a struggle to get forward against a tight Peterhead defence, while at the back both Broadfoot and Goian continued to create problems for their own side with poor marking.
McAllister nearly levelled things up on the hour mark as he received the ball on the edge of the area after a pass from Steven Noble, but Alexander managed to get a hand to the ball in order to divert it for a corner.
The Peterhead striker however would not be denied minutes later, as McAllister got onto the end of a cross from the left, beating American international Carlos Bocanegra, before striking beyond the arms of Alexander to the delight of the Balmoor support.
Desperate to get back ahead, Rangers sent on Spanish signing Francisco Sandaza and he had two opportunities to get the Ibrox side back into the lead, but on both occasions he found Jarvie more than up to the task.
Then Shiels had an effort after being put in on goal by Sandaza, but the former Kilmarnock forward once again found the Peterhead stopper in fine form as he touched the ball away from danger and out for a corner.
McCulloch was next to have a go for Rangers as he received the ball on the right hand side of the penalty area, but Jarvie managed to get his body in between the ball and goal to prevent his side falling behind.
But instead of going ahead, Rangers found themselves behind as Goian gave away a corner, which was swung in from the left and Alexander could only punch directly into the path of McLaughlin, who unleashed an unstoppable strike into the roof of the net.
Rangers' blushes were spared as the game entered injury time, as substitute Kevin Kyle got his head to a corner from the right and after his effort hit the bar, Little poked home from a yard out to save a point.