The Ibrox club claimed their eighth win in a row in the Third Division as Lewis McLeod and Kyle Hutton also scored to help the leaders rout the second-place side
Both Daniel Moore and Mark Nicolson had pulled the hosts level twice during the first half before Rangers raced away to their eighth consecutive win.
Elgin made one change from the side that drew at Hampden against Queen's Park on Saturday, with Jamie Duff replaced by David Niven in defence for Ross Jack's outfit.
His opposite number Ally McCoist brought back Brazilian defender Emilson Cribari in place of Francisco Sandaza, which meant that skipper McCulloch was moved from a defensive role to a more attacking one.
Rangers opened the scoring within the first five minutes as David Templeton played through Little, with the Northern Irish forward breaking the offside trap and placing the ball past John Gibson in the Elgin goal.
The equaliser came just after the quarter of an hour mark as a free kick was launched forward and Moore got control of the ball just outside the box and fired the ball past the outstretched arms of goalkeeper Neil Alexander and into the corner of the net.
McCoist's side got back on the front foot and took control of the game once again as Templeton raced down the left-hand side of the box, before crossing for Little, who from less than six yards out stroked the ball into the net.
On the half hour mark Elgin managed to get level as Chris Hegarty gave away a free kick on the edge of the area and that allowed Nicolson to step up and curl the ball into the top corner, with Alexander left helpless.
Five minutes before the break Rangers once again got back into the lead as the ball was worked up from the back, with Little eventually taking control on the right wing and finding McCulloch, who held off his marker and turned and shot beyond Gibson.
Rangers managed to claim their fourth of the game as McCulloch got his second of the day as he was played in beautifully by Templeton after the winger received the ball from a throw in, playing the ball into the box for his skipper, who placed the ball in the bottom corner.
Minutes later the visitors made it five as Barrie McKay raced down the left wing before crossing to the edge of the area and the ball broke towards McLeod, who struck the ball through the Elgin defence and into the net.
With seven minutes left the scoring was finished as Hutton received the ball in the middle of the Elgin half, before moving forward and smashing the ball into the net.
A minute later Hutton gave away a penalty as he brought down Stuart Leslie, but Alexander came to the rescue to save his second penalty in two games as Rangers saw out the remainder of the game.