New York came into Sunday's second leg needing at minimum a three-goal win after falling 4-1 to the Crew in the first leg.
Ultimately though, the home side fell just short after winning 2-0 on the night and falling to the Crew by a 4-3 aggregate score.
David Villa gave the home side the early goal it needed to get on track, with the former Barcelona star netting from the penalty spot after Rodney Wallace was taken down in the box.
Andraz Struna pulled NYCFC to within a goal early in the second half, his shot from outside the box taking a friendly deflection to find the lower corner.
The second goal made it 4-3 on aggregate and left the home side needing just one more to advance on away goals, and Patrick Vieira's side had numerous chances to find that third goal, with Rodney Wallace's shot off the post in the 70th minute the best opportunity of the lot.
Columbus managed to hang on by the narrowest of margins, setting up an Eastern Conference finals showdown with Toronto FC.