All four sides in Group B started the final round of matches with a chance of progressing and it was Real Madrid who advanced along with Borussia Monchengladbach after a 2-0 victory for the LaLiga giants.
Lautaro Martinez struck the crossbar early on and missed several other chances of a frustrating evening for Antonio Conte's side, who could not fashion a winner to put them through and ended up bottom of the group.
Injury-hit Shakhtar posed more of a threat in the second half but were similarly unable to make it count, though they do have the consolation of a place in the Europa League.
Inter started strongly and they were almost in front when Martinez rattled the crossbar from eight yards out after Nicolo Barella squared the ball from the right just six minutes in.
Martinez then turned sharply and blazed over and Stefan de Vrij was unable to finish from close range when Shakhtar failed to deal with a corner.
Vitao was substituted after seemingly picking up an injury, having been fortunate to escape with a yellow card for a cynical foul on Romelu Lukaku, and Martinez headed straight at goalkeeper Anatolii Trubin late in a first half dominated by the Nerazzurri.
Trubin produced a fine reflex save to keep out Lukaku's glancing header early in the second half and Martinez sliced an effort wide as Inter's anxiety mounted.
Milan Skriniar's strike deflected just wide of the far post, but Shakhtar could have snatched an opener on the break, with Mateus Tete's shot palmed away by Samir Handanovic and Maycon firing off target following up.
Substitute Alexis Sanchez nodded a great chance over in the closing stages and then rose to meet a corner but saw his powerful header hit Lukaku, which summed up Inter's night, and Trubin tipped over Christian Eriksen's drive as both sides bowed out.