City will begin their fifth straight Champions League campaign at home to Juventus on Tuesday, with the Etihad Stadium club having generally struggled in Europe's premier competition.
In City's first two Champions League forays they failed to advance from their group, while the past two have seen them dumped out in the round of 16 by Barcelona.
While City may have moved five points clear in the Premier League with a 1-0 win at Crystal Palace on Saturday, their manager reckons continental success stands between them and legend status.
"It is impossible for you to be remembered as a great team if you do not win the Champions League," Pellegrini said.
"Of course we have been a great team in England because it is not easy to win two titles in four years and be runners-up in the other two seasons.
"I am sure we will continue to be successful in the Premier League but, apart from Yaya [Toure], we didn't have players who have won a title in Europe.
"It isn't easy. There are strong teams like Barcelona, Real Madrid and Bayern Munich, so it is very important to be lucky in the draw."
Along with Juventus, City are set to face Sevilla and Borussia Monchengladbach in Group D.
In City's 135-year history, the Manchester-based club's only continental success was the Uefa Cup Winners' Cup in 1970.