Kompany, 33, is leaving City after 11 seasons to take up a position as player-manager at boyhood team Anderlecht.
But Al Mubarak is confident the club captain, who won four Premier League titles with City, will eventually be back at the Etihad Stadium.
"It's certainly not goodbye," he told City TV.
"Vincent is a part of the family, will always be part of the family and this journey continues.
"I have a good feeling for sure that we'll be seeing Vincent back here in some capacity at some point in the future."
Al Mubarak labelled Kompany a City "legend" as the centre-back prepares for the next chapter in his career.
While he said the Belgian was irreplaceable, Al Mubarak believes Kompany's decision to move on was the right one.
"Vincent sees himself certainly in his career going into coaching and I think having that opportunity to go back to his home team, to coach Anderlecht and to play as a player-coach was a very unique opportunity for that time so I think for him the stars kind of lined up and it made sense," he said.
"From a footballing sense, I think he made absolutely the right decision and I told Vincent, 'Whatever you decide, I'm behind you 100 per cent' and he ultimately took that decision.
"We'll miss, we'll miss him, no doubt about that, we will give him the right tribute, his testimonial will be an amazing event that I'm looking forward to and this club will respect Vincent in the way that he deserves and I'm confident that the story of Manchester City and Vincent Kompany is far from over."