Kompany, who captained the side at Wembley in the absence of Carlos Tevez, saw team-mate Yaya Toure settle the tie with a 52nd minute strike to set up a final date with Stoke, who hammered Bolton on Sunday.
However, City's attentions will now turn to their league clash against Blackburn at Ewood Park on April 25 as they look to secure Champions League football for next season.
Speaking following the victory over Sir Alex Ferguson's side, Kompany told The Times: "I believe when you win games like this it should give you belief for the rest of the season.
"There is no question about us competing. We are among the top five teams at the moment who are competing for a Champions League spot.
"I think we are respected but everything takes time. It’s logical that a team that has been winning so many years gets the benefit of having that extra respect, but we are learning and if we can win this FA Cup, which is not going to be easy at all, it will be the first step towards what we are trying to do."
The 25-year-old, who joined City from Hamburg in 2008, also spoke of the rivalry between City and United, two teams that he feels are getting closer to each other in terms of quality every season.
He said: "There is still a gap between us and the team who are going to be champions but that gap is not in the games we play against each other. The difference is in how we compete with other teams.
"It’s been close in many derby games before this and it was going to happen for us sooner or later."