Ramiro Funes Mori headed in an early opener from Kevin Mirallas' corner and Aaron Lennon and Romelu Lukaku were also on the scoresheet for the visitors at Villa Park.
Rudy Gestede came off the bench to head in a consolation goal but Garde accepted that his side - who are eight points adrift of safety at the bottom of the Premier League table - lack belief.
"When you prepare a plan and so early in the game you have to change it, it's very difficult," Garde told Sky Sports.
"The lack of confidence was so early because we were 1-0 then 2-0 down. We were too fragile, we didn't take the advantage of playing at home.
"The positive sign is that we never gave up and we reduced the score.
"But it's difficult to keep a positive mood when you miss chances like we missed today."
Villa supporters staged a 74th-minute walkout to protest about direction of the club under owner Randy Lerner.
"The situation is difficult but we have to fight until the end," Garde added.
"I try to make sure that everybody still believes it is possible [to escape relegation].
"It is always better when everyone sticks altogether - the fans, the club, the player, the manager, but once again when you are bottom of the league it is because there is problems and the more problems you have, the more problems come as well.
"I don't want to make any comments about [the walkout], I just try to give the players the best solution to win the next game."