The Villa boss launched a scathing attack on the referee, who awarded a controversial penalty in the first half which helped hand Southampton all three points
The game was settled midway through the first half, when Halsey elected to award the Saints a penalty, ruling that Enda Stevens had fouled Jay Rodriguez despite replays suggesting that no contact had been made.
The defeat leaves Villa in the relegation zone, and Lambert was quick to lay the blame at the door of Halsey.
He told reporters: "I've had a chat with Mark after the game, and he was adamant he was a 100% right, and he's guessed. That's never a penalty, nowhere near it. Mark's got it wrong, he'll be embarrassed by it, it's nowhere near it.
"You cannot go down if you've not been touched. In big moments in big games referee's have to get it right, there's no point in guessing or summising that something has happened, you have to get it right. That's where the letdown is."
In spite of the result, Lambert was keen to praise his players for their efforts, and believes they need to keep calm if they are to avoid the drop.
He added: "I can't fault the players for that second half, they were relentless, they created loads of chances, I cannot fault them for the way they went about it in that second half, they gave us absolutely everything.
"You can't panic, you keep going, there's a long way to go and there's going to be loads of twists and
Southampton boss Nigel Adkins was naturally delighted with his side's showing, and believed the penalty decision was correct.
He said: "When you look at the challenge on Jay Rodriguez for me there is no contact I think that's clear, Jay's leg has moved out of the way, which is just as well as I think there would have been contact. The challenge has come in, our player has evaded the challenge. If the player hadn't taken his leg out of the way there would have been clear contact.
"I thought we could have been out of sight in the first half, we had good periods of sustained pressure, but for the team and for the football club it's a massive three points for us. We've come away from home against a team close to us in the division, we've come away with three points and a clean sheet.
"That's 11 games now we've got two defeats in that 11, we're improving all the time and to come away and get three points is obviously very rewarding."