England claimed a 2-1 win over Tunisia in their opening Group G encounter in Volgograd on Monday, but looked set to have to settle for a draw after Harry Kane's opener was cancelled out by a contentious spot-kick from Ferjani Sassi.
Sassi converted from 12 yards after Kyle Walker was penalised for catching Fakhreddine Ben Youssef with his arm, which the defender disagreed with after the match.
England then saw two penalty appeals waved away despite Kane appearing to be impeded in the box on each occasion and the presence of VAR, which could have allowed for review.
The Three Lions seemed to be running out of ideas as they struggled to create chances in the second half, having wasted a plethora of opportunities in the first.
However, Kane headed home the winner at the far post in the 91st minute to spare England's blushes, though Southgate still expressed his bemusement at referee Wilmar Roldan's decisions.
"We recovered from a really harsh decision and kept our composure," Southgate told a media conference. "Our set-plays were a threat all night, we worked on them and got just reward for that.
"I think if it's a penalty at one end it has to be a penalty at the other. Once the first one is given, it wasn't going to be overturned [by VAR] because it wasn't going to be a clear and obvious error.
"If penalties are going to be given for that, it's going to be an interesting tournament, I think there were similar offences at the other end of the pitch."
Meanwhile, Kane admitted the official's decisions were not easy to deal with on the field, but was pleased the team pushed on despite their frustrations.
"It's part of the game," Kane told reporters post-match. "We've seen in some of the other games that there have been a lot of penalties and a lot of what I would say are soft decisions.
"We have to be aware of that and obviously we've had one go against us and probably another couple that didn't go our way. But as players it's difficult - all we can do is get on with the game and it's down to the officials.
"Hopefully when they watch it back they can get it right for the next game. I'm more proud of us that we kept going and fighting and sent a lot of fans home happy."
England are next in action against Panama on June 24.