The Three Lions took the lead in the 11th minute through Harry Kane but were pegged back in controversial fashion as Ferjani Sassi converted from the penalty spot after Kyle Walker was adjudged to have fouled Fakhreddine Ben Youssef by hitting him with his arm.
Having wasted numerous first-half chances and seen two penalty appeals waved away, each after Kane looked to have been impeded in the box, England appeared to run out of ideas in the second half, leaving Tunisia on the verge of an unexpected draw.
But Kane headed home from a Harry Maguire flick-on of a Kieran Trippier corner to spare England's blushes in the 91st minute and give them maximum points ahead of their second Group G game with Panama on Sunday.
Speaking to BBC Sport afterwards, Southgate said: "The way we kept playing, even though the clock was running down, we stayed patient, didn't throw the ball in the box, we waited for good opportunities.
"We made so many clear-cut chances, especially in the first half, and had total control in the second half.
"We were strong on set plays all night. Even if we'd have drawn, which would have made life more difficult, I'd have been proud of the performance.
2 - Harry Kane is the first player to score against two different goalkeepers in the same World Cup match since Diego Forlan vs South Africa in 2010. Mixture. #ENG #ENGTUN #WorldCup #URU — OptaJoe (@OptaJoe) June 18, 2018
"The pleasing thing was the movement, the pace, the changing of position, the control of the ball from the back and we wore Tunisia down. That's what happened. Good teams score late goals.
"It's difficult to highlight individuals. It's a collective. We've got this result over the last three weeks, not just tonight. That's down to everyone training and working hard together.
"We felt Raheem [Sterling] and Dele [Alli] had caused huge problems, but it made sense to use the bench and get some fresh legs.
"We've given ourselves a great platform to build from. [There's] so much to be pleased about, in particular the level of performance.
"Panama will be a tough test in a different way, we've got to keep performing and we can enjoy tonight."