England picked up three points thanks to a dramatic late goal from captain Kane in Volgograd.
Kane had put England ahead before the controversial equaliser from Ferjani Sassi.
But Kane snuck into to seal all three points in stoppage time.
And our man Matthew Dunn was in Russia to give us his thoughts at the final whistle.
“We used to be a team that couldn’t win these games but tonight we won it,” he said.
“Perhaps this is the start of what could be a good tournament for England.”
Kane’s goal was wildly celebrated up and down England – and in the BBC studio – as England clinched all three points.
And former England defender Southgate was thrilled with his side’s performance but knows there is plenty still to do in their remaining Group G games.
Next up for the Three Lions are Panama, who suffered a 3-0 defeat against Belgium earlier on Monday.
Southgate told BBC Sport: “We’ve given ourselves a great platform to build from.
“There’s so much to be pleased about tonight – in particular for me the level of performance – but there’s a lot of work to do still.
“We knew what a tough test Tunisia would be and Panama will be a tough test in a different way because they’ll be disappointed with the way they’ve ended up today.
“So we’ve got to keep performing and we can enjoy tonight. I hope everybody at home enjoys tonight.”
The 47-year-old believed England deserved their victory and said he would have been proud of the performance even if they had not found a late winner.
He added: “The way we kept playing, even though the clock was running down, we stayed patient, we didn’t just throw the ball in the box.
“We waited for good opportunities and I think we deserved the win.
“I thought even if we’d drawn the game, which I know would have made life more difficult for us, I’d have been proud of the performance.”
Southgate was keen to stress the victory was a team effort but reserved special praise for two-goal striker Kane.
He said: “He (Kane) is a top, top striker isn’t he.
“I’m delighted for him because if he doesn’t score tonight we’d be asking questions about him not scoring goals in tournament football.
“It’s great that we’ve got him in the team and he took his goals really well.”
Captain Kane was proud of England’s battling performance but did not think Tunisia should have been awarded the penalty that led to their equaliser.
He said: “I’m so proud of the lads.
“I thought we played really well, especially in the first half, and deserved to be ahead.
“We could have scored a few more and then obviously – I’ve not seen the penalty back but from my point of view it didn’t look like one.
“Credit to the lads, we kept going. That’s what these games are about – it’s a World Cup – you go until the last second and I’m absolutely buzzing.”
Midfielder Jordan Henderson agreed that staying switched on until the final whistle paid off for England.
He said: “We’ve always been saying before the game, no matter what happens in the game we keep going until that whistle blows, we keep going, keep fighting and thankfully it paid off for us tonight with a late goal and a big win in the group stage.”