Kane's double earned a 2-0 win in Saturday's game at the John Smith's Stadium, moving Mauricio Pochettino's side up to fourth in the Premier League.
The England captain has now scored 145 goals for Spurs and has surpassed Defoe to sit in fifth place in the club's record books.
"Jermain - I learned a lot from him when I came through," Kane said to BBC Sport.
"I used to watch him score week in week out and it's nice to go above him.
"We'll have a bit of banter when I see him and it's fantastic to be above him now."
After speculation the striker was suffering from a lack of fitness, Kane has bounced back with three Premier League goals in his last two matches.
The striker had a physical battle with Huddersfield defender Mathias Jorgensen throughout Saturday's match but Kane said his bruises will just allow him to enjoy the result even more.
"Penalties are something I practice a lot in the week so happy to see one go in," Kane said. "I feel confident, keep practicing and hopefully keep scoring.
"They played well and made it tough on a dry pitch, they went long and with the wind it was a fight. The first goal was important and the second came not too long after. We had some chances to finish it off in the second half, but a fantastic clean sheet and a good win.
"It's part of the game - when you go up against teams they try to stop you and don't want to make it easy. I enjoy those ones and when you win it makes it a lot sweeter."