The young striker marked his second start for the Easter Road side with an injury-time clincher after Leigh Griffiths' free-kick had cancelled out Darren Barr's opener
Darren Barr had put the home side into the lead just before the break, but goals from Griffiths and Caldwell secured Hibernian's first Edinburgh derby win in the SPL for four years.
The home side made two changes to the side that defeated St Mirren, with experienced defender Andy Webster and on-loan Liverpool striker Michael Ngoo returning in place of Dylan McGowan and Dale Carrick.
Hibernian made six changes to the side that started the abandoned fixture against Kilmarnock, with Tim Clancy, James McPake, Ryan McGivern, Matt Done, Gary Deegan and Danny Handling all missing out as Pat Fenlon handed a debut to young defender Jordon Forster and a second start to Caldwell.
The first chance came with 10 minutes gone as Hibernian keeper Ben Williams failed to deal with a Kevin McHattie corner, but both Darren Barr or Ngoo saw their headed attempts blocked.
Nine minutes later it was Hibernian's turn to move forward as young winger Alex Harris cut in from the right wing and shot from just inside the area, but Jamie MacDonald got down well to keep the effort out.
The visitors continued to press forward but just before the interval Gary Locke's side took the lead. Williams once again failed to deal with a McHattie corner from the right and after Ngoo's effort was cleared off the line, Barr managed to sneak the ball home whilst lying on his back.
Hibernian drew level three minutes after the break after Mehdi Taouil conceded a free-kick just outside the box for a foul on Harris. Griffiths punished the Moroccan by thundering a stunning effort into the top corner.
The home side began to grow into the game and Jason Holt found Ryan Stevenson on the right-hand side of the box, but his cross into the danger area was cleared out of play by Lewis Stevenson before Ngoo could convert a tap in.
Both sides continued at a frantic pace, with Griffiths looking a threat for Hibernian, while Holt was a nuisance moving forward from his midfield position for Hearts, but neither of them could find a way past either of the opposition goalkeepers.
With six minutes left the home support inside Tynecastle were off their feet as Walker went down inside the penalty area under a challenge from Forster, but instead of awarding a spot kick, referee Steven McLean booked the Hearts winger for diving.
As the game approached injury time, the visitors claimed the winner. Jamie Hamill’s clearance was charged down by Griffiths, allowing Caldwell to curl the ball home from the edge of the area to send the three points back to Leith.