The Roma defender suffered the injury during training on Tuesday, with the German Football Association confirming the centre-back had torn his anterior cruciate ligament.
Rudiger's place has now been taken by Tah, who recently made his senior debut for Germany in the friendly defeat versus England, but missed out on a spot in the world champions' initial squad.
The 20-year-old enjoyed a superb 2015-16 season with Leverkusen, helping them to third place in the Bundesliga table with a string of fine performances.
"It's very unfortunate to pick up such a bad injury on the first day of training. I feel very sorry for Rudiger," Low told a news conference on Wednesday.
"We have decided to call up Jonathan Tah. We wanted to replace Rudiger with a centre-back.
"Jonathan has been completing an individual training routine for the last three weeks. He is very professional and hasn't been lazy whilst on holidays. He'll be ready to play in a couple of days."
Germany are being plagued by injury in defence ahead of their Group C opener against Ukraine on Sunday, with Mats Hummels also unavailable due to a calf injury.
However, Low confirmed that Hummels is making progress with his recovery, adding: "Mats Hummels was working with the ball for the first time today and looks to be on the right path. His individual training is increasing in intensity.
"We've lost players who were concretely in my plans, but we've found solutions. We have a great squad.
"We're not going to complain. We've got to try and make the best of our options and won't be making excuses. I have full faith in the group of players I have at my disposal."