After failing to break down their stubborn opponents over 90 minutes, les Parisiens nicked an advantage through Adrien Rabiot in the first minute of stoppage time at the Parc des Princes, before Zlatan Ibrahimovic sealed the hosts' victory.
PSG had 24 shots to Guingamp's seven but Cavani was simply relieved to get the points as the reigning Ligue 1 champions continued their stuttering start to the 2013-14 season.
"It was a difficult match, very tough," the Uruguayan forward told reporters.
"I think we attacked quite a lot but maybe we were a little bit messy sometimes and this gave us a lack of confidence or solidity in front of the goal.
"But eventually we scored and were able to get three very important points for us."
PSG have started the Ligue 1 season with a pair of comeback draws, while their past two games have seen them score late winners against Nantes and Saturday's opponents.
Despite their sluggish start under new head coach Laurent Blanc, PSG have moved up to second in the Ligue 1, just one point behind leaders Marseille, who have a game in hand.
Top scorer last season Ibrahimovic believes they are beginning to build momentum.
He told PSG TV: "We wanted to score early in the game and we didn’t. I hit the post I think if I had scored with the free-kick the game would have been totally different.
"We are slowly reaching our rhythm and I think we only need to click from the beginning of the game and everything will be much easier."
After the international break PSG travel to Bordeaux before a Champions League clash with Olympiakos on September 17.