The former Italy head coach was sent from the dugout by referee Neil Swarbrick in the 43rd minute of the champions' 1-0 victory at Stamford Bridge.
He has been given until 18:00 GMT on December 5 to respond to the charge.
Conte reacted furiously to the decision not to give his side a corner during the first half of the game and shouted in the face of fourth official Lee Mason, as well as Swarbrick.
The 48-year-old, who apologised after the match for his behaviour, also explained he had been frustrated by what he considered to be time-wasting tactics from the visitors.
"During the game, in the first half, I saw that Swansea were wasting time," he said. "I said this a few times to the fourth official.
"At the same time, I didn't see something change in the first half. I was frustrated for this situation. I tried to tell again, but then the referee took this decision.
"I repeat: I apologise for this. I was frustrated. For sure I made a mistake. During the game I suffer. With my players I suffer. It's a pity.
"They are wasting time. I was a bit animated. I said this and I shout this. It was not the first time. I think for this reason the fourth official lost his patience. But I repeat, the fault was mine, not the fourth official's. I apologise for this. I was frustrated."
Antonio Rudiger's goal 10 minutes into the second half secured the win for Chelsea and kept them third in the Premier League table, 11 points behind leaders Manchester City.