The Spurs manager has come under pressure since the humiliating 6-0 defeat to Manchester City, with his side ninth in the Premier League and 10 points behind leaders Arsenal after Sunday's 2-2 draw with Manchester United.
Villas-Boas, who was sacked by Chelsea in March 2012, claims that Sugar - who has questioned the Portuguese on Twitter and in radio interviews - has been one of the chief influences as he finds his position under scrutiny.
"I think it's a very driven agenda by somebody that doesn't honour the club, neither myself nor my players," he told BBC Radio 5 Live.
|66/1||Tottenham are 66/1 with BetVictor to win the Premier League|
While Villas-Boas looked to play down his criticism of Sugar in his post-match press conference, the 36-year-old turned his attention to the media and singled out one journalist from a national newspaper.
Asked whether he had been shown enough respect, the Tottenham boss responded: "I think it's pretty clear. I don't think so. Some people insult my integrity, my values, my professionalism.
"One of these people is sitting over here and insults the success that I have achieved at other clubs and I don't think it's fair, I think it's a lack of respect and an attack on a person's integrity.
During a tense exchange with the journalist, Villas-Boas added: "You have always chased people, you attack people. You don't attack them by the front, you attack them sitting when you write. You attack integrities, you attack competence, the integrity of a person, their human values and you don't even know that person.
"We never got a chance to sit down and speak about it. Only when I give you that chance, and I know you and you know me, are you able to reach conclusions like the ones you did. I think your articles were completely out of order."