The actor, 47, plays the royal in the upcoming fifth series of the Netflix show.
He told Radio Times that since he was cast people approach him with stories about Andrew, who was in the news over sexual assault allegations that he went on to settle out of court.
"It’s difficult, because a lot of opinions about Andrew in recent years have rightfully been negative, so people will turn any ambiguous story into a negative one if they’ve decided somebody is a wrong 'un," he was quoted as telling Netflix.
"So many people have come up to me since I got the role and told me stories – either they met him, or someone they know met him, or their parents met him – but while the stories themselves were innocuous there was usually a spin put on them."
"His behaviour, and the choices he made, have made him into almost a pantomime villain," he added.
The Crown started in 2016 and so far four seasons have been released.
The show has been a big success with critics and viewers alike but there had been calls for the streaming service to add a disclaimer to make it clear that it is fiction.
Then last month Netflix added one to its marketing, which said the show is a "fictional dramatisation" that was "inspired by real-life events".
The show returns on 9 November.
Watch: Elizabeth Debicki admits royal news makes her 'nauseous'