I don't know. I think with them it's not just a job, but an honor as far as actors are concerned. And actors, especially the caliber of actors that might get a part in an Allen or Polanski film, turn down roles all the time.
And I could see an actor gush about how he would love to work with Polanski or Allen, but if you asked about Mel Gibson, they would say "Not with that Anti Semite"