Regarding Woody's appeal, this is just so subjective. Something that makes a lot of people fall down laughing uncontrollably will make other people roll their eyeballs.
Beyond that, the man can write. There is Jewish humor that might pass over a lot of us, but there is also a lot of insight into what makes people neurotic and awkward in our society. Further, the man can write for women. He is one of the best writers of female characters out there, and most any actress who has worked for him would tell you that. He has evolved from an erratic but often brilliant purveyor of nutty, slapstick humor into an erratic but often brilliant writer of modern relationships, often with humor but sometimes not. His films are not all good. Some are pretty awful. But Allen turns out a movie a year and perhaps one in three of them is a real gem, and another is at least entertaining. How many writer-directors with forty-five films in his resume could say that?
There are almost no pretense or bombastic sound and image in his films. His opening and closing credits almost invariably occur as white text black background. He uses (sometimes redundantly) jazz and classic music rather than overblown, wall-to-wall orchestral scores. His directors of photography know he wants simple camera setups and he hold shots of the actors longer than most directors, allowing their performances (rather than a lot of quick cuts and jerky camerawork) to shine.
There are maybe fifteen movies of Allen's I do not care for and will not own, but the other thirty already are or will eventually be in my collection on Blu-ray.
As for the nay-sayers, maybe you really like Michael Bay movies...?