It depends on the season and how many of my favorite shows aren't returming. Otherwise, i also find it's a good time toi catch up on all the DVDs I've had piling up.
Of course, how much of that has to do with how it had the field to itself and was just about the only original programming going? You can also point to dozens of shows that started during the summer and petered out quickly - see Keen Eddie, The Jury, Method & Red, MYOB, and dozens more that we've completely forgotten. Survivor was unusual, and it's foolhardy to deduce anything from one data point.
I actually look forward to summer aka Rerun Season. It means I can relax and watch more DVDs or even go to bed early. Oh, and catch up on stuff I recorded and haven't had a chance to see yet.