In theory, the image might be better with a direct-to-TV connection. In practice, I doubt you'd see any difference. And of course you'd lose any switching capabilities offered by the receiver.
You only want to run through the receiver if you have 2 or more SVideo sources. (Otherwise there is nothing to switch).
I currently have DVD, Satallite, XBox, VCR all going through my receiver with SVideo and it works fine. Just make sure all your connections are snug and the interconnects are away from the power wires.