There are different quality connectors for each type don't forget. XLR connectors (if your components have them) make a very snug electrical connectors as do Neutrik Speakon connectors (if your components have them). In general coax (RCA) tends to make a better electrical connections. For movie audio, Dolby Digital, for example, the bits are delivered in packets so they tend to be there or not be there, although a better/snugger connection can't hurt.
coax cables do make better electrical connections than optical. Of course, that's largely irrelevant. The bandwidth of coax is, iirc, slightly higher, so if you have a 192/24 source...
On the other hand, using an optical cable eliminates certain ground loops.
The standard specification for digital coax is an RCA terminated 75 Ohm video cable. They wrote the spec to specifically use this existing cable type. Do not ever buy a "Digital Coax" cable. It is just a video cable that costs you double for the word "Digital" on the packaging. I've been using a $5.99 Radio Shack video cable for my cable box digital audio for 7 years and never had a dropout.
In truth, the same ones/zeros come over both wire types so there should be zero difference in amount of information.
What IS different - the RCA/coax cable creates a connection between the signal-grounds on the DVD player and your reciever. On a more sensitive audio system, this can create a slight change in the sound which has caused some people to swear there is a difference. It's not really an issue with a HT system.
My equipment - uses both. I use a coax cable for the DVD player and an optical cable for the HD PVR because that made the connections easier.
I vaguely remember some magazine, perhaps sound and vision, taking a couple if coathanger, and connecting them to cheap RCA plugs and wrapping them in electrical tape to make an S/Pdiff connector. They then connected a DVD player to and expensive DTS decoder, the sort of thing that a cinema might use that actually monitors the bitrate. They compared their homemade effort to an expensive cable.
Guess what they found? Not a single lost bit and nobody could spot the difference in blind listening trials. Mind you it's not a very practical solution, I should imagine that the resulting cable was a bit stiff.
I also use old bits of RCA video cable left over from VCRs and never have a problem.
You can read about the coathanger thing here. You should use the connection that gives you the most flexibility and the least problems (i.e. potential ground loops). Besides, you ought to have both laying around somewhere in a drawer.
I currently have both types of connexion running, because my LaserDisc player has both coaxial and optical outputs, while my computer soundcard has only optical input and output : so I run coax from the LDP to my reciever, and optical from the LDP to the computer, and from the computer to the reciever. Of course, I also run AC-3 RF [another video cable] from the LDP to my AC-3 decoder, and 5.1 discrete analog [LFE is also a video cable] from the decoder to the reciever, and a stereo pair from the LDP to the reciever for analog audio [ditto to the soundcard, through an RCA-miniplug adaptor cable] — but I doubt you have to deal with those!
I'm not sure exactly why it is recommended but I'd guess due to the limited frequencies being handled it is probably not as critical as other things and the cost of a video cable is not all that much compared to other things. When someone needs a really long cable, it is probably a cheap solution to buy an 'F' connector to 'F' connector video cable and adapters to change the connectors to RCA vs. purchasing an RCA terminated cable.
I thought there was something in the manual about using a 75 Ohm cable for the LFE, but I'm not finding it. Truth to tell, I'm using 4.1 [with phantom centre] and so I just used two A/V dubbing cables, with the red and whites for the Front and Rear L/R, one yellow for the AC-3, and the other yellow for the sub.