Well first, it appears that WMP 11 "looks at" the sound settings only on startup, so doing a bunch of fiddling while WMP is open doesn't really help.
I have a fairly clean Vista (Ultimate) setup here, with no additional DVD decoders, and it mostly "just worked". If I chose speakers as the sound output, it decoded the Dolby Digital into analog. And if I disabled the Dolby Digital for the SPDIF, it also decoded correctly into digital stereo. With Dolby Digital enabled, it played the bitstream directly (in addition to the lights/indicators on the receiver, the volume control has no effect). Again, these changes were made when WMP was not open, and then restarting it.
The only time I got it to not work was disabling Dolby Digital for the SPDIF while WMP as open. Stopping playback and then hitting play again, I got no audio.
Maybe it's something you installed that is interfering, especially if it used to work? Are you using speakers, headphones, or SPDIF ("Digital Output Device")? Have you tried another "playback device", if you have one? (Right-click the volume icon in the tray, and choose Playback Devices.)