Not sure; I'm guessing it leaves it, because you can close the program at any time, and unlike a game, it's intended to be run in both a window and full-screen, and switching between the two does not change the video mode. Are you using a less popular resolution like 1152x864? You might be...
I'd bet the ATI player only outputs Dolby Digital through the S/PDIF output on the ATI card, and doesn't work with the Audigy. I had a similar problem with the ATI player and an Radeon 8500 working with the digital out with C-Media sound. Definitely try the PowerDVD that came with the Audigy...
It would help if you describe the problem in more detail :) No sound or stuttering? What have you tried to diagnose the problem? DD support is not built into the OS (yet); it might work with 98. It's probably more a matter of the player supporting the particular sound hardware. What player...