Either can be 5.1. The notation simply indicates a format capable of carrying 5 discrete full-range channel, plus a special channel for so-called "low frequency effects". Both DD and DTS have that capability. The receiver manual is badly written, as most of them are, but it appears to be...
A demodulator is only relevant if you're trying to get Dolby Digital from laserdisc. For DTS (in any form) or DD from DVD, a demodulator is irrelevant. You really need to describe your setup, with make and model of all relevant equipment. M.