I just looked at reviews of the 2017 Vizio M60-C3, and it appears only HDMI 1 on the TV has HDR capability, so that is the port you will want to plug the Sony X800M2 into.
Any of those would work. If DVD is available, I would just use that one. The player sends the audio over that HDMI port toi the receiver, and a 480i screen that will show the following:
If you see this screen, just change the input on your TV to the port that the HDMI1 from the X800M2 was...
The best way for you to hook up that player to your Denon 2311 is to run HDMI 1 from the X800M2 to an open HDMI input on your TV (which you already did), then run HDMI 2 (Audio) from the X800M2 to an open input on your Denon receiver. You will then need to go into the settings menu on your...