Do you have all the video processing for the Denon's HDMI port turned off? Try setting HDMI Pass Through and I/P Scaling to OFF. If that doesn't work, also set Video Conversion to OFF, although that will turn off the Denon screen overlay being sent to your display.
The Game HDMI input on the X3300 is no different than any of the other rear HDMI ports, so you could also try another HDMI port to rule out a defective HDMI input on the Denon. You may also want to try a different HDMI cable -- if the Xbox can only output a maximum of 1080p, then you shouldn't need a high speed certified HDMI cable -- those are only needed for 4K connections.