Peter, the reason it sounds better with the optical is because your RP56 is not doing the digital-to-analog conversion for music. The bitstream is passed to the 320's DAC, which I'm sure is much better than the one in the RP56. I have the RP56 too, but use a dedicated NAD for my 2 channel music...
I second ChrisAG's suggestion. If you're not in a hurry, take your time and listen to all three and try to make as accurate an evaluation as possible. If you go the Marantz route, definitely use an authorized B&M dealer as it will facillitate any potential return hassles. I just placed an order...