If I use only a 5.1 setup with the 5760 will it do the 6.1 matrix and send the rear center info to the left and right surrounds like the 5540 or does it only do discrete 6.1
Not familiar with the yamaha, but you are not understanding how "6.1" functions. It is in essence 5.1, to which matrix processing is applied to CREATE 6.1. Make sure you have your rear back speaker set to OFF, and everything will stay in 5.1. Please see this thread for more details: