Bryan,
Yes, that would be a mistake. You would "shortcut" each of the channel's outputs of one player with the same channel's output of the other player. Even if one is off all the time, that could still possibly even damage the output circuits.
In theory, it could be done with a resistor in series between each "hot" output connection and the "Y", but I certainly wouldn't advise that in practice!
There are solutions, though, but not as inexpensive as 6 Y-connectors (changing the connections by hand each time
would be less expensive, but in the end even hurt the connectors). That solution would be an
audio switch. That one is discussed on this forum in
another thread on this forum, but beware, there are still a few uncertainties about its operation.
Cees