Onkyo HT-S5300 or HT-S6300?

Either one will be a good "starter" system because the Onkyo systems provide an upgrade path - they are built around a real, expandable receiver.  Many cheaper systems are NOT expandable.  The Onkyos are pretty much the gold standard for all-in-ones.

Most likely, you will upgrade the speakers first - they are the weak link in every Home Theater in a Box system.

All 7.1 systems can be configured to run as 5.1 (just be sure you use the "Surround" speaker connections on the back for 5.1 surrounds and not the "Surround Back" speaker connections).  Corner setups are a bit of a challenge, but the built-in calibration and setup utilities are pretty good at compensating for the odd placement.

The two systems are virtually identical except that the 6300 will convert analog video sources to digital (wow - there's a theme today!).  This is handy if you have a Wii, VCR, or any other older, non-HDMI source - it simplifies the connection and operation.

If you don't have any analog sources, the 5300 should serve you well.

Probably not.

When it comes to all-in-one systems you can't do much better than the Onkyos.  You can get better overall system performance by getting a separate reciever, 5.0 speaker package, and subwoofer, but you really can't piece all of that together for $500.

Realistically, a budget of at least $800 is needed (roughly $300 for a receiver, and $500 for a separate 5.1 speaker system), and $1000-1200 will give you many more options.

The only other approach is to get a receiver and start with 2 speakers for stereo and build up your speakers over time, but this means you'll have to "wait" for the full surround sound effect.

At least with a system like the 6300, you're getting a receiver that will last - even if the speakers dont.