For a cheap device, it's weirdly complex and confusing.
For its advertised functionality, it is quite simple. You want to watch Netflix or Youtube on your TV? You just hit the button. Future apps will support it the same way. Even for the secondary feature of showing the Facebook page in your Chrome browser on the TV, same thing.
The issues are for people at the edges, trying to run arbitrary video, which means you need the codec and container support to decode, and then the horsepower and bandwidth to re-encode and transmit without stuttering, losing audio/video sync, etc.
The problem is it is neither fish nor foul. It isn't a fully functional stream device, though it can do netflix on it's own.It can't natively decode anything, but it doesn't exactly mirror your device, only tabs or what you assign to it.
It's hard to really describe what it does because, frankly, it doesn't have a lot of functionality on its own.
It must natively decode H.264 and WebM; otherwise the Netflix and/or Youtube functionality would not work. For the mirroring, the host device encodes to one of those and sends those bits (perhaps over a wire first, and then) over the air.
If you mean that you can't start to watch something by itself, that you need another device -- phone, tablet, computer -- to point at something and say, "I want to watch this"; then that's feature, not a bug. Your other devices are better at finding content, and potentially better at controlling playback than any audio/video remote control ever.
The Chromecast is a very cheap and simple way to get that content from your device onto the big screen. It's not intended to be a thing on its own -- it just makes your TV a second screen for your device.