Short of trying another XBOX or A/V pack I'd say you've done about all you can. I've been working on these things for over a year now and never heard of one doing this, save yours. I'd exchange the A/V pack and give a new one a shot first.
What error code are you getting on the screen? Anytime an XBOX FRAGs (Flashes Red And Green) it will usually display an error code on the TV. The HDD could have up and died on you, it happens. You'll have to chip it and install a new HDD in order to get up and running again, of course all...
The 2 settings in the dash are the only ones for enabling digital out, so let's troubleshoot the whole setup. First check your A/V pack and make sure the optical out is lighting up, use a toothpick or pen to open the little door and see (XBOX must be on). Just becase it lights doesn't mean...