TVs wouldn't make the investment to switch between analog/digital audio because they would have the added cost of multiple D to A and A to D conversions. I'm pretty sure that any HDMI source will not output from a TV's L/R analog RCAs. But I am assuming that it will switch and output from ALL HDMI sources on a digital audio output. However, the digital audio out will not output analog audio sources such as component video or composite video. So if you have a mix I'm guessing you have a no go but possibly for different reasons than you stated. But if all are the same, whether digital or analog, it could work. I am only trying to look at it from the logic of how things 'should' work....but I'm speculating here.
I use my TV as the selector but I stayed with component video on everything as it's only a 37-inch set. If I change the set to a newer unit with more HDMI and less component inputs then I could have new challenges ahead. Btw, did I happen to mention that I have zero passion for video, unlike audio? I watch the content but I don't get into the technology.
I do doodle.