Definitely EE. I'm an EE senior at UT and have been building speakers for a few years now, and there is significant overlap between my course material and my hobby. Some of the more enjoyable courses for me have been:
Circuit Theory
Linear Systems and Signals
Electromagnetic Engineering...