I've been here for about 25 years and love it. It's a nice looking area, housing is relatively cheap, low property taxes, and every other guy is indeed a rocket scientist. I've also lived in Chicago and Kansas City among others, and would never trade them for Huntsville. It's not like most southern cities in that you have a huge non-native Huntsvillian population from pretty much every where in the world.
As was stated before, there is a very active car club scene for most cars, we also host a hillclimb every year through TVR-SCCA, plus it has an almost sister club that's one of the oldest in the US at over 50 years. For such a smallish club, we've also had our fair share of national championship drivers in solo, and a few in road racing. You're pretty much a few hours from Road Atlanta, Barber Motorsports Park, Memphis, and Nashville Speedway, plus a long day to Robling Road, Carolina Motorsports Park, and even Indy.
And one of GRM's usual pro-drivers for the challenge lives here. He's also a home inspector and a realtor so he can help you find a house if you decide to move here, and give you driving lessons all at the same time!
A little more about the area. It has hills they call mountains, it's right on the TN river if you like watersports, and there is a huge nearby lake for additional water activity. Weather is decent pretty much year round, and you do actually get four seasons, but the winters are pretty mild.
And though even here has been hit by the economy, it's not nearly as bad as most places around the country it seems. I highly recommend it!