Between my wife and I, we've had three non-turbo Fozzies: two '01 Ls(one stick, one slush), and one '04 XS slush(current).
'98-02(SF) were the most "rough and ready", least refined, auto box is awful, but the most cheerful to drive of the bunch. Better trunk space than the SG, but a bit smaller in the rear seat. The tweedlike front seats of the '00-01 L are fantastic. Drives much like a GC 2.5RS from the strut towers down, because for all intents and purposes it is one, save for rear drums, softer springs, and a lack of rear LSD. The wife and I both want another.
'03-08(SG) is quite a bit more refined, but still rather squashy for the street. X/XS models are essentially a 2.5i underneath. Aftermarket bushings go a looooong way toward tightening these up, particularly in the front control arms and the whole of the rear suspension. Interior is the nicest in fit and finish of the bunch, though the front seats are flat, hard, unsupportive, and don't wear well in the non-turbo, I can't speak for the XT. Optional sunroof is the largest I've ever seen. Needs more trunk. The auto box is halfway decent. Subaru's exterior styling has always been subjective, but '06-08 is pretty fugly up front. The wife's XS would be the perfect rally weekend getaway car with better front seats and a bit more passing power.
'09(SH) and newer drive like a base model E46... Kind of heavy, underdamped, and numb, though not nearly as responsive. They're quiet and more spacious than previous iterations, but the new NA 2.5 is more suited to a lighter car, the torque band is all wrong for something so bloated. Handling is tuned toward understeer like the others, but is much more Camry than STi in responsiveness. "Giant trunk" is about all I've got to say that's good about it.
If I had it to do over again, I'd have kept my '01 L 5MT. It really was a fantastic daily. Able to be fun in corners one moment, and romp through mud puddles the next. Kind of a golden retriever of sorts. Parts interchangability with GC Imprezas means the aftermarket for handling bits is endless, and they're quite simple to work on.
Say all that to say this: you want a '03-05XT with a stick. You may also consider a turbo Outback, as they're a bit more road ready out of the box, just as good in the rough stuff, good looking, and have a bit more useful interior space.