You'll get the most bang for the buck with insulation as compared to solar. But yeah it is more difficult, walls have to be opened to do it right, etc. You mentioned it's drafty, so I'd start with easy stuff, like properly sealing around doors and windows. That will account for a surprisingly big chunk of the power bill right there.
Solar heat is a good backup but not necessarily a good single source heat system. Radiant solar systems integrated into a building are the most efficient but are also the most difficult to retrofit. In its simplest form, solar heat is just black painted or colored water tanks in an insulated box with a clear surface facing the sun. I can't find a pic now, but I saw one somewhere on teh web a while back that was a wood box with tubes made of that black PVC pipe. It was in the south facing wall of a free standing shop, during the day sunlight heated the black tubes and thus the water inside them. At night, the tubes radiated heat to the inside. Link to only one of zillions of DIY solar heat pages:
http://www.builditsolar.com/Projects/WaterHeating/water_heating.htm#1KSolarWater
If you can replace part of your electricity for heating with passive solar rather than trying to convert sunlight to electricity (big $$$ comparatively speaking), you have saved money.
Passive solar works in other ways as well. There was a house in Arizona? which had a gigantic stone fireplace inside, there were south facing windows in one wall. During the day, sunlight heated the stone fireplace and at night heavy insulated drapes were pulled across the windows. The fireplace then radiated warmth all night. If it was hot outside, the drapes could be pulled closed to cut off heat gain.