If you are looking for the ultimate in compact and removable, I've got a setup in my track car that's pretty awesome. It's not the highest quality sound, not that I could tell anyways in that car, but it's enough to keep me singing along on my longer more boring drives. Total cost is between $50-$100 depending on speaker choice, amp choice, and going deals. This setup does not require an existing audio system or speakers. And the only added weight to the car for the track is about 20 grams of wire.
I have some upgrades coming for it now to see if I can get some quality along with my compact removability. I've got some new less powerful speakers (to match the not so powerful amp), and new amp with a built in FM receiver and 20watts RMS of power as apposed to the 18watts RMS from the one I have now.