How far back should we go?
There used to be races at the Del Mar Fairgrounds - not the IMSA races of the 1980s-90s, but much earlier. I remember about 1965, seeing what must of been a vintage race (upright-seated single-seater, front engine cars) driving at speed around hay bails in the big parking lot. I was a little kid at the time but remember it clearly. Going back even further to the 1950's, they used to race on what was a former WWII base, now buried under what's now Torrey Pines Golf Course. Oh, and lets not forget Paramount Ranch.
I think three things have conspired to end most racing: Cost of participation, liability, and property values. It's a slippery slope, as more tracks close, interest drops, increasing the likelihood that more will close. It is what it is.