I tried a different USB cable (with noise suppressor) but that did not help. Then I found a bad ground on an antenna patch cable, that got rid of most of the spikes. Thanks for the replies, it motivated me to dig deeper.
BTW this highlights a feature of the RSP1A – it is very sensitive. When I used the same patch cable on my GRE PSR-600 there was no apparent problem. Turns out the RSP1A is much more sensitive than the PSR-600.