The humps are caused by band changes taking place within the RSP. Look at the gain tables editor to see the frequency limits for each band. The hump can also be affected by gain settings, set by the reference level. Dropping the ref level down to around -20 to -30dBm can help to reduce it. Do you need to have the ref level at 10dBm ?
If you want a clean trace as a baseline, try using differential mode with offset. Check the manual, pages 6 to 12. I’ve attached a few screen shots so you can see how it works.