Some TF2 Maps running slowly in GMod

Probably some stupid and obvious problem, but whenever I load certain TF2 maps in GMod (seems to be PL maps mostly), my fps drops to around 7. When I pause it, the fps goes back up.

Anyone know what the fix for this is?

I’ve ruled out driver, steam and update problems, as it’s been this way as long as I can remember.