I think it’s pretty normal to have long load times (the game even freezes for a bit) when joining a server with addons. The load times would be entirely dependent on your computer specs, how many background programs/spyware you have running (virus scanners also slow down read times), and the amount of addon files gmod has to process each join. Something that may help (it helped me), is to delete lua files of addons you don’t need access to in singleplayer, but leave the sounds/models/etc. This will allow you to still see everything on a server, and the server will download any needed lua when you join so everything is still usable.
For example, I deleted all the lua from the stargate addons but left the models. I can join and use stargate on servers that have it, but my load times have been largely reduced from the 1000+ lua I deleted. I just can’t spawn anything stargate related in singleplayer other than models.