Workshop items downloading if person has the addons (enabled and disabled)

On my server, the workshop addons download even if I have them enabled or disabled. It gets annoying after a while as it takes like 20 minutes to DL everything. Is this a Steam issue or just some weird GMod issue?

If you can, swap to FastDL instead because workshop download is fucking trash