[Help] How do I make it so clients don't download custom files

How do I make it so that people or friends entering my gmod server don’t download my addons? Is it possible for them to not do anything and for me to just make that happen? I have a pretty much vanilla gmod with addons if that changes anything.