VTF files which are the textures for gmod cannot be compressed. But when the textures are imported into VTFedit, you can choose what format of compression you want.
*Heres some more info taken from http://tf2.gamebanana.com/threads/165481
That said, DXT1 has the best compression. In fact, it’s twice the compression the other versions have but at the cost of alpha compression (it’s either on or off) and it intentionally drops data due to the table size. Personally, I would never use it.
DXT5 has a larger table and (or for?) the alpha channel. Depending on what you’re compressing, this may or may not have a higher quality. Personally, I would use this if it works well enough on the specific file.
BGR888 and BGRA8888 are uncompressed and therefore lossless. I would use this and then 7z the whole thing if I were to upload it anywhere.
DXT3 should also be an option. It has a different way of handing alpha which may work better for whatever you’re creating. It will have the same compression as DXT5.*
*This is a list of compressions I’ve tried with a class skin-job (spy_red for example), at 2048x1024 with the file size and quality result it gets me:
(Note that I haven’t checked any of the flag boxes, or messed with resize/mipmap filters outside of what’s generally reccommended.)
DXT1: 1.4mb - Smallest file size, worst image quality
DXT5: 2.8mb - Slightly larger file size, I didn’t notice a difference from DTX1
BGR888: 8.2mb - Less compression, almost lossless texture
BGRA8888: 10.9mb - No real compression at all it seems, same quality as BGR888.*
Textures can also be compressed when they are in their raw format such as JPG or PNG. Use photoshop to lower the image quality and import VTFedit.
From my own experiance of modelling and textures, I find resizing the texture when bringing it into VTFedit lowers the file size alot so have a play around with that.
For models, you could use a poly reduction tool which most 3D applications have such as Cinema 4D or Maya or 3DS MAX.