Basically when you draw a textured polygon on the screen, each point has UV coordinates attributed to it. UV coordinates basically determine which part of the texture will be “attached” to that point. When you use a function like surface.DrawTexturedRect, the top left corner of the rectangle has (0,0) as its UV coordinates, and the bottom right corner has (1,1), which correspond respectively to the top left and bottom right corners of a texture. That results in the whole texture fitting inside the rectangle.
Now if you use surface.DrawPoly, you can input your own points with their respective UV coordinates, so basically you have full control over which region of the texture is going to appear within your polygon.
It’s not a good solution for the rectangle-in-circle image he posted though. Normally, Source supports a special blend mode on textures which allows you to use them as texture masks (it’s not specific to Garry’s Mod) and TF2 notably uses it on control point progress bars. However Valve seems to have broken this feature a long time ago for all Source games and never fixed it.
There are other ways to achieve such an effect though, but first maybe you should explain what you’re trying to do exactly.