A detailed Stencil tutorial

I’ve been trying to figure out how stencils work for some time now, kept giving up on it, trying it again later, repeat, repeat, and I was wondering why isn’t there a tutorial on this? And I mean a good detailed tutorial on how to use stencils in gmod? The only thing I found was: http://forum.facepunch.com/showthread.php?t=878986 But that thing is outdated and the example doesn’t even work right. Can someone please explain how to use stencils, nice and clean, so peeps like me can actually understand them?

Here’s a compilation of posts I’ve made on the subject.

Hopefully this clears any questions.

I actually get something now, thanks a bunch Bletotum.

Also, first thing I tried:

Anyone know if it’s possible to use a texture as a sort of mask when using stencils? So transparent parts of a texture don’t set the reference value, whereas anything else does?



Woo, thanks bletotum!

I don’t think it’s possible, because it accepts any material as an entire square/rectangle and thats what stencils search for. Closest thing that could work is a 3D model render.

what about making stuff appear bigger on the inside? i took a look at overv’s magic box code but it was too complicated.

The magic box works by having the insides of the box spawn somewhere hidden in the map whilst the exterior is with the player. The interior is then rendered with a stencil applied on a small portion of the exterior so it looks like there is a hole in the box, but it’s just the same as if you were really up there just a small portion of that render view.

While we’re on the topic of Stencils, there was this thread from way back. Talks a bit about stencils. I’m still wondering how _Killburn rendered that hole in the map’s wall.

As Bletotum mentioned you’ll have to add:


to make that code compatible.

This has been one area of code that I tried to stay away from, but now it’s starting to come back around. At least this gives me, and others willing to learn, the opportunity to attempt it. Thanks for the supply!

damn that’s exactly what I needed to do
And I still don’t quite understand it now after reading this thread

I wish this stuff was natively supported by source and you could just make a window that you could walk through anywhere

totally would be great to make bullets that literally put holes in walls :v:

Alas you need a different engine for that…

that’s the first problem

the second one is, there are really no gmod tier sandboxes

Render a quad on the stencil where the “hole” is, then set up a clip plane behind the hole in such a way that anything in front of the hole is clipped off. And then render the scene again. That should render everything that’s behind the wall.

As for the green walls, I simply rendered a mesh there, nothing complicated really. :v:

Does anyone have an example of how to use write/test mask? I never did figure out how to use them properly.

Test mask is applied to both the stencil value and reference value using a bitwise AND before comparing. So the result of the test looks like this:

compare(StencilValue & TestMask, RefValue & TestMask)

Write mask determines which bits in the stencil value will be changed. Basically the result of the operation on the current stencil value is calculated without taking the mask into account, then only the bits which are set to 1 on the mask are written onto the new stencil value. So basically, something like this:

StencilValue = (StencilValue & ~WriteMask) | (stencilop(StencilValue, RefValue) & WriteMask)

Never really used that feature though, but you can essentially have 8 binary stencil buffers in 1. Could definitely do something interesting with that.

So if I wanted to test for 1 and write 2 I would set the reference value to 3? (1 | 2)

What if you wanted to test for 2 and write 1?
(2 | 1) = 3, that doesn’t make sense :stuck_out_tongue:

I would swap the test and write values?