A detailed Stencil tutorial

[CGNick]

I've been trying to figure out how stencils work for some time now, kept giving up on it, trying it again later, repeat, repeat, and I was wondering why isn't there a tutorial on this? And I mean a good detailed tutorial on how to use stencils in gmod? The only thing I found was: [url]http://facepunch.com/showthread.php?t=878986[/url] But that thing is outdated and the example doesn't even work right. Can someone please explain how to use stencils, nice and clean, so peeps like me can actually understand them?

[Bletotum]

Here's a compilation of posts I've made on the subject.

[QUOTE=Bletotum;42343414]
In short, what you want to be doing, inside of a hook appropriate to your task (3D go with PostDrawOpaqueRenderables, 2D go with HUDPaint):

Clear the stencils
Enable the new stencil
     Set the masks for halo compatibility
     Set your reference value[1]
     Set various operations[2]
     Set the comparison function[3]
     Draw something to the screen to work with reference values in order to determine the filtered end drawing.[1]
     Optionally make the stencil more complex by doing more drawing and changing the comparison function/operations.
     Draw your desired object to the screen. It will only be drawn where the comparison function/operators allow.
Disable the new stencil

[1] Whenever you draw to the screen, the points of the screen that are drawn to will adopt the current reference value, should the comparison and operators allow it. The object can even be behind obstructions and count.
[2] The operations determine what points of the screen that something could be drawn to will get that reference value. For example, if the object is behind something then the ZFailOperation (see my link) will perform a stencil operation task such as STENCILOPERATION_REPLACE. STENCILOPERATION_KEEP would cause the points of the screen where the object resides behind another object to keep their original reference values. By using replace, you could draw objects through walls by replacing the reference value of that area and then drawing the object over the now selected area.
[3] The comparison function determines where things are allowed to draw, running before the operators over the points. STENCILCOMPARISONFUNCTION_ALWAYS is required before doing ANY drawing, that way the points drawn to will be accepted for the draw and be added to the stencil. Other comparison functions let you use only points that have a reference value equal to the current set reference value, greater, lower, etc.
Basically, you can draw to a range of screen to select the area the stencil draws onto (even if you make the filtering draw invisibly such as with render.SetBlend), and continue filtering the selected area by changing the comparison and operators until you're ready to do your final draw to the screen, drawing over the selected area.

By combining clever filtering of the comparison function and your operators, you can achieve all sorts of effects.[/QUOTE]

[QUOTE=Bletotum;39671972]A single halo could only harm your FPS if it had crazy settings. Only two colors the way he's doing it is fine.

This code adds a slightly fuzzy white border around entities using stencils.
It does NOT work on players though, because ent:SetModelScale() is broken when used clientside on players.
This is much faster than the halos, but also slightly less visually interesting; every player could have a unique color without significant FPS loss, though.

[lua]
hook.Add("PostDrawOpaqueRenderables","PlayerBorders",function()
--stencil work is done in postdrawopaquerenderables, where surface doesn't work correctly
--workaround via 3D2D
local pos = LocalPlayer():EyePos()+LocalPlayer():EyeAngles():Forward()*10
local ang = LocalPlayer():EyeAngles()
ang = Angle(ang.p+90,ang.y,0)
for k, v in pairs(ents.FindByClass("prop_physics")) do
render.ClearStencil()
render.SetStencilEnable(true)
render.SetStencilWriteMask(255)
render.SetStencilTestMask(255)
render.SetStencilReferenceValue(15)
render.SetStencilFailOperation(STENCILOPERATION_KEEP)
render.SetStencilZFailOperation(STENCILOPERATION_KEEP)
render.SetStencilPassOperation(STENCILOPERATION_REPLACE)
render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_ALWAYS)
render.SetBlend(0) --don't visually draw, just stencil
v:SetModelScale(1.0+math.Rand(0.01,0.013),0) --slightly fuzzy, looks better this way
v:DrawModel()
v:SetModelScale(1,0)
render.SetBlend(1)
render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL)
cam.Start3D2D(pos,ang,1)
surface.SetDrawColor(math.Rand(200,255),math.Rand(200,255),math.Rand(200,255),255)
surface.DrawRect(-ScrW(),-ScrH(),ScrW()*2,ScrH()*2)
cam.End3D2D()
v:DrawModel()
render.SetStencilEnable(false)
end
end)
[/lua]

Here's what it looks like, minus the fuzzy and white part in the above code, with anything other than players:

[thumb]http://i.imgur.com/kxPRS6M.jpg[/thumb]
[thumb]http://i.imgur.com/pGuwoKk.jpg[/thumb]
[thumb]http://i.imgur.com/GKMoEx3.jpg[/thumb]
[thumb]http://i.imgur.com/Ju3SzK0.jpg[/thumb]

Garry's halos use stencils too, but with a bunch of filtering stuff to smooth it out that makes them slow. This code keeps me rocking 110 FPS with more customizability.

[editline]21st February 2013[/editline]

a bit of randomization to the alpha color of these borders could look nice too

[editline]21st February 2013[/editline]

You could make my code sample a bit better by using stenciling to remove the player's real size from the stencil selection, again using setblend(0), before drawing the 3D2D coloring to the screen, and then not draw the player after (the player is drawn again at the end in the sample to appear over the color, but i'm suggesting to make the color only exist around the edge in the first place). It would still look the same, though, and the FPS difference is too small to be seen.[/QUOTE]

[QUOTE=Bletotum;40080246]
and here's a tutorial on stencils

[img]http://i.imgur.com/s2ctqme.png[/img]

[editline]29th March 2013[/editline]

the tutorial example draws 3D2D text onto a 1x1 phoenix square plate, that uses stencils in order to make the text only show up on the surface of the sign entity, clipping off at the edge, despite however many characters are being drawn in 3D2D[/QUOTE]

Hopefully this clears any questions.

[CGNick]

I actually get something now, thanks a bunch Bletotum.

Also, first thing I tried:

[img]http://i.imgur.com/gSrtFVO.jpg[/img]

[rejax]

[t]http://i.imgur.com/TXQEzuB.jpg[/t]
:v:

[t]http://i.imgur.com/YnSOLSe.jpg[/t]

Woo, thanks bletotum!

[CGNick]

[QUOTE=Handsome Matt;43272308]Anyone know if it's possible to use a texture as a sort of mask when using stencils? So transparent parts of a texture don't set the reference value, whereas anything else does?[/QUOTE]

I don't think it's possible, because it accepts any material as an entire square/rectangle and thats what stencils search for. Closest thing that could work is a 3D model render.

[mitterdoo]

what about making stuff appear bigger on the inside? i took a look at overv's magic box code but it was too complicated.

[Rawknee]

While we're on the topic of Stencils, there was [URL="http://facepunch.com/showthread.php?t=1205832"]this thread[/URL] from way back. Talks a bit about stencils. I'm still wondering how _Killburn rendered that hole in the map's wall.

[QUOTE=amkoc;37277483]How do I get this, basically?
[IMG]http://uppix.net/5/1/e/7d6ed149aa1f8a156fdf837951d83.jpg[/IMG][/QUOTE]

[QUOTE=Jinto;37280318]I'm guessing this is related to the question you asked me in PM? I sat down and wrote you a template for the effect you asked me about.

You just need to fill in DrawOverlay() and DrawInterior() with whatever you want to draw.

[img_thumb]http://cloud.steampowered.com/ugc/596979049402475552/0038ED6187826B48F10C1756C3F69DEE73B6E457/[/img_thumb]

function ENT:DrawMask( size )

    local pos = self:GetPos();
    local up = self:GetUp();
    local right = self:GetRight();

    local segments = 12;

    render.SetColorMaterial();

    mesh.Begin( MATERIAL_POLYGON, segments );

    for i = 0, segments - 1 do

        local rot = math.pi * 2 * ( i / segments );
        local sin = math.sin( rot ) * size;
        local cos = math.cos( rot ) * size;

        mesh.Position( pos + ( up * sin ) + ( right * cos ) );
        mesh.AdvanceVertex();

    end

    mesh.End();

end


function ENT:DrawInterior()
end
    

function ENT:DrawOverlay()
end


function ENT:Draw()

    render.ClearStencil();
    render.SetStencilEnable( true );
    render.SetStencilCompareFunction( STENCIL_ALWAYS );
    render.SetStencilPassOperation( STENCIL_REPLACE );
    render.SetStencilFailOperation( STENCIL_KEEP );
    render.SetStencilZFailOperation( STENCIL_KEEP );
    render.SetStencilWriteMask( 1 );
    render.SetStencilTestMask( 1 );
    render.SetStencilReferenceValue( 1 );

    self:DrawMask( 48 );

    render.SetStencilCompareFunction( STENCIL_EQUAL );

    -- clear the inside of our mask so we have a nice clean slate to draw in.
    render.ClearBuffersObeyStencil( 0, 0, 0, 0, true );

    self:DrawInterior();

    render.SetStencilEnable( false );

    self:DrawOverlay();

end

You can draw pretty much whatever you want inside DrawInterior().

[img_thumb]http://cloud.steampowered.com/ugc/596979049402585430/CBEA139959EA70DB3709C6126FE614AA06650B86/[/img_thumb][/QUOTE]

As Bletotum mentioned you'll have to add:

render.SetStencilWriteMask(255)
render.SetStencilTestMask(255)

to make that code compatible.

[Cosmic Seagull]

This has been one area of code that I tried to stay away from, but now it's starting to come back around. At least this gives me, and others willing to learn, the opportunity to attempt it. Thanks for the supply!

[BFG9000]

[QUOTE=CGNick;43271995]I actually get something now, thanks a bunch Bletotum.

Also, first thing I tried:

[img]http://i.imgur.com/gSrtFVO.jpg[/img][/QUOTE]

damn that's exactly what I needed to do
And I still don't quite understand it now after reading this thread

[Curls]

[QUOTE=Ronny;43291030]While we're on the topic of Stencils, there was [URL="http://facepunch.com/showthread.php?t=1205832"]this thread[/URL] from way back. Talks a bit about stencils. I'm still wondering how _Killburn rendered that hole in the map's wall.

As Bletotum mentioned you'll have to add:

render.SetStencilWriteMask(255)
render.SetStencilTestMask(255)

to make that code compatible.[/QUOTE]

I wish this stuff was natively supported by source and you could just make a window that you could walk through anywhere

totally would be great to make bullets that literally put holes in walls :v:

[BlackMadd]

[QUOTE=J!NX;43292332]I wish this stuff was natively supported by source and you could just make a window that you could walk through anywhere

totally would be great to make bullets that literally put holes in walls :v:[/QUOTE]

Alas you need a different engine for that....

[Curls]

[QUOTE=BlackMadd;43292382]Alas you need a different engine for that....[/QUOTE]

that's the first problem

the second one is, there are really no gmod tier sandboxes

[_Kilburn]

[QUOTE=Ronny;43291030]While we're on the topic of Stencils, there was [URL="http://facepunch.com/showthread.php?t=1205832"]this thread[/URL] from way back. Talks a bit about stencils. I'm still wondering how _Killburn rendered that hole in the map's wall.[/QUOTE]

Render a quad on the stencil where the "hole" is, then set up a clip plane behind the hole in such a way that anything in front of the hole is clipped off. And then render the scene again. That should render everything that's behind the wall.

As for the green walls, I simply rendered a mesh there, nothing complicated really. :v:

[ralle105]

Does anyone have an example of how to use write/test mask? I never did figure out how to use them properly.

[_Kilburn]

[QUOTE=ralle105;43302742]Does anyone have an example of how to use write/test mask? I never did figure out how to use them properly.[/QUOTE]

Test mask is applied to both the stencil value and reference value using a bitwise AND before comparing. So the result of the test looks like this:

compare(StencilValue & TestMask, RefValue & TestMask)

Write mask determines which bits in the stencil value will be changed. Basically the result of the operation on the current stencil value is calculated without taking the mask into account, then only the bits which are set to 1 on the mask are written onto the new stencil value. So basically, something like this:

StencilValue = (StencilValue & ~WriteMask) | (stencilop(StencilValue, RefValue) & WriteMask)

Never really used that feature though, but you can essentially have 8 binary stencil buffers in 1. Could definitely do something interesting with that.

[ralle105]

So if I wanted to test for 1 and write 2 I would set the reference value to 3? (1 | 2)

[ash47]

[QUOTE=ralle105;43303245]So if I wanted to test for 1 and write 2 I would set the reference value to 3? (1 | 2)[/QUOTE]

What if you wanted to test for 2 and write 1?
(2 | 1) = 3, that doesn't make sense :P

[ralle105]

I would swap the test and write values?

[jrj996]

I don't understand how this ties into drawing circles in a HUD, like how GTA IV is done. Can anyone explain that part?

[Pugsworth]

[QUOTE=jrj996;43305520]I don't understand how this ties into drawing circles in a HUD, like how GTA IV is done. Can anyone explain that part?[/QUOTE]

You draw a circle, that is the mask for whatever else you want to draw, then you use stencils to test for if what you want to draw is inside that circle.
Then only draw what is inside the circle.

[_Kilburn]

[QUOTE=ralle105;43303245]So if I wanted to test for 1 and write 2 I would set the reference value to 3? (1 | 2)[/QUOTE]

If you wanted to test for 1 and write 2, it'd be probably easier to use STENCIL_INCR as the stencil operation.
If you set the test mask to 1 and the write mask to 2 and then use 3 as your reference value, you'd be testing for 1 and writing 3, since the first bit will be left untouched.

[ralle105]

Say I render a model though and some parts of it overlap, then it would write values >2 in places.

[_Kilburn]

[QUOTE=ralle105;43307328]Say I render a model though and some parts of it overlap, then it would write values >2 in places.[/QUOTE]

Increase only if the stencil value is equal to 1? Only STENCIL_REPLACE uses the reference value for writing, the others do not depend on it.

[ralle105]

Ooh. Major understandment right there. Rhank you!

[Pugsworth]

Since the stencil discussion is back, I was wondering if anyone was able to get the drawn texture from stencils to have alpha?

I tried drawing the silhouette of a model using a texture, but I needed it to be translucent. Using render.SetBlend() didn't do anything and I couldn't figure anything else out.

[Rimbas]

[QUOTE=vexx21322;43308506]Since the stencil discussion is back, I was wondering if anyone was able to get the drawn texture from stencils to have alpha?

I tried drawing the silhouette of a model using a texture, but I needed it to be translucent. Using render.SetBlend() didn't do anything and I couldn't figure anything else out.[/QUOTE]

Oh yeah. A while back I saw someone made a properly transparent cloaking weapon for TTT. Tried doing the same but to no avail.

[luka2477]

Sorry to bring this old thread up, but i can't get this to work. I am trying to make a prop blue when its behind something, and red when its visible. This is my code:

hook.Add("PostDrawOpaqueRenderables","PlayerBorders",function()
	--stencil work is done in postdrawopaquerenderables, where surface doesn't work correctly
	--workaround via 3D2D
	local pos = LocalPlayer():EyePos()+LocalPlayer():EyeAngles():Forward() * 10
	local ang = LocalPlayer():EyeAngles()
	ang = Angle(ang.p + 90, ang.y, 0)
	for _, ent in pairs(ents.FindByClass("prop_physics")) do
		render.ClearStencil()
		render.SetStencilEnable(true)
			render.SetStencilWriteMask(255)
			render.SetStencilTestMask(255)
			render.SetStencilReferenceValue(15)
			render.SetStencilFailOperation(STENCILOPERATION_KEEP)
			render.SetStencilZFailOperation(STENCILOPERATION_REPLACE)
			render.SetStencilPassOperation(STENCILOPERATION_KEEP)
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_ALWAYS)
			render.SetBlend(0)
			ent:DrawModel()
			render.SetBlend(1)
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL)
			cam.Start3D2D(pos, ang, 1)
					surface.SetDrawColor(100, 100, 255)
					surface.DrawRect(-ScrW(), -ScrH(), ScrW() * 2, ScrH() * 2)
			cam.End3D2D()
			ent:DrawModel()

			render.SetStencilReferenceValue(30)
			render.SetStencilFailOperation(STENCILOPERATION_KEEP)
			render.SetStencilZFailOperation(STENCILOPERATION_KEEP)
			render.SetStencilPassOperation(STENCILOPERATION_REPLACE)
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_ALWAYS)
			render.SetBlend(0)
			ent:DrawModel()
			render.SetBlend(1)
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL)
			cam.Start3D2D(pos, ang, 1)
					surface.SetDrawColor(255, 100, 100)
					surface.DrawRect(-ScrW(), -ScrH(), ScrW() * 2, ScrH() * 2)
			cam.End3D2D()
			ent:SetRenderMode(RENDERMODE_TRANSALPHA)
			ent:SetColor(Color(255, 255, 255, 0))
			ent:DrawModel()
		render.SetStencilEnable(false)
	end
end)

And this is the outcome:
[URL="http://imgur.com/a/bjFsG"]http://imgur.com/a/bjFsG[/URL]
The only reason i dicided to use SetRenderMode, was so i could make the prop invis, but it just draws the full model, and not just only the scentil. Any advice?