How do you use stencils?

[Maurdekye]

As far as I know, the knowledge behind the usage of stencils is cryptic, complex, and known only to few. I've looked everywhere and there doesn't seem to be a definite consensus on how they actually work. Even the tutorial scripts I copied from the old wiki and [URL="http://facepunch.com/showthread.php?t=878986"]this forum thread[/URL] don't work properly. So I'm asking for someone who knows how they work to explain how to use them properly, because as of right now, I don't think such an explanation exists.

[James]

[QUOTE=Maurdekye;47535548]As far as I know, the knowledge behind the usage of stencils is cryptic and known in great detail only to few. I've looked everywhere and there doesn't seem to be a definite consensus on how they actually work. Even the tutorial scripts I copied from the old wiki and [URL="http://facepunch.com/showthread.php?t=878986"]this forum thread[/URL] don't work properly. So I'm asking for someone who knows how they work to explain how to use them properly, because as of now I don't think such an explanation exists.[/QUOTE]

I managed to make the script you were asking for:

[lua]
local mat2 = Material( "tools/toolsblack", "unlitgeneric" )

hook.Add( "PostDrawOpaqueRenderables", "Stencil", function()

local pos = LocalPlayer():EyePos()+LocalPlayer():EyeAngles():Forward()*10
local ang = LocalPlayer():EyeAngles()
ang = Angle(ang.p+90,ang.y,0)

render.ClearStencil() --Clear stencil
render.SetStencilEnable( true ) --Enable stencil

render.SetStencilWriteMask(255)
render.SetStencilTestMask(255)
render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS )
render.SetStencilFailOperation( STENCILOPERATION_KEEP )
render.SetStencilPassOperation( STENCILOPERATION_KEEP)
render.SetStencilZFailOperation( STENCILOPERATION_REPLACE )
render.SetStencilReferenceValue(15)

render.SetBlend(0)
for _, pl in pairs(player.GetAll()) do
pl:DrawModel()
end
render.SetBlend(1)

render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )

cam.Start3D2D(pos,ang,1)
render.SetMaterial( mat2 )
render.DrawScreenQuad()
cam.End3D2D()

for _, pl in pairs(player.GetAll()) do
pl:DrawModel()
end

render.SetStencilEnable( false )
end)
[/lua]

the trick was drawing the playermodel after drawing the stencil behind the model in a cam3D2D

[Maurdekye]

Well, I suppose that's useful, but you're probably better off putting that in my other thread; i'll unmark it as solved. This is a more generic question that i'm asking so that I can hopefully have a better understanding of how stencils work; I can't just come to you every time I need new code.

[James]

[QUOTE=Maurdekye;47535587]Well, I suppose that's useful, but you're probably better off putting that in my other thread; i'll unmark it as solved. This is a more generic question that i'm asking so that I can hopefully have a better understanding of how stencils work; I can't just come to you every time I need new code.[/QUOTE]

Code_gs pointed me to [URL="http://facepunch.com/showthread.php?t=1337232"]this thread[/URL] which helped me understand what to do, might be of some help to you aswell. It's no tutorial, but it teaches you things.

[Maurdekye]

So let me get this straight.

- The stencil buffer is a matrix containing grid spaces of every pixel, which each have values from 0-255
- The default value for each pixel on the grid is 0
- While the stencil is active, the pixels of everything that is explicitly rendered goes through a comparison process
---- in this process, if a pixel is obstructed from view, then it runs the operation defined by SetStencilZFailOperation
---- if it is unobstructed but fails the comparison test, then it runs the operation defined by SetStencilFailOperation
---- otherwise it runs the operation defined by SetStencilPassOperation
---- its value on the buffer is then changed as a result of the operation that is run
- you should always use the ALWAYS comparison operator on the first render pass so that a default value is assigned to newly rendered items
- when anything is rendered while it is active, the comparison operation and reference value mask the rendered result to pixels in the buffer that satisfy the comparison

Is that right? Is it even close?

[SaintSin6]

This may be another example to help, I remember trying to figure out stencils and going wtf..
[url]http://facepunch.com/showthread.php?t=1436776[/url]

[Maurdekye]

Thank you all, this is very helpful. But I still don't understand why I need to use cam.Start2D3D. Otherwise, this is starting to make a lot more sense.

[!cake]

It is 3am, the perfect time to be making diagrams.
[img]https://dl.dropboxusercontent.com/u/7290193/Screenshots/Garrysmod/Facepunch/__Acecool_Dev_Base/________Tutorials__/___.png[/img]

Disclaimer: Not guaranteed to be correct. Especially the parts about the test and write masks.

[Maurdekye]

That is incredibly complex, and I don't mean to demean you, but it's a bit too complicated for me to understand as of yet. But I'll definitely look back to it as often as I can whenever I work with stencils. It is useful to know that the comparison test is considered before the depth test; I thought it was the other way around.

[James]

[QUOTE=Maurdekye;47535844]Thank you all, this is very helpful. But I still don't understand why I need to use cam.Start2D3D. Otherwise, this is starting to make a lot more sense.[/QUOTE]

[del]I am using the cam.Start3D2D so I can draw the stencil behind the model that's being drawn.[/del] Right answer below

[Maurdekye]

[QUOTE=James xX;47538110]I am using the cam.Start3D2D so I can draw the stencil behind the model that's being drawn.[/QUOTE]

Why is it necessary? Can you not redraw the model after drawing the quad without the cam.2d3d?

[James]

[QUOTE=Maurdekye;47539721]Why is it necessary? Can you not redraw the model after drawing the quad without the cam.2d3d?[/QUOTE]

Sorry I messed up my answer, the surface library doesn't work as it should in the hook I was using, a quick fix is to shove it inside a cam3D or cam3D2D. If you look at the positions at the top of the file, it's right in front of the player's eyes. I did redraw the model after drawing the quad.

[Maurdekye]

[QUOTE=James xX;47539742]Sorry I messed up my answer, the surface library doesn't work as it should in the hook I was using, a quick fix is to shove it inside a cam3D or cam3D2D. If you look at the positions at the top of the file, it's right in front of the player's eyes. I did redraw the model after drawing the quad.[/QUOTE]

Is there a different hook where it would work?

[James]

[QUOTE=Maurdekye;47539820]Is there a different hook where it would work?[/QUOTE]

Any that draws.

[Maurdekye]

[QUOTE=James xX;47539831]Any that draws.[/QUOTE]

So why not use a different hook rather than using a workaround?

[James]

[QUOTE=Maurdekye;47539843]So why not use a different hook rather than using a workaround?[/QUOTE]

Because different hooks get called at different times. You can't draw something behind a wall in DrawHUD for example.

[Maurdekye]

[QUOTE=James xX;47539865]Because different hooks get called at different times. You can't draw something behind a wall in DrawHUD for example.[/QUOTE]

So this is the only hook that allows you to use the stencils properly in this context, but it is also the one that requires a workaround? Does it need this workaround because of a bug, or because of how stencils work?

[James]

[QUOTE=Maurdekye;47539875]So this is the only hook that allows you to use the stencils properly in this context, but it is also the one that requires a workaround? Does it need this workaround because of a bug, or because of how stencils work?[/QUOTE]

Because of where you are drawing it. You shouldn't be drawing a 2D object in a 3D world, so we add it into a cam.Start3D2D