Time For You to Understand Stencils!

The little hamster that runs the programming section of my brain has always cowered in terror when it caught wift of stencils. So this little write-up has been a step by step way to put him back to work. In the event you have a similar crisis going on, this might be helpful. After this guide it might be beneficial to visit ralle’s stencil tutorial / this old thread.


What is a stencil?

A stencil is basically a pixel that has a value associated with it. By manipulating these pixel values we can re-render parts of the screen in unique ways to create awesome effects. For example, popular uses include: making shadows, mirrors, portals, model outlines, “wall-hacks”, and so forth. So, ultimately, stencils are just another part of the long chain of graphics processing, like the depth test that determines what objects should be drawn on our screen.

Background

Stencil Buffer: a 2d array of unsigned bytes (can be 0 - 255) for each pixel on your screen. Each one of these pixel values are refered to as a value and are 0 by default.

Reference Value: this is a global unsigned byte that you set and use to compare a value with (in some creative way).

Stencil test: a per-frame comparison where we compare the pixel value against our global reference with some operator (<, <=, >, >=, ==, !=). Based on whether our test succeeds we will perform certain operations on that pixel value.

The Full Stencil Test

Every frame each pixel will be tested, and fall into one of these categories.

  • Pass: the pixel is visible to the viewer and passed our comparison test.
  • Fail: the pixel is visible to the viewer but failed our comparison test.
  • ZFail: the pixel is not visible (it is behind something else).

So essentially, every pixel runs this code.




if pixel:isVisible() then
    if ComparisonFunction then
        pixel:RunPassOperation()
    else
        pixel:RunFailOperation()
    end
else
    pixel:RunZFailOperation()
end

-- Once we are ready, we call render.DrawScreenQuad() to draw all of the pixels that made it to the pass operation.  It's that easy!



The tests and resulting operations are chosen with the follow methods in Garry’s Mod.



render.SetStencilCompareFunction( STENCIL_NEVER )
render.SetStencilPassOperation( STENCIL_KEEP )
render.SetStencilFailOperation( STENCIL_KEEP )
render.SetStencilZFailOperation( STENCIL_KEEP )

And here is a list of the enums used with these methods.

Comparison Types
( our test passed if… )



STENCIL_LESS -- reference < value
STENCIL_LESSEQUAL -- reference <= value
STENCIL_GREATER -- reference > value
STENCIL_GREATEREQUAL -- reference >= value

STENCIL_NOTEQUAL -- reference != value
STENCIL_EQUAL -- reference == value

STENCIL_NEVER -- false
STENCIL_ALWAYS -- true

STENCIL_NEVER is the default value of the comparison function.

Operations



STENCIL_DECR -- value--, wraps from 0 to 255
STENCIL_INCR -- value++, wraps from 255 to 0

STENCIL_DECRSAT -- value--, clamped to 0
STENCIL_INCRSAT -- value++, clamped to 255

STENCIL_ZERO -- value = 0
STENCIL_REPLACE -- value = reference
STENCIL_KEEP -- does nothing


STENCIL_KEEP is the default operation for every test.

Last Mentions

I lied a bit, there’s another global besides reference that you can set. It’s called the test mask. When mask is set to the value 255 our comparison test simplifies to:

otherwise, it’s actually has a bit-wise and.

But, let’s be honest, that’s confusing.


Let’s Make A Wallhack!

Add this to a lua/autorun/client script and you will get something along the lines of…

http://s10.postimg.org/75d4nr8ux/gmod_wall_hack.jpg


local propShadow = CreateMaterial( "our_black_material", "UnlitGeneric", {
	[ "$basetexture" ] = "models/debug/debugwhite",
        [ "$color" ] = "{ 0 255 0 }"
})

hook.Add( "PostDrawOpaqueRenderables", "DrawingPropsBehindWallsHook", function()
	render.ClearStencil() -- Zero out all pixel *values* just in case ~glares at Garry~
	render.SetStencilEnable( true ) -- Yes, it's required.
	
		-- Garry's halo effect sets the test *mask* to 0, so we have to put it back to 255
                -- or else value and reference would always be set to 0.  And, WriteMask performs a 
                -- bit-wise and with any values written to the stencil buffer, so likewise this
                -- could also be a source of sabotage that we should try to prevent.
		render.SetStencilWriteMask( 255 )
		render.SetStencilTestMask( 255 )
		
		-- Set some arbitrary reference value.
		render.SetStencilReferenceValue( 57 )
		
                -- We want prop models to trigger ZFail instead of Fail.
                render.SetStencilCompareFunction( STENCIL_ALWAYS )

		-- Pixels drawn that can not be seen will call 'value = 57'.
		render.SetStencilZFailOperation( STENCIL_REPLACE )
		
                -- Redraw the props so that all hidden ones call 'value = 57'.
                -- At this point the world will still draw on top of the props
                -- (hence why the ZFail test is triggered).
		for key, prop in pairs(ents.FindByClass( "prop_physics" )) do
			prop:DrawModel()
		end

		-- Only pixels that have 'value == 57' on the next draw call will be drawn.
                -- Keep in mind the rest of the visible pixels still have a value of 0.
		render.SetStencilCompareFunction(STENCIL_EQUAL)

		-- Objects drawn will use this material so source doesn't freak out
		render.SetMaterial(propShadow)
		
		-- Redraw all pixels that pass our comparison test
		render.DrawScreenQuad()
		
	render.SetStencilEnable( false )
end)

And A Big Hole!

http://s8.postimg.org/9jyptrhk5/gmod_stencils.jpg



hook.Add( "PostDrawOpaqueRenderables", "example", function()
    -- This is where and how our 'cut' will be rendered.
	local pos = Vector(73.877281, 384.037689, -147.95)
	local angle = Angle(0, 0, 0) -- Flat with the ground.
    local scale = 200 -- 1px in 2d = 200 units in 3d.
        
    render.SetStencilEnable( true )
		render.SetStencilWriteMask( 255 )
		render.SetStencilTestMask( 255 )
        
        -------------------------------------------------------
        -- Part 1: Draw the cut in the world
        -------------------------------------------------------

        render.SetStencilReferenceValue( 57 )
        render.SetStencilCompareFunction( STENCIL_ALWAYS )
        render.SetStencilPassOperation( STENCIL_REPLACE )

        cam.Start3D2D( pos, angle, scale )
            surface.SetDrawColor( Color( 0, 0, 0, 255 ) )
            surface.DrawRect( 0, 0, 1, 1 ) -- a 1 x 1 square
        cam.End3D2D()
        
        -------------------------------------------------------
        -- Part 2: Draw items in the cut (even if they're in the world)
        -------------------------------------------------------

        render.SetStencilCompareFunction( STENCIL_EQUAL )
        cam.IgnoreZ( true ) -- see objects through world 
            for key, prop in pairs(ents.FindByClass( "prop_physics" )) do
                prop:DrawModel()
            end
        cam.IgnoreZ( false )

        -------------------------------------------------------
        -- Part 3: Redraw the view model on top of the items in the cut
        -- to counter the IgnoreZ.  Drawing view models is a touchy process.
        -------------------------------------------------------

        local fov = LocalPlayer():GetActiveWeapon().ViewModelFOV or (LocalPlayer():GetFOV() - 21.5)
        cam.Start3D( EyePos(), EyeAngles(), fov + 15)
        cam.IgnoreZ( true )
            LocalPlayer():GetViewModel():DrawModel()
        cam.IgnoreZ( false )
        cam.End3D()
        
    render.SetStencilEnable( false )
end )


Awesome, this cleared quite a few things up for me, great tutorial.

Sorry for any formatting issues. Facepunch’s editor is pretty uh… special. And likes to delete your work when it does it’s verification checks.

Great guide!
I’ve always wondered how you “cut in the world” with stencils. So if you ever want to add to the guide, an example of how to do that would be great.

Wow.
Amazing tutorial.
Ive also heard that thid works like a cutting thing for hud elements, is that true?

Do you have any pictures showing what “cut in the world” would looks like? I am not 100% certain what you have in mind.

I am also not quite sure what you mean by cutting thing here :). But, since you can render on top of any pixel on the screen you can create any hud effect you want with enough thought. Though, for most huds using Garry’s surface, draw, and 3dcam libraries might be a bit easier than resorting to stencils.

Nice tutorial people really needed this!
Great Job!

He probably meant something like this

I heard people using it as a element limitanting effect, as making circles without the DrawPoly func or something sort of.
How can you make that?

Wasn’t talking about what you said.

Ups, wrong reply


local function DrawTyrone()

	draw.NoTexture()
	surface.SetDrawColor(Color(255, 255, 255, 255))
	surface.SetMaterial(mat)
	surface.DrawTexturedRect(0, 0, 184, 184)

end

hook.Add("HUDPaint", "HUDStencil", function()

	render.ClearStencil()
	render.SetStencilEnable(true)

		render.SetStencilWriteMask( 255 )
		render.SetStencilTestMask( 255 )

		render.SetStencilReferenceValue( 25 )
		render.SetStencilFailOperation( STENCIL_REPLACE )

		DrawFilledCircle(92, 92, 92, Color(255, 255, 255))

		render.SetStencilCompareFunction(STENCIL_EQUAL)

		DrawTyrone()

	render.SetStencilEnable(false)

end)

Is that helpful? Just wrote it up should work fine.

Whats DrawFilledCircle?
Thank you anyways

Perhaps he meant this, just in pseudocode. Oh, making a cut in the world should be fairly straightforward, i’ll try to add an example later this evening.

Just a custom function for drawing a circle with surface.DrawPoly. If you do some math it shouldn’t take long.

This is BUUTUTTUTFLIYUYUY

Report and move guys…Report and move

Care to elaborate?

He was referring to the post made by a person who is now banned. When he was perm banned all of his posts were removed. The post he was referring to was an image don’t recall exactly what it was but it was just spam.

I’m looking to do something quite similar to the hole in the ground but I need it to only draw a model in the hole and nowhere else. It’s for a portal-like effect. Is there an easy way to achieve this?