Stencil buffer and it's magic [Outline edition]

Today I decided to work on my old outlined spawnicons, and managed to speed up their rendering like 100x times.

Here is the outcome:

After I showed it to some people, I realized that very few people even know about the existence of stencils, and even less who know how to use them.
What is the logical thing to do? Make a tutorial!

So. How did I made this?

#1: What is the stencil buffer?

The stencil buffer is a secondary buffer, next to the color and depth buffers. Basically a huge image that is laid over the screen, or if you prefer,
a big two dimensional array, covering the entire screen. It can be used with rendering operations to control whether a pixel get rendered to the color buffer, or gets discarded.

When the stencil buffer is enabled (render.SetStencilEnable), every pixel that gets rendered to the screen (or to be more precise, the current render target)
will be processed first by your stencil comparison function. You can check out the avalible modes here: Enums/STENCIL

These comparison functions are simple. They compare the value in the stencil buffer where your pixel is, to a reference value you can change with render.SetStencilReferenceValue

If your pixel passes this check, then fine, it gets rendered to the color buffer, and appears on your screen, if it does not, it gets discarded. But there is more.
You can change what to do with the value of the stencil buffer at your pixels location with:


The two most common you will most likely end up using is STENCIL_KEEP and STENCIL_REPLACE (with the reference value). Their names imply what they do.

Now lets take a look at a real world example:

#2: Rendering those icons

I am not going to give you my code, but I will show you the parts that caused the most problem for me:

Image #1:
You will need to initialize the stencil buffer before you use it. As of now, the halo lib causes the most confusion, because it changes
the stencil write/test mask, breaking your code randomly when you hover objects in the world. To fix this you need to change these
masks before doing anything.

render.SetStencilEnable( true )

render.SetStencilWriteMask( 3 ) -- Fix halo lib ( 3 = 0b11 )
render.SetStencilTestMask( 3 )

Image 2-3-4:
In my example, I decided that I want 3 outlines instead of a plain simple one. Two black, and a coloured one. I achieve the outline effect by drawing the same image
over and over, offsetting it a tiny bit. Here comes the first problem. What might seem completely transparent on an image, might not be. Actually when rendering the
image, it does not only render its visible (alpha > 0) pixels into the color buffer, but the invisible (alpha = 0) pixels too! Therefore you can’t use the alpha channel to
create stencil masks. Or could you! Of course! But you need a custom material that does alpha testing for you.

Alpha testing is really simple, it happens before the stencil buffer comparison function. If your alpha value is above a preset value, the pixel passes, if it is not,
it gets discarded, even before triggering a stencil buffer check.

local iconShadow = CreateMaterial( "icon_shadow", "UnlitGeneric", {
	[ "$basetexture" ] = "error",	
	[ "$alphatest" ] = 1,
 } )

iconShadow:SetTexture( "$basetexture", icon:GetTexture( "$basetexture" ) )

Now with this in place, we can write into the stencil buffer!

Image 2:
I render the barrel texture multiple times, with the following stencil state setup:

render.SetStencilCompareFunction( STENCIL_NEVER )

render.SetStencilFailOperation( STENCIL_REPLACE )
render.SetStencilZFailOperation( STENCIL_REPLACE )

-- Render black outline
render.SetStencilReferenceValue( 1 )
renderOutline( 10, 0.1 )

Since I have my compare function set to STENCIL_NEVER, it assures these pixels will never appear on my screen (I set that to STENCIL_ALWAYS when I took the image),
but: when they fail the check, they trigger a STENCIL_REPLACE at every pixel they cover. So I can fill the stencil buffer with ‘1’-s where I want the outer outline to be.

Image 3-4:
I merely repeat this process with different reference values. ‘2’ for the middle outline, and ‘1’ for inner outline. Take a look at the fourth picture. Where you can see black pixels,
that is where my stencil buffer contains ‘1’-s, and where you see the barrel, ‘2’-s.

Now lets use these values.

Image 5:
First I set the stencil buffer ‘read only’ by changing every operation to STENCIL_KEEP, ensuring my stencil buffer values won’t be messed with.

Then I just draw a semi transparent rectangle onto my render target with the following stencil setup:

-- Stencil is read only from now
render.SetStencilPassOperation( STENCIL_KEEP )
render.SetStencilFailOperation( STENCIL_KEEP )
render.SetStencilZFailOperation( STENCIL_KEEP )

-- Paint stripes
render.SetStencilReferenceValue( 2 )
render.SetStencilCompareFunction( STENCIL_NOTEQUAL )

surface.SetDrawColor( 40, 40, 40, 120 )
surface.DrawRect( 0, 0, iconW, iconH )

STENCIL_NOTEQUAL ensures that no pixels will be written where the stencil buffer value is ‘2’. You can clearly see the transparent middle outline on the image.
You can do this the same with textures, and pretty much everything, so I draw a tiled stripe texture there too.

Image 6-7:
At this part I render the black outlines. Just a simple black rectangle, masked by the stencil buffer. Here is the setup:

-- Paint black where outlines are
render.SetStencilReferenceValue( 1 )
render.SetStencilCompareFunction( STENCIL_EQUAL )
surface.SetDrawColor( 0,0,0 )
surface.DrawRect( 0, 0, iconW, iconH )

The lines look a bit jagged, a little blur never hurts:
(I can do this because I rendered everything onto a render target so far. Also
I had to change the comparison function because the textures garry use to blur
the image do not have $alphatest set.)

-- Blur the outlines a bit
render.SetStencilCompareFunction( STENCIL_ALWAYS )

render.BlurRenderTarget( iconRT, 1, 1, 2 )

(Note how the image got darker, this is because the blurring is achieved
by drawing the same image on itself a couple of times)

Image 8-9:
This is the simplest part. Just turn off the stencil buffer, and render the
original image. Done!

As you can see, the image is completely transparent where the middle outline is,
meaning that you can change the outline color of the image free! (So you would
not have to re render the entire image)

Just render a coloured rectangle behind it, like I did.

Nice description
I recommend if you really want to get into stenciling to look into shader intro books, as they go through bunch of other cool opengl things that will help you learn about how everything fits together.