Draw entity black with stencils

[Pugsworth]

Trying to learn stencils; I want to draw a model completely solid black using stencils. Using my current stencil knowledge, this is what I thought should work, but it just draws the screen black.

[lua]
function ENT:Draw()
render.ClearStencil()
render.SetStencilEnable(true)

render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NEVER)
render.SetStencilPassOperation(STENCILOPERATION_INCR) -- increase to 1 if it's visible
render.SetStencilFailOperation(STENCILOPERATION_KEEP) -- keep at zero
render.SetStencilZFailOperation(STENCILOPERATION_KEEP) -- keep at zero

self:DrawModel()

render.SetStencilReferenceValue( 1 )
render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL) -- only draw if it's equal to 1

render.SetMaterial(matBlack)
render.DrawScreenQuad()

render.SetStencilEnable(false)
end
[/lua]

[Drakehawke]

So I see I'm not the only one who tried to play around with stencils after that WAYWO post earlier

[lua]render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NEVER)[/lua]

Doesn't this mean the comparison function will always fail? Shouldn't you be using STENCILCOMPARISONFUNCTION_ALWAYS?

[Pugsworth]

[QUOTE=Drakehawke;40139360]So I see I'm not the only one who tried to play around with stencils after that WAYWO post earlier

[lua]render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NEVER)[/lua]

Doesn't this mean the comparison function will always fail? Shouldn't you be using STENCILCOMPARISONFUNCTION_ALWAYS?[/QUOTE]
:v: I kept wanting to learn stencils, I acted apon that want this time, finally.

I thought so too, but ALWAYS does the same.

[Map in a box]

Used this for my HUD, if you want some stencil ref:

[lua]
render.SetStencilEnable(true)

render.SetStencilReferenceValue(1)
render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
--render.SetStencilFailOperation( STENCILOPERATION_ZERO )
render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS )

surface.SetDrawColor(0,0,0,126)
surface.DrawPoly(poly)

render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )
render.SetStencilFailOperation(STENCILOPERATION_ZERO)

surface.SetTextPos(poly[4].x,poly[4].y+2)
surface.SetTextColor(255,255,255)

surface.DrawText(obbyTxt)

render.SetStencilEnable(false)
surface.SetDrawColor(0,0,0,126)
[/lua]

It draws the text inside the poly only.

[Drakehawke]

I've been trying out examples from old stencil tutorials and also tried some from addons that used them. None of them seem to work anymore.

Garry uses them in halos, looking at how he's using them might help [url]https://github.com/garrynewman/garrysmod/blob/master/garrysmod/lua/includes/modules/halo.lua[/url]

[Pugsworth]

[QUOTE=Map in a box;40139699]Used this for my HUD, if you want some stencil ref:

[lua]
render.SetStencilEnable(true)

render.SetStencilReferenceValue(1)
render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
--render.SetStencilFailOperation( STENCILOPERATION_ZERO )
render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS )

surface.SetDrawColor(0,0,0,126)
surface.DrawPoly(poly)

render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )
render.SetStencilFailOperation(STENCILOPERATION_ZERO)

surface.SetTextPos(poly[4].x,poly[4].y+2)
surface.SetTextColor(255,255,255)

surface.DrawText(obbyTxt)

render.SetStencilEnable(false)
surface.SetDrawColor(0,0,0,126)
[/lua]

It draws the text inside the poly only.[/QUOTE]

I think the thing that confuses me the most about stencils is, what does the top block of stencil operations do?

From what I can tell, it will always draw the poly, and have a pixel value of 1.

But then how does it tell the text from the poly if the text wasn't set a value?

[Drakehawke]

Got it!

It's garrys fault, in the halo code he sets the TestMask and WriteMask to 0 at the end, when the default is actually 1. Thanks to Tobba who pointed it out here: [url]http://facepunch.com/showthread.php?t=1220543&p=38210090&viewfull=1#post38210090[/url]

but it was never fixed.

Add

[lua]render.SetStencilWriteMask( 1 )
render.SetStencilTestMask( 1 )[/lua]

to the start and it should work.

[Pugsworth]

[QUOTE=Drakehawke;40139852]Got it!

It's garrys fault, in the halo code he sets the TestMask and WriteMask to 0 at the end, when the default is actually 1. Thanks to Tobba who pointed it out here: [url]http://facepunch.com/showthread.php?t=1220543&p=38210090&viewfull=1#post38210090[/url]

but it was never fixed.

Add

[lua]render.SetStencilWriteMask( 1 )
render.SetStencilTestMask( 1 )[/lua]

to the start and it should work.[/QUOTE]

GARRY :v:

So my code does work how I thought it should.

Thanks for all the help.

[editline]a[/editline]

Actually, I have two more questions:

If the first comparison function is set to ALWAYS, the model renders with the black. When set to NEVER nothing draws, but when set to GREATER, only the black renders.
I don't fully understand why.

And how can I make the stencil transparent.

[Drakehawke]

Comparison function set to ALWAYS means the pass operation will always be called, so every pixel drawn next will have its value incremented to 1. You then draw the model, so all the pixels of the model are now at value 1. You then draw all pixels of value 1 as black.

Comparison function set to NEVER means the fail operation is always called, so each pixel of the model will remain at zero, so no pixels are drawn as black.

Comparison function set to GREATER means that the pass operation is called if the value is greater than the reference value - which you haven't set. If the prop is drawing black that means the default pixel value 0 must be greater than the reference value. I'm guessing because you haven't set it to anything the default must be -1 or something.

To control alpha use render.SetBlend( 0 - 1 )

[editline]3rd April 2013[/editline]

Also here's a useful thread I found, there are more examples on the 2nd page: [url]http://facepunch.com/showthread.php?t=878986[/url]

And an old wiki tutorial [url]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/indexd0aa.html[/url]

[Pugsworth]

[QUOTE=Drakehawke;40140118]Comparison function set to ALWAYS means the pass operation will always be called, so every pixel drawn next will have its value incremented to 1. You then draw the model, so all the pixels of the model are now at value 1. You then draw all pixels of value 1 as black.

Comparison function set to NEVER means the fail operation is always called, so each pixel of the model will remain at zero, so no pixels are drawn as black.

Comparison function set to GREATER means that the pass operation is called if the value is greater than the reference value - which you haven't set. If the prop is drawing black that means the default pixel value 0 must be greater than the reference value. I'm guessing because you haven't set it to anything the default must be -1 or something.

To control alpha use render.SetBlend( 0 - 1 )

[editline]3rd April 2013[/editline]

Also here's a useful thread I found, there are more examples on the 2nd page: [url]http://facepunch.com/showthread.php?t=878986[/url]

And an old wiki tutorial [url]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/indexd0aa.html[/url][/QUOTE]

Ah, that makes sense. So setting it to NOTEQUAL would be the best.

I tried the render.SetBlend, but nothing appeared to have happened.

[Map in a box]

Halo also creates lots of tables every time it's used. Same for cam.Start*

[Pugsworth]

Can you even have stencils with transparency?
I can't find any way to make it work.

[Bletotum]

SetBlend works fine for me. Perhaps it only blends certain drawing?

[Pugsworth]

Well, with my current code:

[lua]function ENT:Draw()

render.ClearStencil()
render.SetStencilEnable(true)

render.SetStencilWriteMask(1) -- Thanks garry
render.SetStencilTestMask(1) --

render.SetStencilReferenceValue(-1)
render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NOTEQUAL) -- Don't draw the model
render.SetStencilPassOperation(STENCILOPERATION_INCR) -- increase to 1 if it's visible
render.SetStencilFailOperation(STENCILOPERATION_KEEP) -- keep at zero if not
render.SetStencilZFailOperation(STENCILOPERATION_KEEP) -- keep at zero if not

self:DrawModel()

render.SetStencilReferenceValue(1)
render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL) -- only draw if it's equal to 1

render.SetMaterial(matBlack)

render.SetBlend(0.1)
render.DrawScreenQuad()
render.SetBlend(1.0)

render.SetStencilEnable(false)
end[/lua]

It seems to not do anything.