Using stencil to render effect only behind things

[sirmasters]

[B][U]Edit:[/U][/B] Solved by the two below, thanks!

[B]-Snipped second edit-[/B]

Edited Working code

function ENT:Draw()
		render.ClearStencil()
		render.SetStencilEnable(true)
			render.SetStencilWriteMask(255)
			render.SetStencilTestMask(255)
			render.SetStencilReferenceValue(15)
			render.SetStencilFailOperation(STENCILOPERATION_KEEP)
			render.SetStencilZFailOperation(STENCILOPERATION_REPLACE )
			render.SetStencilPassOperation(STENCILOPERATION_REPLACE )
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_ALWAYS)
			render.SetBlend(0) --don't visually draw, just stencil
			self:DrawModel()
			render.SetBlend(1)
			render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL)
			cam.Start2D()
					render.OverrideColorWriteEnable(true,true)
					surface.SetDrawColor(255,0,0)
					render.SetColorMaterial()
					surface.DrawTexturedRect(0,0,ScrW(),ScrH())
					render.OverrideColorWriteEnable(false,false)
			cam.End2D()
			self:DrawModel()
		render.SetStencilEnable(false)
end

[B][U]Original Post:[/U][/B]

Hello, I've rarely dealt with stencils (So I cannot figure it out myself at the moment), but I know it's possibly the only solution to my quandary. I wish to render a effect only when it's behind something (world, entities, etc), and the parts of the entity that is seen normally get drawn normal. I do have an example of what I mean ([I]which I half achieved without using stencils[/I], but the effect only happens when behind world, otherwise it is treated normal even behind entities, which is not what I want.[B][U] I want the effect to render when behind anything, no matter if entity or world, but if normally visible, just render as normal[/U][/B]):

[IMG]http://i1320.photobucket.com/albums/u526/sirmastercombat/20160525003726_1_zps5er29nbp.jpg[/IMG]
[IMG]http://i1320.photobucket.com/albums/u526/sirmastercombat/20160525005714_1_text_zpsmuudlpzs.jpg[/IMG]

[bitches]

Here's a tutorial I made on stencils some years back:

[url]https://facepunch.com/showthread.php?t=1337232&p=43271812&viewfull=1#post43271812[/url]

It is very thorough in explaining the steps with examples.

Hint: The ZFailOperation needs to be STENCILOPERATION_REPLACE and the PassOperation needs to be STENCILOPERATION_KEEP and the CompareFunction needs to be STENCILCOMPARISONFUNCTION_ALWAYS

Then draw the desired object [I]invisibly (alpha zero)[/I]. This will mark the screen space containing the player to the ReferenceValue. The space now containing the ReferenceValues (the pixels) are now what you call a Render Mask.

You then set the CompareFunction to STENCILCOMPARISONFUNCTION_EQUAL. This means that the next draw will only affect areas equal to the reference value. The first invisible draw was set to not care what the reference value was, only writing the stencil reference values to areas that are obstructed from view (ZFail).

Don't worry about changing the Fail/Pass comparisons by this point; they only control the changing of pixel reference values, not whether or not the drawing is visible! That's why you had to draw it invisibly while writing to the stencil buffer.

Finally, you simply draw a colored quad over the entire screen. It will only show up where the object is obstructed! See the examples for how to do that.

[editline]25th May 2016[/editline]

Invisibly draw by doing render.SetBlend(0) ent:DrawModel() render.SetBlend(1)

as per the examples in the correct hook

Because this takes place in a 3D hook, you can't call an easy 2D function to render the colored quad over the whole screen. Or at least I thought so at the time? In any case, the example with the multicolored objects shows how to manipulate a 3D hook into drawing a quad over the whole screen by use of 3D2D (draws a colored quad in 3d space clientside infront of the local player's eyes).

[B]final edit has been made; re-read this if you saw this post earlier[/B]

[Whiterabbit]

[QUOTE]Because this takes place in a 3D hook, you can't call an easy 2D function to render the colored quad over the whole screen. Or at least I thought so at the time? In any case, the example with the multicolored objects shows how to manipulate a 3D hook into drawing a quad over the whole screen by use of 3D2D (draws a colored quad in 3d space clientside infront of the local player's eyes).
[/QUOTE]

For that you can just use cam.Start2D, no need for 3d position or angle.

[sirmasters]

[B]-Snipped previous edit-[/B]

Thanks, I got this work, though I had to change to

surface.DrawRect(-ScrW(),-ScrH(),ScrW()*2,ScrH()*2)

to

render.DrawScreenQuad()

for it to do anything, in addition to wh1t3rabbit's suggestion of using cam.Start2D().

[bitches]

Definitely worth playing with stencils more; you can achieve all kinds of interesting effects. For example if you wanted a spherical void made to look like it leads to space, you would not want a textured sphere since it would have depth inverse to the desired; instead you could render a scene that appears far away on the sphere's screen space so to appear like looking at the sphere infront of you means seeing something far away. A void.