Getting a stencil model to render between two conditions

Garry's Mod coding
#1

I apologize if this thread is a bit large, but I’m just trying to get a grip on stencils, so I’m spitballing as much information as I can to make sure I’m heading in the right direction.

Essentially, I’m trying to recreate this effect but with stencils. The original method physically moves the player’s position under the floor, which could be quite problematic, and if the player spawns on a floor (which is the ceiling for the room below it), their model would be visible from below. As a result, I took to giving a shot at stencils to attempt to recreate this effect, but without actually moving the player’s model. The CalcView and CalcViewModelView hooks are all finished, as well as making the players invisible while the view is still shifting.Now the last step is to get the stencils to render properly, and the part I’m having trouble with.

So I start by enabling the stencil, setting the write and test mask to 255 (to avoid any problems with other stencils screwing with my ones), and I give my stencil a reference value of 1. Next I want to generate a cube with Hell in it (little easter egg so that players who can look down quickly will see), so I start by setting my compare function to STENCIL_ALWAYS and the pass operation to STENCIL_REPLACE, as I want my hole stencil to always replace the pixels that are on the floor of the map with the reference value 1, and I finally draw the hole using a 3D2D cam. Now I want to create a small cube with hell in it, so I set my compare function to STENCIL_EQUAL, as I only want my hell to render INSIDE the hole. Afterwards I render multiple boxes which are all given seperate textures to create a makeshift hell. All is good from here.

Something like this:
[img_thumb]http://www.kyle93.co.uk/i/b9fd4.png[/img_thumb]

Now this is the part where I’m kind of lost. Essentially, I want to render a model of the player’s model, crawling it’s way out of hell, but to only be visible both outside and inside my hole. I don’t want the model to be visible below the floor where the player spawns, but at the same time be visible inside the hole. I can’t seem to get both of these conditions to work together. Essentially, something like this:
[img_thumb]http://www.kyle93.co.uk/i/a25cb.png[/img_thumb]

Any ideas? Getting this to work with multiple conditions seems to be still a mystery to me.

Here’s the code I have so far. I am drawing the stencil on top of the sawblade model for testing purposes
[lua]
/-----------------------------------------------------
Custom render.Model func
-----------------------------------------------------/

function render.ModelCustom( mdl, pos, ang, animation, animationspeed, scale )

local ent = ClientsideModel( mdl, RENDERGROUP_OTHER )

if ( !IsValid( ent ) ) then return end

ent:SetModel( mdl )
ent:SetNoDraw( true )

ent:SetPos( pos )
ent:SetAngles( ang )
ent:ResetSequence( ent:LookupSequence( animation ) )
ent:SetModelScale(scale)
ent:SetCycle( (CurTime()-math.floor(CurTime()))*animationspeed )
ent:DrawModel()

ent:Remove()

end

/-----------------------------------------------------
Stencil
-----------------------------------------------------/

local comp = 1
local pass = 1
local fail = 1
local zfail = 1

hook.Add( “PostDrawOpaqueRenderables”, “SpookyPlayerSpawningStencils”, function()
for k, v in pairs( ents.GetAll() ) do

	if v:GetModel() == "models/props_junk/sawblade001a.mdl" then//if v:GetNWFloat("DoSpawnAnim") > CurTime() then
		local angle = Angle(0, 0, 0)
		local scale = 48
		local pos = v:GetPos()-Vector(scale/2,-scale/2,0)
			
		render.SetStencilEnable( true )
			render.SetStencilWriteMask( 255 ) 
			render.SetStencilTestMask( 255 )
			
			/*-----------------------------------------------------
						Make the hole
			-----------------------------------------------------*/

			render.SetStencilReferenceValue( 1 )
			render.SetStencilCompareFunction( STENCIL_ALWAYS )
			render.SetStencilPassOperation( STENCIL_REPLACE )

			cam.Start3D2D( pos, angle, scale )
				surface.SetDrawColor( Color( 0, 0, 0, 255 ) )
				surface.DrawRect( 0, 0, 1, 1 )
			cam.End3D2D()
			
			/*-----------------------------------------------------
					 	 Draw Hell
			-----------------------------------------------------*/

			render.SetStencilCompareFunction( STENCIL_EQUAL )
			cam.IgnoreZ( true )
				render.SetMaterial(Material("models/misc/cube_left.png"))
				render.DrawBox( pos,angle,Vector(-1,-scale,-scale),Vector(0,0,0),Color(255,255,255,255),true)
				render.SetMaterial(Material("models/misc/cube_right.png"))
				render.DrawBox( pos,angle,Vector(scale,-scale,-scale),Vector(scale+1,0,0),Color(255,255,255,255),true)
				render.SetMaterial(Material("models/misc/cube_center.png"))
				render.DrawBox( pos,angle,Vector(0,0,-scale),Vector(scale,1,0),Color(255,255,255,255),true)
				render.SetMaterial(Material("models/misc/cube_back.png"))
				render.DrawBox( pos,angle,Vector(0,-scale-1,-scale),Vector(scale,-scale,0),Color(255,255,255,255),true)
				render.SetMaterial(Material("models/misc/cube_bot.png"))
				render.DrawBox( pos,angle,Vector(0,-scale,-scale),Vector(scale,0,-scale+1),Color(255,255,255,255),true)
			cam.IgnoreZ( false )

			/*-----------------------------------------------------
					Draw player crawling out of hell
			-----------------------------------------------------*/				
			
			render.SetStencilCompareFunction( STENCIL_GREATEREQUAL )
			render.SetStencilPassOperation( STENCIL_REPLACE )
			render.SetStencilFailOperation( STENCIL_DECR )
			render.SetStencilZFailOperation( STENCIL_ZERO  )
				//if GetViewEntity() != LocalPlayer() then // Don't draw the model for the player spawning.
					render.ModelCustom("models/Kleiner.mdl",pos+Vector(scale/2,-scale/2,-25),angle,"ragdoll",1, 1)
				//end

		render.SetStencilEnable( false )
	end
end

end )
[/lua]

I’m certain it has something to do with the increment functions but I’m just not quite putting my finger on it.
This code is ALMOST there but it doesn’t render the model if it’s too close to the floor(image 3) and it also leaves some gaps in the model due to the rendering of the hole(image 2). It also creates some weird artifacts if I put my mouse looking at where the hole is supposed to be(image 4).
[lua]
render.SetStencilCompareFunction( STENCIL_GREATEREQUAL )
render.SetStencilPassOperation( STENCIL_REPLACE )
render.SetStencilFailOperation( STENCIL_DECR )
render.SetStencilZFailOperation( STENCIL_ZERO )
[/lua]

#2

Well, I managed to get the model rendering close to the floor using ignorez and I did fix the gap issue by making my boxes 0 pixels thick, but the artifacts issue is still there, and the model renders behind walls, which I don’t want.

#3

Oh boy, this is a tricky one… I haven’t really tried, but by first hunch would be to clear the Z-buffer where you are supposed to look “inside” the cube using a stencil function, then render your cube and player model normally.

You can clear both the color, and depth buffer at a specific location using

render.ClearBuffersObeyStencil

Since you cleared the depth buffer at only that position, it will look like a window inside your cube, while still allowing you to render the players model.

#4

Gave it a shot but no luck. I got rid of the cam.IgnoreZ and I’m still in the same situation, but this time instead of the second stencil rendering transparent, it renders in the color I set it to (in this case white) with no change in alpha. (The one on the right)

1600×900 515 KB

#5

You should show your code so we can see how it works. Are you still following the advice I gave you in the problems thread? http://forum.facepunch.com/showthread.php?t=1548067&p=52857843&viewfull=1#post52857843

#6

I am pretty sure you forgot to clear the depth buffer. Regardless, I gave a shot at it, and managed to implement the desired effect:

While at it, it seems that people still don’t quite understand what the stencil buffer is for, and how it works in detail, so I added quite a lot of comments to the code for future generations.

[lua]

local MODEL = “models/props_junk/sawblade001a.mdl”
local PLASTIC = Material(“hunter/myplastic”)

local function findTargetEntity()
for _, ent in ipairs(ents.GetAll()) do
if ent:GetModel() == MODEL then
return ent
end
end
end

–[[
– Altough the default render library does not provide such a function, I always use something like this
– with my projects involving the stencil buffer.

– This function essentially makes sure you are not forgetting to change the stencil operations, so other
– passes will not accidentally “bleed” into your current workflow.

– Also it is nice to make sure you are not overwriting your stencil buffer accidentally.
–]]
function render.SetStencilOperator(compare, pass, fail, zfail)
render.SetStencilCompareFunction(compare)

render.SetStencilPassOperation(pass           or STENCIL_KEEP)
render.SetStencilFailOperation(fail           or STENCIL_KEEP)
render.SetStencilZFailOperation(zfail or fail or STENCIL_KEEP)

end

–[[
– A good hook to use for our rendering is PreDrawOpaqueRenderables, as it is called after
– opaque world geometry has been already drawn, but before any props or entities are rendered.

– Therefore if we modify the depth buffer here, we can make the world geometry seetrough!

– (For a complete list of rendering hooks, and their order take a look at the wiki: wiki.garrysmod.com/page/Render_Order)
–]]
hook.Add(“PreDrawOpaqueRenderables”, “StencilPlayground”, function()
local target = findTargetEntity()

-- Do not attempt without target
if not target then return end

local pos = target:GetPos()
local ang = target:GetAngles()

-- Enable stencil operations
render.SetStencilEnable(true)

	--[[
	--	It is always good practice to clear the buffer if you are using it for the first time, who knows
	--	what odd values are there already...
	--	
	--	Another recommended thing is to restore the stencil write/test mask to the defaults, as the halo
	--	library leaves these mangled. This could cause issues when a halo is being drawn before your code.
	--]]
	render.ClearStencil()
	render.SetStencilTestMask(255)
	render.SetStencilWriteMask(255)
	
	--[[
	--	Set the reference value to whatever you want, the buffer is initialized with all zeros right now.
	--	
	--	Set the stencil operation to always pass, and make the pass operation 'REPLACE'. This means that the
	--	pixels we are about to draw (that pass the zbuffer/depth test) will replace the value in the stencil
	--	buffer with our reference value. This way our window will not draw over world geometry it is not
	--	supposed to. (Opaque world geometry has been already drawn into the buffer)
	--	
	--	A small caveat of this is that we are 'tainting' the render buffer with our window. That is no problem
	--	however, as we wish to override those pixels with 'hell' anyway.
	--]]
	render.SetStencilReferenceValue(1)
	render.SetStencilOperator(STENCIL_ALWAYS, STENCIL_REPLACE)
	
	-- Render our 'window into the underworld'
	do
		local ext = Vector(64, 64, 0.1)
		local off = Vector()

		render.SetColorMaterial()
		render.DrawBox(pos + off, ang, -ext, ext, Color(0, 255, 0), true)
	end
	
	--[[
	--	At this point our stencil buffer is all zeros, except the places where the window was drawn, and
	--	passed the depth test, therefore is visible to the player (not occluded by the world)-
	--	
	--	Change the stencil comparison function to only pass if the value in the stencil buffer is equal to
	--	the reference value set earlier. This will ensure that 'hell' will be only visible trough our window.
	--
	--	(My utility function SetStencilOperator defaults all operations to keep the value in the stencil buffer here!)
	--	
	--	We can also use this test to clear parts of the original buffer. ClearBuffersObeyStencil (as the name implies)
	--	clears the currently bound color buffer, but leaves the values intact that do NOT pass the stencil test.
	--]]
	render.SetStencilReferenceValue(1)
	render.SetStencilOperator(STENCIL_EQUAL)

	render.ClearBuffersObeyStencil(0, 0, 0, 255, true)

	--[[
	--	Now that the depth buffer has been cleared, previously drawn world geometry will not occlude
	--	anything that we draw there.
	--	
	--	Also, our stencil function is still active, so if we draw our 'hell' geometry here,
	--	only the parts visible trough the window will be drawn onto the screen buffer.
	--]]
	do
		local ext = -Vector(1,1,1) * 64

		render.SuppressEngineLighting(true)
			render.SetMaterial(PLASTIC)
			render.DrawBox(pos, ang, -ext, ext, Color(255, 180, 180), true)
		render.SuppressEngineLighting(false)
	end

	-- Now that everything is all done, be nice to the next coder and clear the stencil buffer, just in case
	render.ClearStencil()

render.SetStencilEnable(false)

end)
[/lua]

#7

That’s really insightful, thanks a TON!

Though I’m still getting a minor visual issue, and if this didn’t affect you then I’m certain it’s something on my end, but the inside of the stencil sometimes flashes between having the texture or black. Not really the entire thing, sometimes just certain faces of the box.

#8

That happened for me aswell, but I assumed it was because I used the PHX plastic material on an inverted cube. It should not happen with your skybox like material.

#9

Yeah giving it a shot with my material works perfectly fine. Thank you ^^