Stencil questions. (A lot of questions)

Garry's Mod coding
#1

Hello. I have been messing around with stencils, and there are still some things that just aren’t clear to me. I made a thread because people might have similar questions, which I couldn’t find the answer to which searching through other old threads. I read through this thread, and admittedly, it raised more questions than it had answered. So I wrote some code based off the tutorials I saw and what I read through. It doesn’t work, but that doesn’t really matter because I don’t know what each line is doing anyways lol. The purpose of the code is just to help me(and hopefully other people?) better understand whats going on. So, I really have more of a theory that I picked up when I went through the tutorials. It’s in the code. If someone could just confirm or deny that I’m doing it right or wrong OR better yet correct me, it would much appreciated. =D

Please be advised: Code below might contain a lot of retardery lol



hook.Add("PostDrawOpaqueRenderables","PlayerBorders",function()

	--[[ What I was trying to do is draw the prop when it dissapears from the players view through the wall. When the prop 
	goes behind the wall, it should turn red or something. ]]

	-- Again, everything below is my interpretaion of what is going on, not what is actually going on. 

	for k, v in pairs( ents.FindByClass("prop_physics")) do

		render.ClearStencil() -- clears stencil data(which I assume is clearing any previously set settings by other addons or even my own stencil render)
		render.SetStencilEnable( true ) -- starts stencil ( in my mind, essentially like declaring a new function )

		render.SetStencilWriteMask( 0-255 ) -- I have no idea what this does.
		render.SetStencilTestMask( 0-255 ) -- or this

		render.SetStencilReferenceValue(1) -- sets a value that you use to compare to other pixel values? I assume this is for drawing, like, layers of stencils?
		-- That sounds retarded, but currently what I think it is.

		render.SetStencilFailOperation(STENCILOPERATION_KEEP) --[[ if the stencil fails for any reason, we keep the current value. What value this is, I have no idea. Is it
		pixel value or the reference value? ]]
		render.SetStencilZFailOperation(STENCILOPERATION_INCR) --[[ If the pixel value is behind something and blocking the players sight then we increase the current value. ]]
		render.SetStencilPassOperation(STENCILOPERATION_KEEP) --[[ If the stencil passes then we keep the current value. Again, don't know what value we are even changing 
		with this. ]]

		--[[ Okay. While my understanding is bad, I think(SetStencilCompareFunction) these work like regular if-statements. The only way to end these
		'if statements' is SetStencilEnable(false) or just calling a different SetStencilCompareFunction. Also, before you call these, you have to have a 
		previously set Reference Value and all SetStencilPassOperation set. ]]
		render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_ALWAYS) -- this one runs always, so here is where you would update stuff every frame if you needed to.
			// Code that runs as a result of the above SetStencilCompareFunction passing.

		-- Only the code directly under this check doesn't run if this check fails. BUT other check still run below this one. 
		render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_GREATER)
			// code that runs as a result of the above SetStencilCompareFunction passing.

		-- this next check isn't needed for what I want(I think), but here just for the sake of questions.
		render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NOTEQUAL) -- This check would still run even if above checks failed.
			// code that runs as a result of the above SetStencilCompareFunction passing.

		render.SetStencilEnable(false) -- stops stencil (the end of our 'function')

	end

end)

So, main questions -

  1. What does a reference value do/What does it represent? I know it’s used for comparisons, but can someone maybe provide an example of how it’s used? Confusing.
  2. What is the pixel value?
  3. Why is, whatever the ‘value’ is, reset to 0 when it goes above 255? A better question is why stop at 255? What does 255 represent?

Sorry for the long post. If anyone has any past experience with stencils, please, drop some knowledge below. :stuck_out_tongue:

#2
  1. easier to explain after point 2 and 3 are understood
  2. The pixel value is a number 0-255 that you can set depending on the comparison operations, increment, decrement, and even some bitwise operations
  3. It’s an overflow. The pixel value is held in 8 bits. Which means that there are only 256 possible positive integers it can give. 255+1=0 and 0-1=255. if instead you had 9 bits, you could make 512 numbers, but again at that point 511+1 = 0

back to 1. the reference value is like a pixel value apart from it’s another number you use to test against the pixel value. with all 3 points understood, you will be able to create a stencil that countains information about what should be draw.

Someone can probably explain this better than me - with the benefit of code examples.

#3

Makes more sense now. So modifying pixel values won’t have any direct effect on what’s being rendered on screen?

#4

An example of a stencil:

[lua]
function ENT:Draw()

local viewent = GetViewEntity()
local pos = ( IsValid( viewent ) and viewent != LocalPlayer() ) and GetViewEntity():GetPos() or EyePos()

if IsInFront( pos, self:GetPos(), self:GetForward() ) then

	render.ClearStencil() --Clear stencil
	render.SetStencilEnable( true ) --Enable stencil
		
		render.SetStencilReferenceValue( 15 )
		render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS )
		render.SetStencilFailOperation( STENCILOPERATION_KEEP )
		render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
		render.SetStencilZFailOperation( STENCILOPERATION_KEEP )
		
		self:DrawModel()
		
		render.SetStencilReferenceValue( 15 ) --Reference value 1
		render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )
		render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
		
		self:DrawToScreen( )
		
	render.SetStencilEnable( false )
	
	self.LastRender = CurTime( )
	
end

end
[/lua]

The first set of stencil function calls are to setup the stencil. This basically means "anything drawn will change the pixel value to 15 (arbitrary). self:DrawModel() then draws, and so basically creates a stencil of the model (imagine taking a picture of the screen, set everything that isn’t the model to black, and the pixels that are the model to white).

The next set of stencil function calls then say that I can only draw to the pixels with value 15 (or in the case of the comparison above, any white pixel). I then call a draw operation defined elsewhere in the code.

This means that the draw operation called could fill the entire screen, but the only parts of it to make it through to being displayed are the parts that satisfy the stencil condition.

#5

I was wondering about that too. Thanks for the example.

So I kinda get what’s going on:



render.ClearStencil() --Clear stencil
render.SetStencilEnable( true ) --Enable stencil
	
	render.SetStencilReferenceValue( 15 ) -- sets arbitrary value
	render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS ) -- if true then (always runs)
	render.SetStencilFailOperation( STENCILOPERATION_KEEP ) -- will always keep pixel value on fail
	render.SetStencilPassOperation( STENCILOPERATION_REPLACE ) -- if you can see the pixel, then replace its value with reference value, which is 15
	render.SetStencilZFailOperation( STENCILOPERATION_KEEP ) -- if is behind something, keep value
	
	self:DrawModel() -- will always draw the model
	
	render.SetStencilReferenceValue( 15 ) --Reference value 1
	render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL ) -- if reference value and pixel value are the same then
	render.SetStencilPassOperation( STENCILOPERATION_REPLACE ) -- switch the values (that makes no sense? wtf)
	
	self:DrawToScreen( ) -- draw the screen
	
render.SetStencilEnable( false )

but I am confused with the second part. Why are you switching the reference var if it is the same as the pixel value? I’m guessing that I am thinking about this the wrong way, shit idk.

#6

[lua]
render.SetStencilPassOperation( STENCILOPERATION_REPLACE ) – switch the values (that makes no sense? wtf)

self:DrawToScreen( ) -- draw the screen

[/lua]

This basically means “If you are drawing to a pixel that passes the comparison function [if the pixel you write to has value 15] then replace that pixel with whatever you are drawing, else do nothing”. In other words, we have our black and white stencil (discussed above where our model is white), this basically makes a hole where the white pixels are, as you would a real life stencil, and you can only draw onto places that are in a hole. Then we call the draw to screen function, where the entire screen is drawn over, except only the pixels inside the hole are actually drawn.

#7

This is making a lot more fucking sense now. :dance:
Just had a curious thought though. How would we go about drawing a model inside the world?
Is that as simple as getting a client side model and then placing it under the world, and then placing a black square down, then doing checks to make sure that the players is looking inside the hole?

#8

http://forum.facepunch.com/showthread.php?t=1569948&p=52514823&viewfull=1#post52514823

[editline]11th August 2017[/editline]

I have no clue how stencils work either so god knows how that works

#9

:mindblown:

Seriously though, thanks. Is there any code out there that I can look at to see how to at least begin making such cool shit?

[editline]11th August 2017[/editline]

Sick. Thanks everyone for taking the time. Not as confusing anymore, and was able to make this:
(It’s garbage but I’m still proud of it)

I’ll leave the thread open in case anyone can find an example for making in-world model type shit. :slight_smile: