Only render a player if they are behind a wall

[Maurdekye]

I have a feeling the solution might have to do with stencils, but for the life of me I cannot figure out how to accomplish it. Basically, I want all players to be rendered in front of obstacles, but if they would normally be visible, then they are invisible.

[James]

[QUOTE=Maurdekye;47539900]I have a feeling the solution might have to do with stencils, but for the life of me I cannot figure out how to accomplish it. Basically, I want all players to be rendered in front of obstacles, but if they would normally be visible, then they are invisible.[/QUOTE]

Draw the playermodel onto a stencil. Make your stencil set the pixels that Fail or FailZ, Selectively return true in PrePlayerDraw (don't return true when you are drawing for the stencil, or for the following). Draw player onto screen while respecting the stencil.

[Maurdekye]

[QUOTE=James xX;47540530]Draw the playermodel onto a stencil. Make your stencil set the pixels that Fail or FailZ, Selectively return true in PrePlayerDraw (don't return true when you are drawing for the stencil, or for the following). Draw player onto screen while respecting the stencil.[/QUOTE]

From what I understand about how stencils work, I don't need to set a Fail operation since I assume I'll be using ALWAYS, which never reaches a failure state. Also, how do I draw a player onto the screen while respecting the stencil?

[James]

[QUOTE=Maurdekye;47541112]From what I understand about how stencils work, I don't need to set a Fail operation since I assume I'll be using ALWAYS, which never reaches the failure operation. Also, how do I draw a player onto the screen while respecting the stencil?[/QUOTE]

That's only the case if you want to draw everything drawn onto the stencil. You use the operations if drawing it to the stencil follows a condition. Player:DrawModel()

[Maurdekye]

Wait, will drawing a model while the stencil is active prevent it from being drawn?

[James]

[QUOTE=Maurdekye;47541144]Wait, will drawing a model while the stencil is active prevent it from being drawn?[/QUOTE]

Prevent what from being draw?

[Maurdekye]

[QUOTE=James xX;47541154]Prevent what from being draw?[/QUOTE]

The playermodel.

Basically, if I were to use this code;

[lua]
hook.Add("PrePlayerDraw", "stencilwork", function(ply)
ply:DrawModel()
return true
end)
[/lua]

Then the player would be drawn. But if I used this code, making the draw call while the stencil is enabled;

[lua]
hook.Add("PrePlayerDraw", "stencilwork", function(ply)
render.SetStencilEnable(true)
ply:DrawModel()
render.SetStencilEnable(false)
return true
end)
[/lua]

Then the player wouldn't be rendered?

[James]

[QUOTE=Maurdekye;47541187]The playermodel.

Basically, if I were to use this code;

[lua]
hook.Add("PrePlayerDraw", "stencilwork", function(ply)
ply:DrawModel()
return true
end)
[/lua]

Then the player would be drawn. But if I used this code, making the draw call while the stencil is enabled;

[lua]
hook.Add("PrePlayerDraw", "stencilwork", function(ply)
render.SetStencilEnable(true)
ply:DrawModel()
render.SetStencilEnable(false)
return true
end)
[/lua]

Then the player wouldn't be rendered?[/QUOTE]

Don't call pl:DrawModel in PrePlayerDraw, you're causing an infinite loop that will make you crash without an error. Do your work in "PostDrawOpaqueRenderables", and control the return of the PrePlayerDraw via a local variable in the global scope, that you can change when needed.

[Maurdekye]

So what you're saying is, override the default rendering of players, but render them when it's most appropriate anyway?

If I'm understanding you correctly, then what you want me to do is something like this;

[lua]
local shouldDraw = false

function renderModel(ply)
shouldDraw = true
ply:DrawModel()
shouldDraw = false
end

hook.Add("PrePlayerDraw", "preventplayerdraw", function(ply)
return not shouldDraw
end)
[/lua]

And call the renderModel function in place of ply:DrawModel() while i'm tinkering with the stencils?

[James]

[QUOTE=Maurdekye;47541244]So what you're saying is, override the default rendering of players, but render them when it's most appropriate anyway?

If I'm understanding you correctly, then what you want me to do is something like this;

[lua]
local shouldDraw = false

function renderModel(ply)
shouldDraw = true
ply:DrawModel()
shouldDraw = false
end

hook.Add("PrePlayerDraw", "preventplayerdraw", function(ply)
return not shouldDraw
end)
[/lua]

And call the renderModel function in place of ply:DrawModel() while i'm tinkering with the stencils?[/QUOTE]

Yeah that's how you would do it.

[Maurdekye]

One last thing then; how would I render the player's model with respect to the stencil?

[James]

[QUOTE=Maurdekye;47541279]One last thing then; how would I render the player's model with respect to the stencil?[/QUOTE]

The same way you would draw anything else with respect to the stencil, without the need for the Cam3D2D (I don't have the code to hand, but you do)

[Maurdekye]

[QUOTE=James xX;47541299]The same way you would draw anything else with respect to the stencil, without the need for the Cam3D2D (I don't have the code to hand, but you do)[/QUOTE]

Well, I know the code necessary to draw a material over the screen with respect to the stencil. I just don't know how to do the same for a playermodel. Is it any similar?

[James]

[QUOTE=Maurdekye;47541305]Well, I know the code necessary to draw a material over the screen with respect to the stencil. I just don't know how to do the same for a playermodel. Is it any similar?[/QUOTE]

It's exactly the same, but not in a cam3D2D

[Maurdekye]

[QUOTE=James xX;47541343]It's exactly the same, but not in a cam3D2D[/QUOTE]

Well, this is the final code I have;

[lua]
local shouldDraw = false

function renderModel(ply)
shouldDraw = true
ply:DrawModel()
shouldDraw = false
end

hook.Add("PrePlayerDraw", "preventplayerdraw", function(ply)
return not shouldDraw
end)

hook.Add("PostDrawOpaqueRenderables", "stencilwork", function()
for _, p in pairs(player.GetAll()) do
render.SetStencilEnable(true)

render.ClearStencil()
render.SetStencilReferenceValue(20)
render.SetStencilCompareFunction(STENCIL_ALWAYS)
render.SetStencilPassOperation(STENCIL_KEEP)
render.SetStencilZFailOperation(STENCIL_REPLACE)

render.SetBlend(0)
renderModel(p)
render.SetBlend(1)

render.SetStencilCompareFunction(STENCIL_EQUAL)

renderModel(p)

render.SetStencilEnable(false)
end
end)
[/lua]

However, it doesn't work properly; the player is just invisible everywhere no matter what.

[editline]16th April 2015[/editline]

STENCIL_* enums work by the way, you can see them on the wiki's enum page; [url]http://wiki.garrysmod.com/page/Enums/STENCIL[/url]

[AnonTakesOver]

Cause your renderModel code will always evaluate to false in PrePlayerDraw (!true == false)

[Maurdekye]

[QUOTE=AnonTakesOver;47541501]Cause your renderModel code will always evaluate to false in PrePlayerDraw (!true == false)[/QUOTE]

Yes, but according to james, I'm supposed to render it manually with respect to the stencil, which I'm having trouble with.

[James]

[QUOTE=AnonTakesOver;47541501]Cause your renderModel code will always evaluate to false in PrePlayerDraw (!true == false)[/QUOTE]

[snip] Didn't see the "not"

[MeepDarknessM]

local IN_HOOK = false;
hook.Add("PrePlayerDraw", "", function(p)
	if(IN_HOOK) then return; end
	
	IN_HOOK = true;
	
	render.SetStencilEnable(true);
		
		render.ClearStencil();
		render.SetStencilReferenceValue(1);
		render.SetStencilCompareFunction(STENCIL_ALWAYS);
		render.SetStencilPassOperation(STENCIL_ZERO);
		render.SetStencilFailOperation(STENCIL_ZERO);
		render.SetStencilZFailOperation(STENCIL_REPLACE);
		render.SetBlend(0);
			p:DrawModel();
		render.SetBlend(1);
		
		render.SetStencilCompareFunction(STENCIL_EQUAL);
		
		cam.IgnoreZ(true);
		p:DrawModel();
		cam.IgnoreZ(false);
		
	render.SetStencilEnable(false);
	
	IN_HOOK = false;
	return true;
end)

this should work

[Willox]

render.IgnoreZ should be cam.IgnoreZ

[MeepDarknessM]

[QUOTE=Willox;47541592]render.IgnoreZ should be cam.IgnoreZ[/QUOTE]

ehh close enough
(fixed)

[Maurdekye]

It should be SetStencilZFailOperation instead of SetStencilReplaceOperation.
Also, it doesn't work; players just render everywhere.

[James]

[QUOTE=Maurdekye;47541608]It should be SetStencilZFailOperation instead of SetStencilReplaceOperation.
Also, it doesn't work; players just render everywhere.[/QUOTE]

FYI, source will not draw the playermodels if they are outside your PVS.

[lua]
local IN_HOOK = false;
hook.Add("PrePlayerDraw", "", function(p)
if(IN_HOOK) then return false; end

IN_HOOK = true;

render.SetStencilEnable(true);

render.ClearStencil();
render.SetStencilReferenceValue(1);
render.SetStencilCompareFunction(STENCIL_ALWAYS);
render.SetStencilPassOperation(STENCIL_ZERO);
render.SetStencilFailOperation(STENCIL_ZERO);
render.SetStencilZFailOperation(STENCIL_REPLACE);
render.SetBlend(0);
p:DrawModel();
render.SetBlend(1);

render.SetStencilCompareFunction(STENCIL_EQUAL);

cam.IgnoreZ(true);
p:DrawModel();
cam.IgnoreZ(false);

render.SetStencilEnable(false);

IN_HOOK = false;
return true
end)
[/lua]

[MeepDarknessM]

[QUOTE=Maurdekye;47541608]It should be SetStencilZFailOperation instead of SetStencilReplaceOperation.
Also, it doesn't work; players just render everywhere.[/QUOTE]

[vid]https://a.pomf.se/ytgrxw.mp4[/vid]
???

[Maurdekye]

Wait, it does work, I modified your version slightly and it somehow didn't work initially.