Stencils and RenderView

At present I have code that looks something like this


            render.ClearStencil()
             render.SetStencilEnable(true)


            render.SetStencilWriteMask( 1 )
            render.SetStencilTestMask( 1 )


            render.SetStencilFailOperation( STENCIL_REPLACE )
            render.SetStencilPassOperation( STENCIL_ZERO )
            render.SetStencilZFailOperation( STENCIL_ZERO )
            render.SetStencilCompareFunction( STENCIL_NEVER )
            render.SetStencilReferenceValue( 1 )


            draw.RoundedBox(((s2.x-s1.x)+(s2.y-s1.y))/8, s1.x, s1.y, s2.x-s1.x, s2.y-s1.y, Color(255,255,255,255))
            --average of diameters, divided by 2.


            render.SetStencilFailOperation( STENCIL_ZERO )
            render.SetStencilPassOperation( STENCIL_REPLACE )
            render.SetStencilZFailOperation( STENCIL_ZERO )
            render.SetStencilCompareFunction( STENCIL_EQUAL ) -- STENCILCOMPARISONFUNCTION_EQUAL will only draw what you draw as the mask.
            render.SetStencilReferenceValue( 1 )


            render.RenderView( CamData )


            render.SetStencilEnable(false)
            render.ClearStencil()

With the important bit being that I draw a roundedbox to try and round off a renderview. Later this will be changed to a much smarter method.

However, this always makes the RenderView render just black. Whereas removing all the stenciling code makes it render correctly.

Is what I’m trying to do possible? Am I even close to doing it correctly?

I ran into this issue similarly. I don’t have a solution.

I know it’s not your main question, but rounded boxes use materials, so the corners are really just rotated squares afaik

to use stencils for rounded stuff you have to use polys

I know that. In my most recent attempt, I am using polys. But I simply cannot get the renderview to appear no matter how I clip it with a stencil. It always renders black.

Render your view to a render-target and then perform your stencil operations on that, otherwise it’s going to be processed throughout every render process your scene rendering causes.

I do this exact thing here https://github.com/wiox/gmod-gta-radar/blob/master/lua/autorun/client/gta_radar.lua (lines 101-132 & 156).