2D Stencil Operation Help

[Dr. Inco]

I'm racking my brain trying to get stencils to work in a 2D environment. Can someone walk me through how I would use this green stencil (not visible in the end product):

https://files.facepunch.com/forum/upload/111825/10a1e9c1-98ae-45aa-961d-092d773cf33a/image.png

And draw this in its place:

https://i.imgur.com/AOIK1KU.png

The different stencil operations and setting pixel values are really tripping me out. All help is greatly appreciated.

[LoweChaser]

What have you tried so far? There is a useful library - Blue's Masks and Shadows!

[Dr. Inco]

I took a look at his library, and it looks great but seems to only work based off alpha channels in bitmap images. I'm drawing the stencil with real-time calculations.

I did manage to get semi-successful results at one point:

https://i.imgur.com/EBSTIVa.png

The black rectangle was successfully cut off by the stencil, but the blur behind it remains rectangular if you look close enough, despite being drawn with the black banner...
render.ClearStencil()
render.SetStencilEnable(true)
    --
    -- STENCIL
    --
    render.SetStencilReferenceValue(1)
    render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_NEVER)
    render.SetStencilPassOperation(STENCILOPERATION_ZERO)
    render.SetStencilZFailOperation(STENCILOPERATION_REPLACE)
    render.SetStencilFailOperation(STENCILOPERATION_REPLACE)

    surface.SetDrawColor(stencilCol)
    surface.DrawRect((bannerW * 0.25) , 0, (bannerW * 0.5), bannerH)
    draw.Circle(bannerW * 0.25, 0, bannerH, 90, 90, stencilCol)
    draw.Circle(bannerW * 0.75, 0, bannerH, 90, 0, stencilCol)

    --
    -- BANNER
    --
    render.SetStencilCompareFunction(STENCILCOMPARISONFUNCTION_EQUAL)

    draw.Blur((bannerW * 0.25) - bannerH, 0, (bannerW * 0.5) + bannerH * 2, bannerH)
    surface.SetDrawColor(bannerCol)
    surface.DrawRect((bannerW * 0.25) - bannerH, 0, (bannerW * 0.5) + bannerH * 2, bannerH)
render.SetStencilEnable(false)