Drawing props on players, disappearing when many players are present

For a gamemode I’m working on I want to draw some props on players’ models when they have certain upgrades. I’m doing this using PostPlayerDraw and ClientsideModel. For every prop I need, I’m making just one ClientsideModel and stashing it in a table. Then, I loop over the players and do something that looks like this:

hook.Add( "PostPlayerDraw", "DrawModelsPart2", function (ply)        
        if ply.hassomeupgrade then
            for k,v in pairs(modelsForThisUpgrade) do
                local boneid = ply:LookupBone(translate_bone(v.bone))
                if not boneid then return end
                local matrix = ply:GetBoneMatrix(boneid)
                if not matrix then return end
                local offpos, offang = v.pos or Vector(0,0,0), v.ang or Angle(0,0,0)
                local newpos, newang = LocalToWorld(offpos, offang, matrix:GetTranslation(), matrix:GetAngles())
                local model = modelCache[v.model]
                model:SetPos( newpos )
                model:SetAngles( newang )

This works GREAT for one player. It works okayish for two players. Much beyond that, and when observing a player from any notable distance, the props simply refuse to draw.

I know there’s a better way to do it. What am I doing wrong? What’s the best way to do this?