Ok so ive played prior to ladders, ive played during. Ive done plenty of base building and plenty of raiding. Here is my overall solution.
Keep this change, make ladders building priority only. However we need to stop a few things.
1 - Dont allow bases to be built on rocks. Make all rocks in game limited to AT MOST a 2x2 foundation, so make rocks pointed and wonky.
2 - Make tool cupboards only cover areas above them and not below. So for example a tool cupboard on the second level of a base will only effect the second, third, fourth etc levels, but not the first level because it is lower on the y-Axis. So then people will be able to destroy bottom level cupboards of a base and get up instead of cheeky builders having a tool cupboard on the 4th floor and no way to access it.
3 - Make is so that all stairs and building half blocks cannot be made as twig, only wood and up (much like doors in a door frame where you cant make them twig). So then base owners cant be cheeky and bust out the means they used to enter and exit a base.
4 - Make ladders expensive and hard to destroy. Same reason as point 2, it will make it hard for base builders to be cheeky and destroy the same ladder they used to get into their base.
These 2 points will sway the balance a little bit back to the raider, but not as much as ladder going anywhere as of recent, or as much as rock and elevator base simulator that we had for the last 6 months prior to ladders being introduced.