So everyone hates waking up and having 2 lvl6’s outside their door right?
Make a system that magnifies your ability to destroy walls on a building where the multiplier is determined by some variable such as the number of keys you have to doors attached to walls which are connected to the structure, their levels and the age of the structures. Age of structure is easy to track and doesn’t tie to any individual. It also prevents people just building a ton of keyed doors adjacent to the house to get a multiplier. At max multiplier it takes as long to take down a lvl6 wall as it takes to put it up.
I know immediately the counter argument is that people could then destroy your base completely if they get the keys. But the base is f-cked if you lose your keys anyway. Keep it some effort to take down walls but not so much that it is a total pain when people grief you.