Switch your server to Steam group whitelisting, but make the group publicly-joinable. This way, you can ban people from the group if they’re fuckwads, and there’s a SMALL barrier to any random-ass yahoo joining in without ever reading your server rules. However, it’s still a public server so you can grow it without tedious hand-approval (and if you aren’t really fast on approving, they’ll just go elsewhere anyway unless your server is amazing).
As Snickerdoodle says, administrate the server actively and fairly, and curate a community that can play nicely together while still killing each other. It will take time to grow a server with a stable and durable population, and there is a risk to trying to grow huge too fast.
Offering cash prizes for inflated server population numbers, not that you’ve already done this yourself but others have, is a terrible idea, just in case you see threads and think you might try it. You get a huge disruptive flood (if you’re successful in getting attention and traffic) that yields only a small handful of people hanging around afterwards.
Another advantage of Steam group whitelisting is, you have a safety brake that you can pull. In the event that the server gets a troll flood (i.e. a pack of people show up to roll the server and invite their friends and create multiple Steam accounts with Rust to bypass Steam ID blocks), you can lock the group from public joining, kick the idiots out, and leave it on admin-invite-only membership for a few hours or days until the dust settles and the trolls get bored. This way, any respectable player in your server’s community can still join and play, since the whitelisting on the Rust server itself wasn’t modified at all. All that changed was pulling up the drawbridge on the Steam group and then cleaning out the trolls.