Something is limiting my ARK dedicated server to 3 people.

[Haburdier]

So I have an ARK dedicated server with six slots because I play with a small group, but only 3 peopel can join, if a fourth tries to join they get "connection failed" or something like that. We've tried several times and we just can't get it to work. I'm not on DSL or anything, but here are my specs for reference:

Speedtest results:
Ping - 30 MS
Download - 101.67 Mbps
Upload - 10.58 Mbps

CPU - i7-6700k 4.00 ghz
32 GB Ram