Issues playing with others on same router.

Long story short I am playing Rust with my roommates and we keep having issues only when we are in the same server together. The main issue is that our hot bar becomes inactive and at times we lose the ability to interact with anything leading to DC. Thanks for the help.