I have a unique multiplayer lan problem

Me and my father wish to play a garr’ys mod with each other. Though we run into a unique problem when we do.
We are able to get on and play the lan game “even though we have to directly connect because for some odd reason we can see each other on the play multiplayer game server searching thing, and yet when we click it says server is down” and so we just use the console and direct connect using the “connect Ip” command.
We are able to successfully connect play and have near to no lag ect just as I hoped since I dont like waste’ing money. Yet, when we play My father is able to not only undo the things I create, but also use the numpad controls of everything I own from wiremod to the offical thrusters that come with garry’s mod.

Please if anybody has experiences with this odd problem refer me to a solution. I have searched on this forum/Wiki/Faq and can’t find something similar to this dilemma

Are you on the same Steam account?

Also have you tried as the host to type in “sv_lan 1” in the console?

We once were on the same. However now n days we have two seperate games and two seperate accounts so I really dont know whats going on.