Friend can't connect to my server while others can

[Victus]

I set up a listen server in gMod, opened the needed Ports (27015), most people can join my server just fine. So please no replies like "U gave your friend your lan ip" or "Open those ports".
The funny thing is: He could join, then his router reconnected (He's one of those rich people that never shuts down their router to save energy), like it does every 24h. After that he couldn't join my server, even though MY IP didn't change at all. He just got those all saying "failed after 4 retries" errors.

We can ping (around 60ms) each other, so there should be no problem with the internet connection itself.
We do not want to use Hamachi or Tunngle or whatever VPN Program, because they tend to be more laggy than direct connection...
We also both checked for viruses without results.
Firewalls should not interfere with garrys Mod. (As said: people join my server, but he can't)

Is there anything else we can do about this?

[Dynn]

You don't only need to open port 27015?

Open these ports:
PROTOCOL / INBOUND / OUTBOUND
UDP 1200 1200
UDP 26901 26901 (This will allow the server to show up on the master server list.)
UDP 27000 27015
TCP 27020 27039
UDP 27015 27015 (you may not be able to open this one, but it is suggested that you try)
UDP 27020 27020
TCP 27015 27015

Source from GMod Wiki: [url]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/indexb076.html#Port_Forwarding[/url]