No Logon Steam / Timeout

Some Players are complaining because sometimes they got the Error: No Logon Steam then Disconnect, on Server shows Code 6, or They got the Timeout message, and they say its only on my Server.
The funny thing is sometimes are 2 or 3 players on server and 1 or 2 players try join get the Timeout or No Steam Logon.
Why this happen? How to fix it? Is a Router Problem?

I Have a Core 2 Quad, 4GB Ram, 1.5TB, 2.5MBPS Download, 120kbps Upload, I’m putting 5 Slots