Issue with L4D coop/vs server & lobby

I run a linux dedicated server on my 2nd computer and my ports are forwarded correctly (1200, 27000-27050 I realize I don’t need that many ports open but I was combining several sets into 1 rule) for both tcp/udp and my linux firewall has the same ports open as well, when starting l4d people can see the server in the server browser and in the steam groups area (its linked to my group) however I have tried both using sv_search_key and ms_force_dedicated_server (but not together as I believe they would conflict) to connect a lobby to it and at best it works maybe 10% of the time (and works well when it does) and when it does not the lobby just sits there searching for a server with no errors in either console output. Now connecting to the server locally or through steam groups/server browser works fine every time and connecting through my external ip through console works all the time as well but matchmaking does not work when done this way I was wondering if anyone had any thoughts as to why this was occurring as its kind of driving me nuts.

Although I don’t think specs should matter the computer running the server is built as follows:
C2D E6600
6GB DDR2 667Mhz
EVGA 650i mobo
EVGA 8800GTS 640mb
/root on 120GB IDE HD
/home on 400GB SATA HD

NOTE: when using either method: sv_search_key or ms_force_dedicated_server i set the value in console when I am in lobby not before if this is the problem I will facepalm.