Load balancing and some weird MySQL sleeping connections & threads. - Developers

Hey guys, So I was wondering if anyone has come across a similar problem in their career of web development. I have recently joined a team and have been trying to work out an underlying issue which has been plaguing the system for a while now. At the moment, we have several web servers which are in rotation (two currently stable) that have a load balancer pointed at them. The strange thing is, whenever we stick a new server in (same architecture). Once it's put into rotation after around half an hour or so, we start seeing a lot of threads being created in the MySQL process list. These are NULL's, eventually - this brings the site to a stand still and we have to revert back to the previous two servers. We are running some legacy code on PHP 5.3.1 with the OS being Centos 6.X It's rather peculiar. I have tried checking for pconnect, which isn't set anywhere and I've been checking for persistent connections / closing off connections, all to no avail. The strange thing is it works fine for individual servers or those two servers. But when we stick a third in this problem kicks in, which makes debugging a real pain in the backside. Any clues? I have already tried checking for any Ajax calls which could be triggering this on high load times, but there isn't anything there which could be causing this.

[QUOTE=graymic;40545817]Hey guys, So I was wondering if anyone has come across a similar problem in their career of web development. I have recently joined a team and have been trying to work out an underlying issue which has been plaguing the system for a while now. At the moment, we have several web servers which are in rotation (two currently stable) that have a load balancer pointed at them. The strange thing is, whenever we stick a new server in (same architecture). Once it's put into rotation after around half an hour or so, we start seeing a lot of threads being created in the MySQL process list. These are NULL's, eventually - this brings the site to a stand still and we have to revert back to the previous two servers. We are running some legacy code on PHP 5.3.1 with the OS being Centos 6.X It's rather peculiar. I have tried checking for pconnect, which isn't set anywhere and I've been checking for persistent connections / closing off connections, all to no avail. The strange thing is it works fine for individual servers or those two servers. But when we stick a third in this problem kicks in, which makes debugging a real pain in the backside. Any clues? I have already tried checking for any Ajax calls which could be triggering this on high load times, but there isn't anything there which could be causing this.[/QUOTE] Perhaps there are some cronjobs running?

[QUOTE=P1raten;40545927]Perhaps there are some cronjobs running?[/QUOTE] Good point, I'll have to check the crontab list.

If you add a new server but turn off one of the two that are already running do you still see the issue? Are you 100% sure that the exact same application code is being run on all the servers? If the answer to both of those are yes, is your infrastructure setup automated with something like puppet or chef so that you know the new server is set up exactly the same way as the old servers?

[QUOTE=KmartSqrl;40548617]If you add a new server but turn off one of the two that are already running do you still see the issue? Are you 100% sure that the exact same application code is being run on all the servers? If the answer to both of those are yes, is your infrastructure setup automated with something like puppet or chef so that you know the new server is set up exactly the same way as the old servers?[/QUOTE] I've been wanting to implement puppet / chef. I've been trying to work out the problem that's been plaguing them for months. It literally only occurs when a new server is added. The first two servers generally are fine. I.E. Server A + B = Okay Server A + C = Nope Server B + C = Nope Server A + B + C = Nope Server C can literally be an image of any combination, I've tried a vanilla server on Centos / Ubuntu, compiled PHP from scratch and only enabled the very basic modules. It's starting to be a real pain to debug. As a previous user stated, I've tried checking to see if there were potentially a rogue script etc or something reference incorrectly within crontab, nothing there.

What happens if you image server a to server c and then turn server a off and run b and c only?

[QUOTE=KmartSqrl;40548788]What happens if you image server a to server c and then turn server a off and run b and c only?[/QUOTE] That's a good point, I'll have to do that early morning in case anything goes wrong. I'll get back tomorrow then. Cheers KmartSqrl.

For sure :) the trick for finding out the cause behind issues like this is figuring out ways to isolate every piece that could be causing the issue, so if you do that and it works, it means the issue is probably related to the load balancing, otherwise it's likely that it's an issue with the way the new servers are being set up.

[QUOTE=KmartSqrl;40549299]For sure :) the trick for finding out the cause behind issues like this is figuring out ways to isolate every piece that could be causing the issue, so if you do that and it works, it means the issue is probably related to the load balancing, otherwise it's likely that it's an issue with the way the new servers are being set up.[/QUOTE] Cheers bud, we resolved an approach. We now think it could even be a kernel related problem ironically. We're puppeting the working servers and going to try and deploy them to a new instance see if that resolves the problems. Cheers again Kmart.