Facepunch server setup

[compwhizii]

I think this might be interesting for some of you. I'm redoing our server instances because they have issues. They're a pain in the ass to maintain and look after, comparable to little children. I've learned several lessons over the past week and I'm going to start fresh and do this again. This thread will serve as a log while I do this.

Goals:

[list]
[*]Better monitoring, install munin and monit
[*]Pull configuration and code off SVN every x minutes
[*]Be able to deploy new images and have them set themselves up
[*]Rotate logs
[/list]

Simple stuff. Feel free to comment and suggest, I really appreciate that. We begin....

[img]http://grab.by/grabs/3579e4c38b860ba553b69b0a1a05342a.png[/img]

Starting off with a basic Debian 5 EC2 instance.

[img]http://grab.by/grabs/e3aa0b5a3fa2845cf77edea721cfa60c.png[/img]

Creating a 10GB EBS volume that will serve as our new image home.
Attached to the instance on /dev/sdf.
Connected to the instance.

mke2fs -j /dev/sdf

Partition the EBS volume for ext3

[editline]16th October 2010[/editline]

mkdir /mnt/build
mount /dev/sdf /mnt/build

Mount the EBS volume

wget http://ec2ubuntu.googlecode.com/svn/trunk/bin/ec2ubuntu-build-ami
chmod +x ec2ubuntu-build-ami

Get the script.

TCPKeepAlive yes
ClientAliveInterval 30
ClientAliveCountMax 99999

Put this in [i]/etc/ssh/sshd_config[/i], prevents SSH timeouts.

/etc/init.d/ssh restart

Restart ssh

./ec2ubuntu-build-ami --ebs --user <User ID> --access-key <Access Key> --secret-key <Secret Key> --distribution debian --codename lenny

Run the script and wait

[eXeC64]

Interesting, I was curious about how facepunch was setup.

Also I think you've missed a wget.

[compwhizii]

Yup. Thanks.

[editline]16th October 2010[/editline]

You now need to:

 1. umount /mnt/build
 2. detach the EBS volume (ec2-detach-volume)
 3. create an EBS snapshot AMI (ec2-create-snapshot)
 4. (optional) delete the volume (ec2-delete-volume)
 5. register the snapshot as an AMI (ec2-register)

You can then use ec2-run-instances with your new AMI.

Script finishes

chroot /mnt/build mount -t proc none /proc
chroot /mnt/build mount -t devpts none /dev/pts
chroot /mnt/build

Switch into the new system

[editline]16th October 2010[/editline]

Add that stuff from above to sshd_config.

[b]nginx[/b]

wget http://nginx.org/download/nginx-0.8.52.tar.gz
tar xvzf nginx-0.8.52.tar.gz
cd nginx-0.8.52

Get the nginx code.

[Enjoi19]

Fascinating stuff. I follow you on twitter and enjoy seeing stuff like this

[compwhizii]

Following this guide loosely: [url]http://timothybowler.com/2009/11/25/compiling-nginx-on-debian-lenny/[/url]

apt-get install libc6 libpcre3 libpcre3-dev libpcrecpp0 libssl0.9.8 libssl-dev zlib1g zlib1g-dev lsb-base

Get stuff we need.

./configure \
--sbin-path=/usr/sbin \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--user=www-data \
--group=www-data \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--with-http_stub_status_module \
--with-http_realip_module

Configurate nginx.

make

Compile it annnnnnnndddd

make install

I bet you didn't expect that!

[Computermaster]

Thanks, Comp.

This provides great insight into one of my favorite places on the web.

[codenamecueball]

Really interesting compwhizii, always wondered what's it's like behind my fav web site.
To quote Brohoster [I]"Even a hooker wouldn't fuck compwhizii, he's that ugly"[/I]

But yeah, intresting stuff

[compwhizii]

#! /bin/sh
 
# Description: Startup script for nginx webserver on Debian. Place in /etc/init.d and
# run 'sudo update-rc.d nginx defaults', or use the appropriate command on your
# distro.
#
# Author:       Ryan Norbauer
# Modified:     Geoffrey Grosenbach http://topfunky.com
 
set -e
 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="nginx daemon"
NAME=nginx
DAEMON=/usr/sbin/$NAME
CONFIGFILE=/etc/nginx/nginx.conf
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
 
# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0
 
d_start() {
 $DAEMON -c $CONFIGFILE || echo -n " already running"
}
 
d_stop() {
 kill -QUIT `cat $PIDFILE` || echo -n " not running"
}
 
d_reload() {
 kill -HUP `cat $PIDFILE` || echo -n " can't reload"
}
 
case "$1" in
 start)
 echo -n "Starting $DESC: $NAME"
 d_start
 echo "."
 ;;
 stop)
 echo -n "Stopping $DESC: $NAME"
 d_stop
 echo "."
 ;;
 reload)
 echo -n "Reloading $DESC configuration..."
 d_reload
 echo "reloaded."
 ;;
 restart)
 echo -n "Restarting $DESC: $NAME"
 d_stop
 # One second might not be time enough for a daemon to stop,
 # if this happens, d_start will fail (and dpkg will break if
 # the package is being upgraded). Change the timeout if needed
 # be, or change d_stop to have start-stop-daemon use --retry.
 # Notice that using --retry slows down the shutdown process somewhat.
 sleep 5
 d_start
 echo "."
 ;;
 *)
 echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
 exit 3
 ;;
esac
 
exit 0

Use this for /etc/init.d/nginx

chmod +x /etc/init.d/nginx
/etc/init.d/nginx start

Test it out. Go to the instance's public DNS name, make sure it works.

[X'Trapolis]

Should have stuck with Windows.

[compwhizii]

/usr/sbin/update-rc.d -f nginx defaults

Set up nginx to launch on boot.

[xboomguy]

I bet those servers have pretty damn fast hardware.
Facepunch has never been this fast since.. forever. I also assume VB4 helps alot with performance.

Just one question, how large, in GB, is Facepunch right now?

[andersonmat]

[QUOTE=xboomguy;25450855]I bet those servers have pretty damn fast hardware.
Facepunch has never been this fast since.. forever. I also assume VB4 helps alot with performance.

Just one question, how large, in GB, is Facepunch right now?[/QUOTE]
:hurr:

More like nginx is baller and apache likes big dicks in its asshole along with IIS. It's one huge gay fest with apache and IIS.

Although, I admit that IIS is good, just not with PHP.

[thelinx]

FP never used Apache... (right?)

[andersonmat]

Oh, IIS.

[X'Trapolis]

[QUOTE=andersonmat;25451111] It's one huge gay fest with apache and IIS.[/QUOTE]

This graph tells a different story:

[img]http://ahb.me/GJD[/img]

nginx on the left, IIS on the right.

[andersonmat]

Too bad you're using graphs that don't properly represent what's happening because your application uses ASP.NET while Facepunch uses PHP. You can't cite data out of context.

[thelinx]

[QUOTE=X'Trapolis;25451201]This graph tells a different story:

[img_thumb]http://ahb.me/GJD[/img_thumb]

nginx on the left, IIS on the right.[/QUOTE]

I remember seeing this picture before. Except that time, nginx was on the right.

[X'Trapolis]

[QUOTE=andersonmat;25451111]Although, I admit that IIS is good, just not with PHP.[/QUOTE]

Sneaky edit.

[andersonmat]

[QUOTE=X'Trapolis;25451244]Sneaky edit.[/QUOTE]
:ssh:

[Enjoi19]

[QUOTE=xboomguy;25450855]I bet those servers have pretty damn fast hardware.
Facepunch has never been this fast since.. forever. I also assume VB4 helps alot with performance.

Just one question, how large, in GB, is Facepunch right now?[/QUOTE]

Posts alone are over 5.2GB.

[thelinx]

[QUOTE=thelinx;25451226]I remember seeing this picture before. Except that time, nginx was on the right.[/QUOTE]

And when I say "that picture" I mean [b]that picture[/b]. Nice steal.

[compwhizii]

[QUOTE=X'Trapolis;25451201]This graph tells a different story:

[img_thumb]http://ahb.me/GJD[/img_thumb]

nginx on the left, IIS on the right.[/QUOTE]

What the fuck is that graph telling me?

Nothing.

[Turki Azamat]

Suggestion: Screw E2, [url]http://linode.com[/url]

Also:

[quote]Pull configuration and code off SVN every x minutes[/quote]

[url]http://svnbook.red-bean.com/en/1.4/svn.ref.reposhooks.post-commit.html[/url]

post-commit exists for a reason

[X'Trapolis]

[QUOTE=Turki Azamat;25451478]Suggestion: Screw E2, [url]http://linode.com[/url][/QUOTE]

This. I looked into Amazon a while ago and they wanted $150 for a terabyte of data transfer.

:wtc:

[PvtCupcakes]

[QUOTE=X'Trapolis;25451201]This graph tells a different story:

[img_thumb]http://ahb.me/GJD[/img_thumb]

nginx on the left, IIS on the right.[/QUOTE]

What is the y axis trying to say?
And where the hell does it say anything about nginx or apache? I honestly don't know what the fuck this post is.

[compwhizii]

[QUOTE=Turki Azamat;25451478]Suggestion: Screw E2, [url]http://linode.com[/url]

Also:

[url]http://svnbook.red-bean.com/en/1.4/svn.ref.reposhooks.post-commit.html[/url]

post-commit exists for a reason[/QUOTE]

No.

[editline]16th October 2010[/editline]

I've used linode before, they're great but not for scalability.

Keep in mind we're using 6 instances.

[editline]16th October 2010[/editline]

[b]Zend Server[/b]

Follow [url]http://files.zend.com/help/Zend-Server/deb_installation.htm[/url]

deb http://repos.zend.com/zend-server/deb server non-free

Add to [i]/etc/apt/sources.list[/i]

wget http://repos.zend.com/zend.key -O- |apt-key add -
apt-get update

Self explanatory.

apt-get install zend-server-ce-php-5.3

Install PHP 5.3

[editline]16th October 2010[/editline]

Set up Zend Server by connecting to port 10081 in your web browser.

[editline]16th October 2010[/editline]

apt-get install php-5.3-extra-extensions-zend-server

Install extra stuff.

[editline]16th October 2010[/editline]

[b]Build xcache[/b]

Following [url]http://devzone.zend.com/article/4512[/url] Scenario 2

wget http://xcache.lighttpd.net/pub/Releases/1.3.0/xcache-1.3.0.tar.gz
tar xvzf xcache-1.3.0.tar.gz
cd xcache-1.3.0

Get the XCache source

apt-get install php-5.3-source-zend-server autoconf

Get the Zend Server code and the build tools needed

/usr/local/zend/bin/phpize

Preps the extension source.

./configure --with-php-config=/usr/local/zend/bin/php-config

Configure...

make

...make...

make install

...install

extension=xcache.so

Throw this into [i]/usr/local/zend/etc/php.ini[/i] to make sure it works.

[img]http://grab.by/grabs/6ea409b0c48c87a7ffaf8a49f5ae40e3.png[/img]

[editline]16th October 2010[/editline]

[b]Zend and FastCGIs[/b]

Zend comes with a fastcgi build of PHP and spawn-fcgi, but most don't know about it.

cp /usr/local/zend/gui/lighttpd/bin/spawn-fcgi /usr/local/zend/bin
cp /usr/local/zend/gui/lighttpd/sbin/php /usr/local/zend/bin/php-cgi

Move them out of hiding

#!/bin/bash
SCRIPT="/usr/local/zend/bin/spawn-fcgi -a 127.0.0.1 -p 9000 -C 5 -u www-data -g www-data -- /usr/local/zend/bin/php-cgi -c /usr/local/zend/etc/php.ini"
RETVAL=0
case "$1" in
    start)
      $SCRIPT
      RETVAL=$?
  ;;
    stop)
      killall -9 php-cgi
      RETVAL=$?
  ;;
    restart)
      killall -9 php-cgi
      $SCRIPT
      RETVAL=$?
  ;;
    *)
      echo "Usage: php-fastcgi {start|stop|restart}"
      exit 1
  ;;
esac      
exit $RETVAL

Throw this in [i]/etc/init.d/init-fastcgi[/i]

chmod +x /etc/init.d/init-fastcgi

Make it executable.

[editline]16th October 2010[/editline]

Start it and make sure it works, then stop it.

[Vbitz]

Awesome Work, the tutorial is... Informative.
Also is the database hosted on one server or all of them assessing the data files.

[compwhizii]

[QUOTE=Vbits;25454162]Awesome Work, the tutorial is... Informative.
Also is the database hosted on one server or all of them assessing the data files.[/QUOTE]

Database is hosted on Amazon RDS.

[Vbitz]

Are the forums updated via a script?

[compwhizii]

[QUOTE=Vbits;25454272]Are the forums updated via a script?[/QUOTE]

When I make changes I push them to the server through Webistrano, which causes the server to export the code from SVN.