gm_video quality

Is there anyway to make it record in a higher quality?

You mean a higher resolution?

You can use the -w and -h steam parameters to change your game resolution to something higher and just record from that if you want. Note, though, that it will look pretty bad on a smaller screen and requite a lot more power.