http.Fetch() Serverside vs. Clientside performance

To start, I have a Cinema server and I’ve been noticing some lacking performance lately in the form of Video API Requests using http.Fetch(). I created a simple shared script to test how long it takes for a http.Fetch() request to process that prints the results to Console/a text file.

Code


function TestHTTPFetch(url)
	if !url then return end
	local Start = CurTime()
	
	http.Fetch(url,
		function(body, length, headers, code)
			-- OnSuccess
			if code == 200 or code == 0 then
				file.Append("httptest.txt", "[TestHTTPFetch] Success (" ..(CurTime() - Start).. " seconds): " .. code .. "
")
				print("[TestHTTPFetch] Success (" ..(CurTime() - Start).. " seconds): " .. code)
			else
				file.Append("httptest.txt", "[TestHTTPFetch] Unexpected Code (" ..(CurTime() - Start).. " seconds): " .. code .. "
")
				print("[TestHTTPFetch] Unexpected Code (" ..(CurTime() - Start).. " seconds): " .. code)
			end
		end,
		function(err)
			-- OnFailure
			file.Append("httptest.txt", "[TestHTTPFetch] Failure (" ..(CurTime() - Start).. " seconds): " .. err .. "
")
			print("[TestHTTPFetch] Failure (" ..(CurTime() - Start).. " seconds): " .. err)
		end
	)
end

timer.Create("TestHTTPPerformance", 1, 20, function()
	TestHTTPFetch(/* TEST URL GOES HERE */)
end)

Using that code I tested a few websites, and overall it appears that performance on the Server is significantly slower than it is on the Client, sometimes even failing to fetch the page Serverside. It also appears to get slower each consecutive time Serverside, and if you wait a while between running http.Fetch() calls, the performance appears to get better on the Server.

Results



-- www.google.com

-- SERVER
[TestHTTPFetch] Success (2.966748046875 seconds): 200
[TestHTTPFetch] Success (3.7444984375 seconds): 200
[TestHTTPFetch] Success (4.5 seconds): 200
[TestHTTPFetch] Success (6 seconds): 200
[TestHTTPFetch] Success (7.7333984375 seconds): 200
[TestHTTPFetch] Success (9.2666015625 seconds): 200
[TestHTTPFetch] Success (10.8990234375 seconds): 200
[TestHTTPFetch] Success (12.36669921875 seconds): 200
[TestHTTPFetch] Success (13.966552734375 seconds): 200
[TestHTTPFetch] Success (15.60009765625 seconds): 200
[TestHTTPFetch] Success (17.166748046875 seconds): 200
[TestHTTPFetch] Success (19.3666015625 seconds): 200
[TestHTTPFetch] Failure (20.566650390625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.066650390625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.09985351625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.09985351625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.09985351625 seconds): unsuccessful
[TestHTTPFetch] Failure (19.966552734375 seconds): unsuccessful

-- CLIENT
[TestHTTPFetch] Success (12 seconds): 200
[TestHTTPFetch] Success (10 seconds): 200
[TestHTTPFetch] Success (11 seconds): 200
[TestHTTPFetch] Success (9.166748046875 seconds): 200
[TestHTTPFetch] Success (8.166748046875 seconds): 200
[TestHTTPFetch] Success (7.166748046875 seconds): 200
[TestHTTPFetch] Success (5.166748046875 seconds): 200
[TestHTTPFetch] Success (6.33349609375 seconds): 200
[TestHTTPFetch] Success (4.33349609375 seconds): 200
[TestHTTPFetch] Success (2.33349609375 seconds): 200
[TestHTTPFetch] Success (1.33349609375 seconds): 200
[TestHTTPFetch] Success (3.33349609375 seconds): 200
[TestHTTPFetch] Success (0.5 seconds): 200
[TestHTTPFetch] Success (0.2333984375 seconds): 200
[TestHTTPFetch] Success (0.266845703125 seconds): 200
[TestHTTPFetch] Success (0.300048828125 seconds): 200
[TestHTTPFetch] Success (0.433349609375 seconds): 200
[TestHTTPFetch] Success (0.33349609375 seconds): 200
[TestHTTPFetch] Success (0.13330078125 seconds): 200
[TestHTTPFetch] Success (0.33349609375 seconds): 200

-- www.youtube.com

-- SERVER
[TestHTTPFetch] Success (3.966796875 seconds): 200
[TestHTTPFetch] Success (5.5 seconds): 200
[TestHTTPFetch] Failure (20.5 seconds): unsuccessful
[TestHTTPFetch] Failure (20.10009765625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.10009765625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.10009765625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.10009765625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful

-- CLIENT
[TestHTTPFetch] Success (4.7001953125 seconds): 200
[TestHTTPFetch] Success (5.60009765625 seconds): 200
[TestHTTPFetch] Success (3.2333984375 seconds): 200
[TestHTTPFetch] Success (2.1669921875 seconds): 200
[TestHTTPFetch] Success (1.2333984375 seconds): 200
[TestHTTPFetch] Success (3.2333984375 seconds): 200
[TestHTTPFetch] Success (0.7001953125 seconds): 200
[TestHTTPFetch] Success (0.56689453125 seconds): 200
[TestHTTPFetch] Success (0.56689453125 seconds): 200
[TestHTTPFetch] Success (0.56689453125 seconds): 200
[TestHTTPFetch] Success (0.53369140625 seconds): 200
[TestHTTPFetch] Success (0.56689453125 seconds): 200
[TestHTTPFetch] Success (0.5 seconds): 200
[TestHTTPFetch] Success (0.53369140625 seconds): 200
[TestHTTPFetch] Success (0.5 seconds): 200
[TestHTTPFetch] Success (0.53369140625 seconds): 200
[TestHTTPFetch] Success (0.60009765625 seconds): 200
[TestHTTPFetch] Success (0.60009765625 seconds): 200
[TestHTTPFetch] Success (0.53369140625 seconds): 200
[TestHTTPFetch] Success (0.53369140625 seconds): 200

-- www.yukitheater.org

-- SERVER
[TestHTTPFetch] Failure (17.7998046875 seconds): unsuccessful
[TestHTTPFetch] Failure (17.66650390625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.5 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.10009765625 seconds): unsuccessful
[TestHTTPFetch] Failure (19.966796875 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.13330078125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06689453125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20 seconds): unsuccessful

-- CLIENT
[TestHTTPFetch] Success (1.43359375 seconds): 200
[TestHTTPFetch] Success (1.10009765625 seconds): 200
[TestHTTPFetch] Success (1.39990234375 seconds): 200
[TestHTTPFetch] Success (1 seconds): 200
[TestHTTPFetch] Success (0.93359375 seconds): 200
[TestHTTPFetch] Success (1 seconds): 200
[TestHTTPFetch] Success (2.400390625 seconds): 200
[TestHTTPFetch] Success (1.1669921875 seconds): 200
[TestHTTPFetch] Success (2.466796875 seconds): 200
[TestHTTPFetch] Success (1 seconds): 200
[TestHTTPFetch] Success (0.86669921875 seconds): 200
[TestHTTPFetch] Success (0.900390625 seconds): 200
[TestHTTPFetch] Success (0.93359375 seconds): 200
[TestHTTPFetch] Success (1.03369140625 seconds): 200
[TestHTTPFetch] Success (0.93359375 seconds): 200
[TestHTTPFetch] Success (0.966796875 seconds): 200
[TestHTTPFetch] Success (1 seconds): 200
[TestHTTPFetch] Success (0.900390625 seconds): 200
[TestHTTPFetch] Success (1.10009765625 seconds): 200
[TestHTTPFetch] Success (0.900390625 seconds): 200

-- My local network's Web Server (I waited a few minutes after running the last one)

-- SERVER
[TestHTTPFetch] Success (10.36669921875 seconds): 200
[TestHTTPFetch] Success (0.033203125 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (11.96630859375 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (0.033203125 seconds): 200
[TestHTTPFetch] Success (0.06640625 seconds): 200
[TestHTTPFetch] Success (13.56640625 seconds): 200
[TestHTTPFetch] Success (0.06640625 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (15.099609375 seconds): 200
[TestHTTPFetch] Success (0.033203125 seconds): 200
[TestHTTPFetch] Success (16.599609375 seconds): 200
[TestHTTPFetch] Success (18.2666015625 seconds): 200
[TestHTTPFetch] Success (19.7666015625 seconds): 200
[TestHTTPFetch] Failure (20.033203125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06640625 seconds): unsuccessful
[TestHTTPFetch] Failure (20.13330078125 seconds): unsuccessful
[TestHTTPFetch] Failure (20.06640625 seconds): unsuccessful

-- CLIENT
[TestHTTPFetch] Success (11.36669921875 seconds): 200
[TestHTTPFetch] Success (10.36669921875 seconds): 200
[TestHTTPFetch] Success (8.36669921875 seconds): 200
[TestHTTPFetch] Success (9.36669921875 seconds): 200
[TestHTTPFetch] Success (7.36669921875 seconds): 200
[TestHTTPFetch] Success (6.36669921875 seconds): 200
[TestHTTPFetch] Success (5.36669921875 seconds): 200
[TestHTTPFetch] Success (2.03369140625 seconds): 200
[TestHTTPFetch] Success (2 seconds): 200
[TestHTTPFetch] Success (1.966796875 seconds): 200
[TestHTTPFetch] Success (1.36669921875 seconds): 200
[TestHTTPFetch] Success (0.06689453125 seconds): 200
[TestHTTPFetch] Success (0.16650390625 seconds): 200
[TestHTTPFetch] Success (0.06640625 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (0.099609375 seconds): 200
[TestHTTPFetch] Success (0.13330078125 seconds): 200
[TestHTTPFetch] Success (0.16650390625 seconds): 200
[TestHTTPFetch] Success (0.06640625 seconds): 200
[TestHTTPFetch] Success (0.13330078125 seconds): 200


Does anybody have any theories as to why this could be the case?

The http library can only fetch a single file at a time, the rest are queued. Maybe your server just is slower at downloading the files.

I thought the same. However I ran that same code on my GMod Client in a Singleplayer game and got the same results. It’s always worse running Serverside for some reason, at least on Windows.

I feel like we had a discussion on fetch in the beta and the problem was strictly related to windows servers. I could be wrong and I’m pretty sure the older thread is pruned or hidden.

I feel like we did too, but I can’t find the Thread to save my life.

Fetch has always been problematic on my servers. It’d be great if it was more reliable, seems like it randomly fails half the time for the exact same URL.

It might be NFO’s firewall blocking the connections but they said it’s not their end.

There also seems to be a problem with HTTP redirects. It only works for 1-2 requests before it will return with no data (client and server on Windows).

http://samuelmaddock.com/up/2014-06-01_15-13-17.png