Really impressed with the speed for being a scraper. Less than a second for getting new posts on WAYWO is kind of impressive. I also really like the API console, props to you.
Is it just me, or is down? I get a 502 Bad Gateway when I try to access facepun.ch and api.facepun.ch.
It was down, fixed.
PHP-FPM keeps dying, in a meeting at the moment but I'll investigate afterwards.
Its down again :(
Maybe you could make it cache older threads and posts (things that probably won't change) so it doesn't have to hit the server every time?
[QUOTE=garry;34678919]Maybe you could make it cache older threads and posts (things that probably won't change) so it doesn't have to hit the server every time?[/QUOTE]
I could make read-only requests go through CloudFlare I think, would that improve the situation?
It means those requests would appear to be from a single IP address, but any actions like posts, threads etc would still show as the requesting users IP.
No what I mean is - if someone requests page 1 of the WAYWO thread and it's been requested before you don't really need to touch the facepunch servers at all if you've cached it.
[QUOTE=garry;34679705]No what I mean is - if someone requests page 1 of the WAYWO thread and it's been requested before you don't really need to touch the facepunch servers at all if you've cached it.[/QUOTE]
The page looks different for different users. For example, mods see posts that have been deleted, I don't think users can. If we cache the version from a mod, normal users could see that data. It's a security risk sharing cached versions between users, but caching per user wouldn't be feasible.
[QUOTE=Hexxeh;34679745]The page looks different for different users. For example, mods see posts that have been deleted, I don't think users can. If we cache the version from a mod, normal users could see that data. It's a security risk sharing cached versions between users, but caching per user wouldn't be feasible.[/QUOTE]couldn't you cache per user group?
[QUOTE=danharibo;34679903]couldn't you cache per user group?[/QUOTE]
There's no reliable way of detecting user groups, the ID isn't included in the page source anywhere iirc.
You could look at username colours, but then there are special cases like garry's.
[QUOTE=Hexxeh;34679918]There's no reliable way of detecting user groups, the ID isn't included in the page source anywhere iirc.
You could look at username colours, but then there are special cases like garry's.[/QUOTE]
And I guess adding some hidden fields to the forum just for the API is not possible.
Wouldn't just hitting CloudFlare for read requests solve the problem? That way we're only hitting with the same load a normal web user would hit with, and it involves no extra complexity API side.
Not really, if you hit cloudflare or the server directly it's the same result. I was just looking for a way to stop the effect of people reading a thread of 2,000 posts over and over.. but I guess there's no way for it to be done nicely.
You could add <!-- YOU ARE A GOLD MEMBER --> to the header?
[editline]13th February 2012[/editline]
(or whatever usergroup the user is)
Maybe reading public forums could always be done `logged out` and cached?
You don't let logged out users browse the forums.
Yeah I do
[img]http://puu.sh/gMEN[/img]
Huh, when did that change?
Last week I had to log in on someone else's computer because it wouldn't let me browse otherwise.
[QUOTE=garry;34681031]Maybe reading public forums could always be done `logged out` and cached?[/QUOTE]
We wouldn't have the rating keys then to allow post ratings, or the security tokens to allow posting.
I'd imagine your api would be one call to post - which would load the page, get the security token and then post. Seems odd to make the user of the API manually load the page to make a post?
[QUOTE=garry;34681884]I'd imagine your api would be one call to post - which would load the page, get the security token and then post. Seems odd to make the user of the API manually load the page to make a post?[/QUOTE]
For posting, yeah, but for ratings they need to provide the key from the original getposts call, we don't retrieve that for them. I try to keep duplicate requests like the secinfo ones to a minimum because they increase the request response time.
Why wouldn't CloudFlare for read requests solve the problem? Doesn't that prevent a hit on the server for a previously loaded page?
Another thing I could do is implement request rate limiting per client, if that'd help.
It is taking a long time for requests to be processed.
[editline]13th February 2012[/editline]
nvm it's probably just facepunch.
Should we have a separate thread for tests? Like for testing different positing functionalities?
EDIT:
Also, how do I send a new post in a URL? Is that even possible?
[editline]13th February 2012[/editline]
I'm posting alot but I found this:
[code]\/avatar\/63187.png?garryis=awesome[/code]
In one of the responses , garry u so silly
Hexxeh, are there a limit to how many login requests can be done?
[editline]14th February 2012[/editline]
Also, you should list all the error codes you can get on the API Actions page.
[QUOTE=Hexxeh;34682027]For posting, yeah, but for ratings they need to provide the key from the original getposts call, we don't retrieve that for them. I try to keep duplicate requests like the secinfo ones to a minimum because they increase the request response time.
Why wouldn't CloudFlare for read requests solve the problem? Doesn't that prevent a hit on the server for a previously loaded page?
Another thing I could do is implement request rate limiting per client, if that'd help.[/QUOTE]
No cloudflare doesn't work like that for dynamic pages, it just passes right through (unless it detects you as a DDOS attack, then it blocks it).
[QUOTE=garry;34691132]No cloudflare doesn't work like that for dynamic pages, it just passes right through (unless it detects you as a DDOS attack, then it blocks it).[/QUOTE]
Why use cloudflare then?
Because it protects against DDOS and caches static content
[QUOTE=garry;34691170]Because it protects against DDOS and caches static content[/QUOTE]
It doesn't stop someone attacking the real Facepunch server(s?) and static content is something you can just stick appropriate cache headers on
Oh great
[editline]14th February 2012[/editline]
Worked
[QUOTE=dije;34690769]Hexxeh, are there a limit to how many login requests can be done?
[editline]14th February 2012[/editline]
Also, you should list all the error codes you can get on the API Actions page.[/QUOTE]
No limit on successful ones, but too many incorrect ones and Facepunch will block you for a while.
[QUOTE=swift and shift;34691187]It doesn't stop someone attacking the real Facepunch server(s?) and static content is something you can just stick appropriate cache headers on[/QUOTE]
They don't know the real server addresses, only the CloudFlare addresses. CloudFlare knows the Facepunch addresses and passes requests onwards on your behalf. That's my understanding at least.
Sorry, you need to Log In to post a reply to this thread.
