Built something last night to spider the external URLs for products on spoonfedstyle.com every night at 1 am. If a product page is unable to be loaded for 3 nights in a row the product is deleted from the site.
Now I'm working on a script that grabs all images from the product page and finds higher resolution versions of the product image, because some of the APIs that I'm using to find products for the site give back really junky images that the APi has stuck up on s3 even though there are better ones on the merchant sites. So far I've got the bit that pulls all the image tags on the page and downloads them done, along with the part that compares images to determine if it's the same image (using phash through the phashion gem for the comparison) It's been surprisingly easy so far, the most trouble I had was fighting with graphicsmagick to get it to actually build properly haha.
[QUOTE=KmartSqrl;41872298]Built something last night to spider the external URLs for products on spoonfedstyle.com every night at 1 am. If a product page is unable to be loaded for 3 nights in a row the product is deleted from the site.
[/QUOTE]
So if someone were to DOS the site at 1am for 3 nights in a row they could wipe out the products? I know you probably monitor traffic but this could be a problem.
Edit:[editline][/editline]
Seeing that most/all of the products are on Amazon, that wont be a problem.
[QUOTE=CBastard;41872584]So if someone were to DOS the site at 1am for 3 nights in a row they could wipe out the products? I know you probably monitor traffic but this could be a problem.
Edit:[editline][/editline]
Seeing that most/all of the products are on Amazon, that wont be a problem.[/QUOTE]
I'll have more and more products coming from other smaller retailers, but I don't really think it's likely that someone is going to DOS a women's dress retailer's website for 3 days, and if they do, the products will be gone. I have nothing to gain from showing people inaccessible products.
[QUOTE=KmartSqrl;41872819]I'll have more and more products coming from other smaller retailers, but I don't really think it's likely that someone is going to DOS a women's dress retailer's website for 3 days, and if they do, the products will be gone. I have nothing to gain from showing people inaccessible products.[/QUOTE]
Not for 3 whole days, but for a just for a minute at 1am each day, My point is that you should probably randomise the time at which it checks or not tell anyone when it checks.
[QUOTE=CBastard;41873256]Not for 3 whole days, but for a just for a minute at 1am each day, My point is that you should probably randomise the time at which it checks or not tell anyone when it checks.[/QUOTE]
I'm checking when my website traffic is lower, for long running jobs like this I want them to happen at a reliable time that is off peak-hours (not that peak hours are at all intensive on this site yet haha).
I'm not going to waste time building safeguards for an issue that is very very very unlikely to occur, and that won't cause any real damage if it does occur. This script only soft-deletes products by switching a bit in the db so if I notice that half the stuff on the site suddenly gets disabled one night it's not going to cause any damage.
[IMG]http://i.imgur.com/THpXXaJ.png[/IMG]
i made a blast from the past
[QUOTE=Giraffen93;41875945][IMG]http://i.imgur.com/THpXXaJ.png[/IMG]
i made a blast from the past[/QUOTE]
Will you be releasing it?
[QUOTE=James xX;41880683]Will you be releasing it?[/QUOTE]
Yes
[QUOTE=DarKSunrise;41881458][t]http://files.1337upload.net/Screenshot_from_2013-08-18_20_03_53.png[/t][/QUOTE]
Just looked up that zlib license.
Is it just GPL but without the no sub licensing bit?
It's fucking hilarious to me that web/interface designers seem to be getting a sudden boner for well done motion design and talking about it like it's a new frontier. Apparently everyone forgot about how key that was when people were doing flash sites and doing them really well lol... I think it's partially a consequence of the flat trend and people trying to figure out how to make their stuff stand out when all the flat designs look pretty similar, and partially that it's easier to do fancy motion design with flat designs.
[QUOTE=KmartSqrl;41882646]It's fucking hilarious to me that web/interface designers seem to be getting a sudden boner for well done motion design and talking about it like it's a new frontier. Apparently everyone forgot about how key that was when people were doing flash sites and doing them really well lol... I think it's partially a consequence of the flat trend and people trying to figure out how to make their stuff stand out when all the flat designs look pretty similar, and partially that it's easier to do fancy motion design with flat designs.[/QUOTE]
I think it's quite annoying that most people here are focusing so much on design. I'd really like to see some more back-end codes around here. I'm not the artistic kind of person, which is why I don't know anything about design but lots about back-end.
Eye-candy to me are back-end workarounds and libraries.
[url]https://dl.dropboxusercontent.com/u/1963496/Facepunch_Profile_Ratings_CLIENT.user.js[/url]
first version
[QUOTE=Svenskunganka;41882955]I think it's quite annoying that most people here are focusing so much on design. I'd really like to see some more back-end codes around here. I'm not the artistic kind of person, which is why I don't know anything about design but lots about back-end.
Eye-candy to me are back-end workarounds and libraries.[/QUOTE]
I'm developing a PHP Framework/library, if you want me to post some of that then I've got no problem doing so.
[QUOTE=Dorkslayz;41883276]I'm developing a PHP Framework/library, if you want me to post some of that then I've got no problem doing so.[/QUOTE]
I'd love to see that! =)
Working on a silent letter parser as I'm bored, only spent about 30 minutes on it, seems pretty interesting learning about
"Romantic" Rhyming phrases vs Germanic ones and how they affect silent constructs.
[img]http://img837.imageshack.us/img837/5487/p6b3.png[/img]
[QUOTE=Svenskunganka;41883696]I'd love to see that! =)[/QUOTE]
Okay.
So i'm writing a PHP function library that's similar to CakePHP/Codeigniter in it's objective to reduce the amount of code that the developer/user has to write. The problem with these frameworks is that they're too bulky and they don't actually explain what they do. So i'm focusing my library on being simple, it's not meant to be something that does everything for you, it's supposed to do the simple stuff that you shouldn't have to write.
At the moment it's got classes to deal with Data, Encryption, Dependencies* and Email and i'm planning to add a few others like User stuff, etc.
* The dependencies class is built so that dependency requirement is automatic and doesn't require any interaction from the user. You can feed it a bunch of parameters (or use the preset ones) and it'll search the library dependency folder for the specified dependency, and if it can't find it then it downloads it from Github (provided you specify a GitHub URL, Bitbucket support is coming soon). Example usage would be something like $DependencyManager = new Dependency_Manager; $DependencyManager->RequireDependency("PHPMailer"); and PHPMailer would be automatically included in your project, even if it doesn't exist (yet).
I'd post the GitHub link to the project but it's not in a workable state and it's currently a private project - but once it's in a state that I can be proud of then i'll release it.
Opinions?
[QUOTE=Dorkslayz;41883840]Okay.
So i'm writing a PHP function library that's similar to CakePHP/Codeigniter in it's objective to reduce the amount of code that the developer/user has to write. The problem with these frameworks is that they're too bulky and they don't actually explain what they do. So i'm focusing my library on being simple, it's not meant to be something that does everything for you, it's supposed to do the simple stuff that you shouldn't have to write.
At the moment it's got classes to deal with Data, Encryption, Dependencies* and Email and i'm planning to add a few others like User stuff, etc.
* The dependencies class is built so that dependency requirement is automatic and doesn't require any interaction from the user. You can feed it a bunch of parameters (or use the preset ones) and it'll search the library dependency folder for the specified dependency, and if it can't find it then it downloads it from Github (provided you specify a GitHub URL, Bitbucket support is coming soon).
I'd post the GitHub link to the project but it's not in a workable state and it's currently a private project - but once it's in a state that I can be proud of then i'll release it.
Opinions?[/QUOTE]
I would personally advise using composer for your dependencies, provide the composer.json and that's basically it. Saves you having to deal with that, you can then even put your own system on packagist.
Re the other stuff, would be good to have a nice router. Personally keep it array based and outside of YAML, etc for speed etc. Sure the overheads aren't much but stacked up they become a pain.
[QUOTE=graymic;41883878]I would personally advise using composer for your dependencies, provide the composer.json and that's basically it. Saves you having to deal with that, you can then even put your own system on packagist.
Re the other stuff, would be good to have a nice router. Personally keep it array based and outside of YAML, etc for speed etc. Sure the overheads aren't much but stacked up they become a pain.[/QUOTE]
I've heard about Composer before, but this is kinda aimed towards working with the basics, no extra packages (such as Composer) but thanks for the suggestion anyway.
Can you elaborate on what you mean by "a nice router"?
[QUOTE=Dorkslayz;41884056]I've heard about Composer before, but this is kinda aimed towards working with the basics, no extra packages (such as Composer) but thanks for the suggestion anyway.
Can you elaborate on what you mean by "a nice router"?[/QUOTE]
Sure, that makes sense.
By a router I mean a class which allows you to set up the application routing nicely.
So for instance if you're thinking of allowing event dispatching / filtering
[code]
static $routes = array (
'/' => array ('controller' => '\App\Controller\Main', 'action' => 'Index', 'filter' => 'framework.before')
)[/code]
That's the way my one works at least, probably not the best, but allows me to define HTTP routes, there's an optional field for 'methods' => 'GET|POST|DELETE' as an example.
[QUOTE=Dorkslayz;41883840]Okay.
So i'm writing a PHP function library that's similar to CakePHP/Codeigniter in it's objective to reduce the amount of code that the developer/user has to write. The problem with these frameworks is that they're too bulky and [B]they don't actually explain what they do[/B].[/QUOTE]
What? Those both have pretty good documentation. I learned how MVC works with cakePHP and I was able to just dive right in because the docs were great.
You should post up the github link now so people can give you feedback on what you're doing earlier on.
[QUOTE=Dorkslayz;41883840]Okay.
So i'm writing a PHP function library that's similar to CakePHP/Codeigniter in it's objective to reduce the amount of code that the developer/user has to write. The problem with these frameworks is that they're too bulky and they don't actually explain what they do. So i'm focusing my library on being simple, it's not meant to be something that does everything for you, it's supposed to do the simple stuff that you shouldn't have to write.
At the moment it's got classes to deal with Data, Encryption, Dependencies* and Email and i'm planning to add a few others like User stuff, etc.
* The dependencies class is built so that dependency requirement is automatic and doesn't require any interaction from the user. You can feed it a bunch of parameters (or use the preset ones) and it'll search the library dependency folder for the specified dependency, and if it can't find it then it downloads it from Github (provided you specify a GitHub URL, Bitbucket support is coming soon). Example usage would be something like $DependencyManager = new Dependency_Manager; $DependencyManager->RequireDependency("PHPMailer"); and PHPMailer would be automatically included in your project, even if it doesn't exist (yet).
I'd post the GitHub link to the project but it's not in a workable state and it's currently a private project - but once it's in a state that I can be proud of then i'll release it.
Opinions?[/QUOTE]
Sounds very useful to me as I'm always writing those very basic functions. Do you have any function for preparing SQL statements? ^^. I hate preparing 'em, even though it's very good to do.
[QUOTE=KmartSqrl;41884297]What? Those both have pretty good documentation. I learned how MVC works with cakePHP and I was able to just dive right in because the docs were great.
You should post up the github link now so people can give you feedback on what you're doing earlier on.[/QUOTE]
I'd like to see the real codes aswell.
[QUOTE=Dorkslayz;41884056]I've heard about Composer before, but this is kinda aimed towards working with the basics, no extra packages (such as Composer) but thanks for the suggestion anyway.[/QUOTE]
I would use composer too if I were you, "no extra packages" is kind of a silly goal for a development project and something like composer is [I]always[/I] going to be useful. Avoiding using third party libraries without a really good reason is just creating more work for yourself and wasting your time.
[QUOTE=Dorkslayz;41884056]Can you elaborate on what you mean by "a nice router"?[/QUOTE]
I kind of get the feeling that you've tried to play around with a framework like cake before and not really done it long enough to understand what it's doing for you and why things are done the way they are. You should take the time to really get to know a few frameworks before you write your own.
[QUOTE=Svenskunganka;41884304]Sounds very useful to me as I'm always writing those very basic functions. Do you have any function for preparing SQL statements? ^^. I hate preparing 'em, even though it's very good to do.
I'd like to see the real codes aswell.[/QUOTE]
Just write a PDO wrapper, look into some ORM solutions. Ultimately you're just going to have to write a bunch of helpers. I do something like
$result = $this->db->select("SELECT * FROM table WHERE a = ? AND b = ?", array ($a, $b));
(Edit: Which I know isn't an ORM by the way...)
KmartSqrl probably is the guy to listen to though.
[QUOTE=graymic;41884282]Sure, that makes sense.
By a router I mean a class which allows you to set up the application routing nicely.
So for instance if you're thinking of allowing event dispatching / filtering
[code]
static $routes = array (
'/' => array ('controller' => '\App\Controller\Main', 'action' => 'Index', 'filter' => 'framework.before')
)[/code]
That's the way my one works at least, probably not the best, but allows me to define HTTP routes, there's an optional field for 'methods' => 'GET|POST|DELETE' as an example.[/QUOTE]
Yeah, that's something i'll definitely add. I had some basic stuff like that but it only dealt with redirects.
[editline]18th August 2013[/editline]
[QUOTE=KmartSqrl;41884297]What? Those both have pretty good documentation. I learned how MVC works with cakePHP and I was able to just dive right in because the docs were great.
You should post up the github link now so people can give you feedback on what you're doing earlier on.[/QUOTE]
Sorry, I worded my post a bit wrong. When I visited both CakePHP and CodeIgniters sites there wasn't really a brief description of what they actually did, how they would be useful to me, etc. I'm sure both frameworks are stellar at what they do - but I couldn't find a list of things it covers, if you understand what i'm trying to say. I'm planning to release the GitHub link probably tomorrow once i've cleaned up one of the classes that is completely broken.
[editline]18th August 2013[/editline]
[QUOTE=Svenskunganka;41884304]Sounds very useful to me as I'm always writing those very basic functions. Do you have any function for preparing SQL statements? ^^. I hate preparing 'em, even though it's very good to do.
I'd like to see the real codes aswell.[/QUOTE]
Unfortunately not, but there is a SQL "factory" that you can put data into, select what you want to do with it and it outputs the SQL.
[editline]18th August 2013[/editline]
[QUOTE=KmartSqrl;41884356]I would use composer too if I were you, "no extra packages" is kind of a silly goal for a development project and something like composer is [I]always[/I] going to be useful. Avoiding using third party libraries without a really good reason is just creating more work for yourself and wasting your time.
I kind of get the feeling that you've tried to play around with a framework like cake before and not really done it long enough to understand what it's doing for you and why things are done the way they are. You should take the time to really get to know a few frameworks before you write your own.[/QUOTE]
I get where you're coming from, and composer is really good for package/dependency management, but I don't want to label my library as something that requires another library/package/whatever, I don't want the developer to have to work around it. Honestly, I've not really used PHP Frameworks before, I started this off as just a little side project to suit my own needs, but decided to take it a step further - that's why i'm not labeling it as a framework as such, more of a library.
Writing a basic framework is a good way to learn decent PHP OOP and can be good for your own small projects. Bare in mind that for anything serious you probably should use one of the established frameworks.
Here's mine [url]https://github.com/jonathanredwood/noodles[/url] that I've been slowly updating for a couple of years now as the backend for my garry's mod server site ([url]http://electronoodles.co.uk/[/url]).
[url]http://electronoodles.co.uk/?dev=1[/url] enables the developer console which shows some of what is going on.
[QUOTE=Dorkslayz;41884650]I get where you're coming from, and composer is really good for package/dependency management, but I don't want to label my library as something that requires another library/package/whatever, I don't want the developer to have to work around it. Honestly, I've not really used PHP Frameworks before, I started this off as just a little side project to suit my own needs, but decided to take it a step further - that's why i'm not labeling it as a framework as such, more of a library.[/QUOTE]
Download cake or laravel and spend a couple hours doing a getting started tutorial for one of them. It'll be a good learning experience even if you decide to make your own still. You might get some ideas for what you're building as well.
hey guys, this is my latest side project:
[url]http://curate.im[/url]
It's a site that allows you to collect interesting content by creating collections of links on whatever topic you want, and it lets others discover interesting content by viewing your collections of links.
Let me know what you think : )
Finally set up data backups for spoonfedstyle.com today. I was bad/lazy and didn't do it for quite a while but I got everything working really fast today thanks to the 'backup' gem ([url]https://github.com/meskyanichi/backup[/url])
Redis and postgres data both get exported and gzipped and then it gets uploaded to s3 for safe keeping at midnight every night.
[editline]18th August 2013[/editline]
[QUOTE=Alcapwne;41885846]hey guys, this is my latest side project:
[url]http://curate.im[/url]
It's a site that allows you to collect interesting content by creating collections of links on whatever topic you want, and it lets others discover interesting content by viewing your collections of links.
Let me know what you think : )[/QUOTE]
I like it so far, assuming it's not done since the search doesn't do anything. Are you planing on making the search happen automatically in that side area like google does now? I think I would if it were my project. Would be cool if you could link directly to images and have them show as a thumbnail or have media like videos/music (links directly to files like mp3s and youtube/soundcloud/whatever) embedded so you don't have to actually open the links to engage with everything in a list. You'd need to find a good way to give links directly to pages enough emphasis that the videos/thumbs/etc don't drown them out though.
[QUOTE=Dorkslayz;41884650]
I get where you're coming from, and composer is really good for package/dependency management, but I don't want to label my library as something that requires another library/package/whatever, I don't want the developer to have to work around it. Honestly, I've not really used PHP Frameworks before, I started this off as just a little side project to suit my own needs, but decided to take it a step further - that's why i'm not labeling it as a framework as such, more of a library.[/QUOTE]
No modern PHP developer will use a third-party library/package/framework that is not composer-ready, I can assure you of that.
You don't want to "complicate" your library with composer, but the alternative is forcing developers using your library to jump through some pretty shitty hoops:
- Your library's files have to be included directly in the codebase (a-la last decade's lib or includes directories, yuck!), or hidden away in git submodules. (composer handles this for you)
- Custom autoloading mechanics have to be included just to use your library. (composer handles this for you, with the generated autoload.php file).
- Developers using your library have to manage your package's versioning manually (composer allows you to define version constraints in your composer.json, and install or update packages with a single command).
This is an opportunity for you to learn something useful for you, as a guy working with PHP. You can try to dodge the matter as much as you want, and choose not to package your library with composer, but it will be the wrong choice.
The documentation is piss easy to follow for all skill levels, and there are additional tutorials around if you need something more hand-holdy - no excuses: [URL]http://getcomposer.org/doc/[/URL]
Sorry, you need to Log In to post a reply to this thread.
