Curl and screen scraping advice please :) - Hi All, as im sure some of you are aware im setting up a price comparison feature from an api

ljackson · 4th February 2010, 09:06 PM

Hi All,

as im sure some of you are aware im setting up a price comparison feature from an api protocal which stores millions of products,

the main problem with this is that its not very reliable for example if i search modern warfare 2 on xbox 360 it might return 10 stores which sells it but half the products it finds are the communicator and there is no record of the actual game from particular merchants. for example play and zavvi dont have the game listed that i can find, so i was wondering how easy curl and screen scraping was and if im more likley to get better result set :)

i have found a website Find DVD - Compare DVD prices from dozens of UK retailers which seems to be spot on when it comes to the price comparison all the links link to the actual product im searching for but im not sure how they do it :)

any advice would be appreciated :)
cheers
Luke

CloudedVision · 6th February 2010, 01:13 AM

Well, first you would need a spider:

cURL to a product page
Parse it with an XML/HTML parser. The default PHP is rather complex, but this looks promising: PHP Simple HTML DOM Parser
Find price, product picture, etc.
Put into a database

And then users can search the database on your website. Here's a good link on how to create a PHP search function. PHP search engine

ljackson · 7th February 2010, 09:47 PM

interesting.. thanks mate

4th February 2010, 09:06 PM	#1
ljackson Member Join Date: Feb 2009 Location: Cornwall Posts: 270 Thanks: 14 Thanked 1 Time in 1 Post Expertise: PHP Experience: Intermediate	Curl and screen scraping advice please :) Hi All, as im sure some of you are aware im setting up a price comparison feature from an api protocal which stores millions of products, the main problem with this is that its not very reliable for example if i search modern warfare 2 on xbox 360 it might return 10 stores which sells it but half the products it finds are the communicator and there is no record of the actual game from particular merchants. for example play and zavvi dont have the game listed that i can find, so i was wondering how easy curl and screen scraping was and if im more likley to get better result set :) i have found a website Find DVD - Compare DVD prices from dozens of UK retailers which seems to be spot on when it comes to the price comparison all the links link to the actual product im searching for but im not sure how they do it :) any advice would be appreciated :) cheers Luke __________________ www.kernow-connect.com - Online Shopping \| Price Comparison \| Discount Codes

6th February 2010, 01:13 AM	#2
CloudedVision Join Date: Jan 2009 Location: Your Imagination Posts: 624 Blog Entries: 4 Thanks: 1 Thanked 22 Times in 21 Posts Expertise: PHP Experience: Professional	Well, first you would need a spider: cURL to a product page Parse it with an XML/HTML parser. The default PHP is rather complex, but this looks promising: PHP Simple HTML DOM Parser Find price, product picture, etc. Put into a database And then users can search the database on your website. Here's a good link on how to create a PHP search function. PHP search engine __________________ Panthr - Web Design and Development

7th February 2010, 09:47 PM	#3
ljackson Member Join Date: Feb 2009 Location: Cornwall Posts: 270 Thanks: 14 Thanked 1 Time in 1 Post Expertise: PHP Experience: Intermediate	interesting.. thanks mate __________________ www.kernow-connect.com - Online Shopping \| Price Comparison \| Discount Codes

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode