Data feed file integration price comparison shopping script

Documentation Wiki - - Online merchant datafeed integration for affiliates : DFFDIYPricecomparison

HomePage | Categories :: PageIndex :: RecentChanges :: RecentlyCommented :: Login/Register
Home Affiliate FAQ Merchant FAQ Examples Merchants Networks & Programs Forum Blog Wiki Knowledge Base Login

Why you should not create your own price comparison site from scratch

A bit of advice to those looking to download raw data feed files to create your own price comparison shopping site from scratch:

So many people have asked me "where can I download merchant product data feed files?" I always asked them what they are going to do with those feeds. Many of them will say they would like to create a price comparison shopping site. Since I have been down that road before and have 5 years experience processing data feeds, I would like to share some of my experience about creating your own price comparison site from scratch.

I believe price comparison shopping sites are NOT a DIY (do it yourself) small project. Why? because normally a good price comparison site should at least have 10+ million products, several thousands categories, several hundred merchants and frequent updates. Let's talk about those 10+ million products first. Most database developers would probably agree with me that search through 10+ million product descriptions using text search under 1 second is difficult, not to mention matching products against merchants who sells those products and figuring out lowest price etc. Unless you are ready to spend or have at least 4 database servers and have knowledge of full-text search indexing + parallel SQL queries against a database farm, I recommend to think twice.
My point is ... price comparison shopping functionality is expensive and difficult to create and maintain. Why recreate the wheel to create your own system to import, store, process and search through your own database again?

Another issue you will encounter is business development / relationship with merchants. If you are thinking of establishing direct relationship with merchants you should while creating your web site from scratch you should probably research the difficulty of this step.

Next are the daily chores of managing products, merchants imports, daily database refresh from hundreds of merchants data feed files, billing, etc, etc...

Buying ready-made price comparison software
There are a few company selling PHP or ASP software package to create price comparison shopping site. I am not going to mention any name, but I have evaluated many of them as many of those software owners have spoken to me before with issues they have encountered. The first major drawback of those software package is the fact that they are designed to be run on a server or a webserver + a database server. There is no capability for distributing data and performing parallel query. As soon as the total products in your database reach 2+ millions you will notice your database server slowing down while doing text search (for most single database server). At 5+ million products most SQL servers takes 3+ seconds to return search results. Add a good amount of traffic then your site will be very-very slow.

I am not here to crush your dream, I am a technologist, programmer and most importantly a realist after several dot-com ventures. I would like to offer my experience and advice to those who would like to consider it. These are just my opinion.

On the other hand, if your budget and skills fits the description I described above maybe you should. :-)

There are no comments on this page. [Add comment]

Integrate price comparison shopping data feed file from thousands of online merchant product data feeds | Valid CSS

Page was generated in 0.0132 seconds