Case Study: How Compare Pet Products Uses Content Egg to Automate Price Comparisons

When we started building Compare Pet Products, our goal was simple: help Australians find the cheapest place to buy dog and cat products – from food to flea and worming treatments and more, all in one place.

But creating a nationwide comparison site that automatically collects live prices from dozens of different online pet stores is anything but simple. Each store has a unique layout, uses different technologies, and they handle things in different ways (for example, out of stock products and product variations). Manually creating and coding scrapers ourselves for each site would have taken months and been almost impossible to sustain.

That’s where KeywordRush and their tools Content Egg Pro and Affiliate Egg Pro became absolutely essential.

Why We Chose Keyword Rush

Affiliate Egg Pro and Content Egg Pro gave us the foundation to build a large-scale, automated comparison site on WordPress. The combination of flexibility, extensibility, and reliability meant we could focus on user experience, adding features and growth rather than data extraction.

The real breakthrough for us came when we discovered KeywordRush’s custom parser service. For about $45 AUD per store, their team develops a parser that perfectly integrates a merchant’s data onto our site.

Instead of writing our own scraping scripts for every store, we simply provide a merchant’s URL (and sometimes details about handling variations of products) and within a day or two, KeywordRush delivers a fully functional parser that automatically retrieves prices, titles, product descriptions, photos and stock.

This approach has saved us hundreds of hours and allowed us to scale our site a lot more quickly than without them.

What about using an API or Data Feeds

Before settling on custom parsers, we tried contacting dozens of pet stores to request API or data feedaccess. Practically all of them didn’t reply but of those who did, none gave us API access and only about 15%  of stores offered us access to a data feed. Some of the data feeds were made available to us through third party affiliate websites. We initially started using the data feeds where we could, but we found them to be unreliable:

• Some feeds didn’t include all products, or the stores managing their feeds would randomly remove a product from it that was still available on their site
• Product prices weren’t always updated quickly
• Some feeds only updated every 24 hours and our site checks prices every 24 hours, which meant our website prices were sometimes be up to 48 hours old

One of the main lessons we learned: start with custom parsers from day one. They provide more accurate, up-to-date data, cover all products, and are more reliable than data feeds, especially as pet product prices change regularly.

Customer support

We use several technical products and plugins on our WordPress site and have found that one of the most important things is to use products that have good documentation and customer support available. Otherwise we’d be stuck using technology we don’t know how to use. The KeywordRush support team is one of the best we’ve ever dealt with. They typically respond within 24 hours, often much sooner, and their replies always contain the exact help or solution we need.

For a technical product like this, that kind of consistent support makes a huge difference. It’s allowed us to build our site better and faster.

Handling Stock Status and Product Variations

Automatic stock status detection has been a huge time-saver for us and a big benefit for our website users. Around 90% of our stores provide accurate stock updates through their parsers. The remaining 10% generally use JavaScript to load product variations (for example, 2kg, 4kg, and 10kg pet food bags on a single product page).

Because these pages rely on dynamic loading, the parser sometimes can’t access every variation. One useful trick we’ve found that works for most of the stores (like Pet Chemist) where the URL doesn’t change when you click between variations are:

1. Go to the store’s search results page
2. Click the variation you want
3. Click the product to go to the page with it’s own URL and use it for the parser

This worked for most of sites that didn’t have a clear URL for each variation.

Optimizing Performance and Reliability

At first, we set our Content Egg Pro modules (the custom parsers) to update products every time a user clicked a product link. That worked well early on but slowed page loads once each product got more comparisons.

We now use daily scheduled updates instead. It provides accurate pricing while maintaining fast load times and was also recommended by the KeywordRush team.

We also wanted to avoid being blocked by merchants for sending too many requests from one IP address. To manage that, we integrated ScrapingDog, a third-party proxy rotation service that works seamlessly with Affiliate Egg and Content Egg. It automatically rotates IPs and manages delays between requests. It was not very cheap, but it’s extremely reliable and was easy to set up. We were also originally blocked completely from being able to scrape a few websites and ScrapingDog helped us bypass the block.

Managing Out-of-Stock Items

We chose to keep out-of-stock offers visible on our site. Using Content Egg’s settings, we set out-of-stock handling to “do nothing,” then added a small piece of code that labels those offers as “Out of Stock.”

This approach helps users see which stores normally stock the product and creates a more consistent comparison experience even when availability changes. This being said, if all offers for a product are out-of-stock, we automatically hide the product from our site.

Presentation and Branding

Content Egg lets us assign a merchant logo to each store, and we uploaded high-resolution versions of each one. It’s a small detail, but we think it makes the comparisons on our site look professional, polished, and trustworthy. We specifically uploaded high-resolution versions of the logos to our site and link to those, rather than linking directly to the merchants’ logos, in case they change their files or file paths.

Combining Automation with Manual Comparisons

While much of our system is automated through Affiliate Egg and Content Egg, we also maintain a significant manual comparison process.

We currently have more than 20,000 manual comparisons across thousands of products. One of the main challenges is that different pet stores (and our site) use different naming conventions for the exact same product and variation. For example, currently on the Compare Pet Products website, we list a product named Hill’s Science Diet Adult Chicken Dry Dog Food 12kg. As of time of writing they are listed on the following websites with the 12kg being text in buttons:

• Pet Circle as “Hills Science Diet Adult Dry Dog Food”
• Vets Love Pets as “Hill’s Science Diet Chicken & Barley Recipe Adult Dog Dry Food”
• Petbarn as “Hill’s Science Diet Adult Dog Food Chicken”

Each merchant has its own naming rules, and these conventions can even change by brand or product type.

Rather than relying entirely on fuzzy matching (which risks linking products incorrectly or creating duplicate products on our site), we use fuzzy matching in spreadsheets to manually narrow down potential matches, then manually verify each product. This ensures the data on our site is as accurate, consistent, and trustworthy as possible.

In addition, for bulk updates such as assigning product attributes like dry food, wet food, or types of flea and worming treatments (tablets, liquids, etc.), we export the products into a CSV file, make the necessary changes, and then import them back into WordPress. This saves a lot of time compared to updating products one by one with the UI.

Performance Tips and Workflow Advice

Every website is different and what works for us may not work for you. But from our experience creating and running a large pet product price comparison website, here are a few practical tips for other developers and website owners using or looking to use Affiliate Egg or Content Egg for a similar purpose:

• Use custom parsers instead of data feeds or building your own scrapers – it’s faster, cheaper, and more reliable
• Schedule updates daily rather than on every click to improve load times
• Integrate proxy rotation early to prevent IP bans
• Keep out-of-stock listings visible to improve transparency
• Manual verification may be necessary if merchants use inconsistent naming for the same products
• Use CSV bulk updates for attributes to save time and ensure consistent categorization

Real-World Results

Using Affiliate Egg Pro and Content Egg Pro with KeywordRush’s custom parsers, we now manage thousands of products across dozens of merchants with minimal manual work.

The setup automatically:

• Checks price updates daily
• Displays dynamic comparison tables
• Flags out-of-stock items
• Integrates smoothly with WordPress caching and optimization plugins

Thanks to Affiliate Egg & Content Egg Pro, we have been able to build Compare Pet Products into a platform that helps Australians quickly find the best deals on dog food, cat food and thousands of other pet essentials.

Affiliate Egg Pro and Content Egg Pro have turned what would have been a complex, resource-heavy project into a more scalable, maintainable system that runs efficiently and reliably.

Without KeywordRush’s tools, Compare Pet Products simply wouldn’t exist at the scale we operate today.

Their combination of technical flexibility, automation, and outstanding support has allowed us to focus on improving user experience, expanding our database, and helping Australians save money on pet care.

If you’re planning to build a comparison site, for retail, tech, travel, or other niche markets, Affiliate Egg Pro and Content Egg Pro could not be more recommended. They provide all the structure you need to create a live, automated, and scalable comparison platform with confidence.

---

We wanted to dig a bit deeper into Brandon’s experience building Compare Pet Products with Content Egg and Affiliate Egg Pro — so we threw a few fun questions his way. Here’s what he shared!