How to Scrape Amazon Data: Products, Pricing, Reviews, etc.

2021-10-19 - 6 min read
Nicolae Rotaru
Nicolae Rotaru
@NRotaru

Introduction

Amazon.com is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon.com’s millions of customers.


In this article, you will read about the easiest way to scrape Amazon products and reviews with Page2API.


You will find code examples for Ruby, Python, PHP, NodeJS, cURL, and a No-Code solution that will import Amazon products into Google Sheets.


Amazon scraping can be very useful if you want to accomplish such tasks as:

  • competitor analysis
  • improving your products and value proposition
  • identifying market trends and what influences them
  • price monitoring


Luckily, Amazon.com is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.


In this article, we will learn how to:

  • Scrape Amazon products
  • Scrape Amazon reviews

Prerequisites

To perform this simple task, you will need the following things:


  • A Page2API account
    The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.

  • A product or a category of products that we are about to scrape.
    In our case, we will search for 'luminox watches'
    and then scrape the reviews for a random product.

How to scrape Amazon products

First what we need is to type 'luminox watches' into the search input from amazon's search page.


This will change the browser URL to something similar to:

  
    https://www.amazon.com/s?k=luminox+watches

The URL is the first parameter we need to perform the scraping.


The page that you see must look like the following one:

Amazon results page

If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

Amazon result

The HTML for a single result element will look like this:

Amazon result breakdown

Now, let's handle the pagination.
There are two approaches that can help us scrape all the needed pages:

1. We can iterate through the pages by clicking on the Next page button
2. We can scrape the pages using the batch scraping feature

Let's take a look at the first approach.


In our case, we must click on the Next → button while the list item's class will be active: 

  
    document.querySelector('.s-pagination-next').click()

Next button is enabled

And stop our scraping request when the Next → button became disabled.

In our case, a new class (.s-pagination-item.s-pagination-next.s-pagination-disabled) is assigned to the list element where the button is located:

Next button is enabled
The stop condition for the scraper will be the following javascript snippet: 
  
    document.querySelector('.s-pagination-item.s-pagination-next.s-pagination-disabled') !== null

Now it's time to prepare the request that will scrape all products that the search page returned.

The following examples will show how to scrape 3 pages of products from Amazon.com

If we decide to go with the next button approach, our payload will look like:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": "[data-component-type=s-search-result]" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector('.s-pagination-next').click()" }
          ],
          "stop_condition": "document.querySelector('.s-pagination-item.s-pagination-next.s-pagination-disabled') !== null",
          "iterations": 3
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }

Code examples (next button approach)

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape 3 pages of products from Amazon.com

    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: "https://www.amazon.com/s?k=luminox+watches",
      real_browser: true,
      merge_loops: true,
      scenario: [
        {
          loop: [
            { wait_for: "[data-component-type=s-search-result]" },
            { execute: "parse" },
            { execute_js: "document.querySelector('.s-pagination-next').click()" }
          ],
          stop_condition: "document.querySelector('.s-pagination-item.s-pagination-next.s-pagination-disabled') !== null",
          iterations: 3
        }
      ],
      parse: {
        watches: [
          {
            _parent: "[data-component-type=s-search-result]",
            title: "h2 >> text",
            link: ".a-link-normal >> href",
            price: ".a-price-whole >> text",
            stars: ".a-icon-alt >> text"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

If we decide to go with the batch scraping approach, our payload will look like:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "batch": {
        "urls": "https://www.amazon.com/s?k=luminox+watches?page=[1, 3, 1]",
        "concurrency": 1,
        "merge_results": true
      },
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type=s-search-result]",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }

Code examples (batch scraping approach)

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape 3 pages of products from Amazon.com

    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      batch: {
        urls: "https://www.amazon.com/s?k=luminox+watches?page=[1, 3, 1]",
        concurrency: 1,
        merge_results: true
      },
      parse: {
        watches: [
          {
            _parent: "[data-component-type=s-search-result]",
            title: "h2 >> text",
            link: ".a-link-normal >> href",
            price: ".a-price-whole >> text",
            stars: ".a-icon-alt >> text"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

The result

  
    {
      "result": {
        "watches": [
          {
            "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
            "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
            "price": "$223.47",
            "stars": "4.5 out of 5 stars"
          },
          {
            "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
            "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
            "price": "$254.12",
            "stars": "4.3 out of 5 stars"
          },
          {
            "title": "Leatherback SEA Turtle Giant - 0323",
            "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
            "price": "$179.00",
            "stars": "4.3 out of 5 stars"
          },
          ...
        ]
      },
      ...
    }

How to scrape Amazon reviews

First what we need is to click on the See all reviews link from the Product page.


This will change the browser URL to something similar to:

  
    https://www.amazon.com/product-reviews/B072FNJLBC

The URL is the first parameter we need to perform the reviews scraping.


The HTML from a single review will look like this:

Amazon review HTML

Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.

If we decide to go with the next button approach, our payload will look like:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector('.a-pagination li.a-last a').click()" }
          ],
          "stop_condition": "document.querySelector('.a-last.a-disabled') !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }

Code examples (next button approach)

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape multiple pages of reviews from Amazon.com

    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: "https://www.amazon.com/product-reviews/B072FNJLBC",
      real_browser: true,
      merge_loops: true,
      scenario: [
        {
          loop: [
            { wait_for: "[data-hook=review]" },
            { execute: "parse" },
            { execute_js: "document.querySelector('.a-pagination li.a-last a').click()" }
          ],
          stop_condition: "document.querySelector('.a-last.a-disabled') !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            _parent: "[data-hook=review]",
            title: ".review-title >> text",
            author: ".a-profile-name >> text",
            stars: ".review-rating >> text",
            content: ".review-text >> text"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

If we decide to go with the batch scraping approach, our payload will look like:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "real_browser": true,
      "batch": {
        "urls": "https://www.amazon.com/product-reviews/B072FNJLBC/?pageNumber=[1, 3, 1]",
        "concurrency": 1,
        "merge_results": true
      },
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook=review]",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }

Code examples (batch scraping approach)

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape 3 pages of reviews from Amazon.com

    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      real_browser: true,
      batch: {
        urls: "https://www.amazon.com/product-reviews/B072FNJLBC/?pageNumber=[1, 3, 1]",
        concurrency: 1,
        merge_results: true
      },
      parse: {
        reviews: [
          {
            _parent: "[data-hook=review]",
            title: ".review-title >> text",
            author: ".a-profile-name >> text",
            stars: ".review-rating >> text",
            content: ".review-text >> text"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

The result

  
    {
      "result": {
        "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
          ...
        ]
      },
      ...
    }

How to export Amazon products to Google Sheets

In order to be able to export our Amazon products to a Google Spreadsheet we will need to slightly modify our request to receive the data in CSV format instead of JSON.

According to the documentation, we need to add the following parameters to our payload: 
  
    "raw": {
      "key": "watches", "format": "csv"
    }

Now our payload will look like:

{ "api_key": "YOUR_PAGE2API_KEY", "real_browser": false, "raw": { "key": "watches", "format": "csv" }, "batch": { "urls": [ "https://www.amazon.com/s?k=luminox+watches?page=1", "https://www.amazon.com/s?k=luminox+watches?page=2", "https://www.amazon.com/s?k=luminox+watches?page=3" ], "concurrency": 1, "merge_results": true }, "parse": { "watches": [ { "_parent": "[data-component-type=s-search-result]", "title": "h2 >> text", "link": ".a-link-normal >> href", "price": ".a-price-whole >> text", "stars": ".a-icon-alt >> text" } ] } }

Please note that the batch URLs are defined explicitly to make it simpler to edit the payload.


Now, edit the payload above if needed, and press Encode →

The URL with encoded payload will be:


  Press 'Encode'

Note: If you are reading this article being logged in - you can copy the link above since it will already have your api_key in the encoded payload.

The final part is adding the IMPORTDATA function, and we are ready to import our Amazon products into a Google Spreadsheet.
  Press 'Encode'

The result must look like the following one:

Amazon products import to Google Sheets

Conclusion

That's pretty much of it!
As we've previously mentioned, Amazon.com is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.

You might also like:

Nicolae Rotaru
Nicolae Rotaru
@NRotaru
2023-05-15 - 6 min read

How to Download Instagram Videos with iPhone Shortcuts

In this article, you will read about the easiest way to download videos from Instagram with iPhone shortcuts and Page2API.

Nicolae Rotaru
Nicolae Rotaru
@NRotaru
2023-05-10 - 9 min read

How to Scrape News Articles and Summarize the Content with AI

In this blog post, we'll explore how to scrape news articles with Page2API and summarize the extracted content using GPT-3.5-turbo

Nicolae Rotaru
Nicolae Rotaru
@NRotaru
2023-04-30 - 10 min read

How to Scrape Trustpilot Reviews and Perform Sentiment Analysis with AI

In this blog post, we will explore the step-by-step process of scraping Trustpilot reviews using Page2API, and then performing sentiment analysis on the extracted data using GPT-3.5-turbo.

Ready to Scrape the Web like a PRO?

1000 free API calls.
Based on all requests made in the last 30 days. 99.85% success rate.
No-code-friendly.
Trustpilot stars 4.5
Trustpilot
Try Now For Free