How to Scrape Yelp Data: Business Info, Reviews and more.

Introduction

Yelp is a popular online directory for discovering local businesses ranging from bars, restaurants, and cafes to hairdressers, spas, and gas stations.

Why do we may need to scrape Yelp?
Collecting data from Yelp will help us to perform:

competitor analysis
sentiment analysis
lead generation
price monitoring

To scrape Yelp data, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

In this article, we will learn how to:

Scrape Yelp Business Information
Scrape Yelp Reviews

Prerequisites

To start scraping Yelp, we will need the following things:

A Page2API account
A category of businesses in a specific location that we are about to scrape.
In our case, we will search for Restaurants in Redwood City, CA, and then pick a restaurant and scrape its reviews.

How to scrape Yelp business information

First what we need is to open yelp.com and type 'Restaurants' into the search input from the Yelp search page and pick the location we need.

This will change the browser URL to something similar to:

  
    https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood+City%2C+CA

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

The HTML for a single result element will look like this:

Now let's handle the pagination.

To go to the next page, we must click on the next page link if it's present on the page:

  
    var next = document.querySelector('.next-link'); if(next) { next.click() }

From this page, we will scrape the following attributes from each business:

Name
URL
Reviews count
Stars
Thumbnail image
Tags

Now, let's define the selectors for each attribute.

  
    /* Parent: */
    //*[@id='main-content']/div/ul/li[*]

    /* Name */
    h3

    /* URL */
    h3 a

    /* Reviews count */
    span[class^=reviewCount]

    /* Stars */
    div[role=img]

    /* Thumbnail image */
    img

    /* Tags */
    a[role=link] button p

Let's build the request that will scrape all the results that the search page returned.

Setting the api_key as an environment variable

  
    export API_KEY=YOUR_PAGE2API_KEY

The payload for our scraping request will be:

  
    {
      "url": "https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood%20City%2C%20TX",
      "premium_proxy": "us",
      "merge_loops": true,
      "scenario": [
        {
          "loop": [
            { "wait_for": "h3" },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector('.next-link'); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "places": [
          {
            "_parent": "//*[@id='main-content']/div/ul/li[*]",
            "_require": ["name", "url"],
            "name": "h3 a >> text",
            "url": "h3 a >> href",
            "reviews_count": "span[class^=reviewCount] >> text",
            "stars": "div[role=img] >> aria-label",
            "thumbnail": "img >> src",
            "tags": ["a[role=link] button p >> text"]
          }
        ]
      },
      "real_browser": true
    }

Running the scraping request with cURL

  
    curl -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood%20City%2C%20TX",
      "premium_proxy": "us",
      "merge_loops": true,
      "scenario": [
        {
          "loop": [
            { "wait_for": "h3" },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector(\".next-link\"); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "places": [
          {
            "_parent": "//*[@id=\"main-content\"]/div/ul/li[*]",
            "_require": ["name", "url"],
            "name": "h3 a >> text",
            "url": "h3 a >> href",
            "reviews_count": "span[class^=reviewCount] >> text",
            "stars": "div[role=img] >> aria-label",
            "thumbnail": "img >> src",
            "tags": ["a[role=link] button p >> text"]
          }
        ]
      },
      "real_browser": true
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  
    {
      "result": {
        "places": [
          {
            "name": "Palmer’s Restaurant Bar & Courtyard",
            "url": "https://www.yelp.com/biz/palmers-restaurant-bar-and-courtyard-san-marcos?osq=Restaurants",
            "reviews_count": "520",
            "stars": "4 star rating",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/qNfJPDa27awAllb2Wa07pQ/348s.jpg",
            "tags": [
              "Bars",
              "American (Traditional)",
              "Steakhouses"
            ]
          },
          {
            "name": "North Street",
            "url": "https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants",
            "reviews_count": "139",
            "stars": "4.5 star rating",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/Ci3mK0aF2UFzkauMlIZaCg/348s.jpg",
            "tags": [
              "Indian",
              "Beer Bar",
              "Coffee & Tea"
            ]
          }, ...
        ]
      }, ...
    }

How to scrape Yelp reviews

First what we need is to open any URL from the previous step.

This will change the browser URL to something similar to:

  
    https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants

The URL is the first parameter we need to perform the reviews scraping.

Luckily, the pagination handling is similar to the one described in the previous step, so we will use the same flow.

From this page, we will scrape the following attributes from each business:

Name
Tags
Phone number
Address
Reviews count
Stars

and the following fields for each review:

User
Location
Stars
Content