How to Scrape Yelp Data: Business Info, Reviews and more.


2021-12-07 - 5 min read

Nicolae Rotaru
Nicolae Rotaru

Introduction

Yelp is a popular online directory for discovering local businesses ranging from bars, restaurants, and cafes to hairdressers, spas, and gas stations.


Why do we may need to scrape Yelp?
Collecting data from Yelp will help us to perform:

  • competitor analysis
  • sentiment analysis
  • lead generation
  • price monitoring


To scrape Yelp data, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.


In this article, we will learn how to:

  • Scrape Yelp Business Information
  • Scrape Yelp Reviews

Prerequisites

To start scraping Yelp, we will need the following things:


  • A Page2API account
  • A category of businesses in a specific location that we are about to scrape.
    In our case, we will search for Restaurants in Redwood City, CA, and then pick a restaurant and scrape its reviews.

How to scrape Yelp business information

First what we need is to open yelp.com and type 'Restaurants' into the search input from the Yelp search page and pick the location we need.


This will change the browser URL to something similar to:

  
    https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood+City%2C+CA


The URL is the first parameter we need to perform the scraping.


The page that you see must look like the following one:

Yelp results page

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

Yelp result

The HTML for a single result element will look like this:

Yelp result HTML

Now let's handle the pagination.

To go to the next page, we must click on the next page link if it's present on the page:

  
    var next = document.querySelector('.next-link'); if(next) { next.click() }
  
Yelp next page active From this page, we will scrape the following attributes from each business:

  • Name
  • URL
  • Reviews count
  • Stars
  • Thumbnail image
  • Tags

Now, let's define the selectors for each attribute.

  
    /* Parent: */
    //*[@id='main-content']/div/ul/li[*]

    /* Name */
    h3

    /* URL */
    h3 a

    /* Reviews count */
    span[class^=reviewCount]

    /* Stars */
    div[role=img]

    /* Thumbnail image */
    img

    /* Tags */
    a[role=link] button p
  

Let's build the request that will scrape all the results that the search page returned.

Setting the api_key as an environment variable

  
    export API_KEY=YOUR_PAGE2API_KEY
  

The payload for our scraping request will be:

  
    {
      "url": "https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood%20City%2C%20TX",
      "premium_proxy": "us",
      "merge_loops": true,
      "scenario": [
        {
          "loop": [
            { "wait_for": "h3" },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector('.next-link'); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "places": [
          {
            "_parent": "//*[@id='main-content']/div/ul/li[*]",
            "_require": ["name", "url"],
            "name": "h3 a >> text",
            "url": "h3 a >> href",
            "reviews_count": "span[class^=reviewCount] >> text",
            "stars": "div[role=img] >> aria-label",
            "thumbnail": "img >> src",
            "tags": ["a[role=link] button p >> text"]
          }
        ]
      },
      "real_browser": true
    }
  

Running the scraping request with cURL

  
    curl -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.yelp.com/search?find_desc=Restaurants&find_loc=Redwood%20City%2C%20TX",
      "premium_proxy": "us",
      "merge_loops": true,
      "scenario": [
        {
          "loop": [
            { "wait_for": "h3" },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector(\".next-link\"); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "places": [
          {
            "_parent": "//*[@id=\"main-content\"]/div/ul/li[*]",
            "_require": ["name", "url"],
            "name": "h3 a >> text",
            "url": "h3 a >> href",
            "reviews_count": "span[class^=reviewCount] >> text",
            "stars": "div[role=img] >> aria-label",
            "thumbnail": "img >> src",
            "tags": ["a[role=link] button p >> text"]
          }
        ]
      },
      "real_browser": true
    }' 'https://www.page2api.com/api/v1/scrape' | python3.10 -mjson.tool
  

The result

  
    {
      "result": {
        "places": [
          {
            "name": "Palmer’s Restaurant Bar & Courtyard",
            "url": "https://www.yelp.com/biz/palmers-restaurant-bar-and-courtyard-san-marcos?osq=Restaurants",
            "reviews_count": "520",
            "stars": "4 star rating",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/qNfJPDa27awAllb2Wa07pQ/348s.jpg",
            "tags": [
              "Bars",
              "American (Traditional)",
              "Steakhouses"
            ]
          },
          {
            "name": "North Street",
            "url": "https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants",
            "reviews_count": "139",
            "stars": "4.5 star rating",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/Ci3mK0aF2UFzkauMlIZaCg/348s.jpg",
            "tags": [
              "Indian",
              "Beer Bar",
              "Coffee & Tea"
            ]
          }, ...
        ]
      }, ...
    }
  

How to scrape Yelp reviews

First what we need is to open any URL from the previous step.


This will change the browser URL to something similar to:

  
    https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants


The URL is the first parameter we need to perform the reviews scraping.


Luckily, the pagination handling is similar to the one described in the previous step, so we will use the same flow.

From this page, we will scrape the following attributes from each business:

  • Name
  • Tags
  • Phone number
  • Address
  • Reviews count
  • Stars

and the following fields for each review:

  • User
  • Location
  • Stars
  • Content

Now, let's define the selectors for each attribute.

  
    /* Name */
    h1

    /* Tags */
    /html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/span[3]/span[*]

    /* Phone number */
    //*[contains(text(),'Phone number')]/../p[2]

    /* Address */
    //*[contains(text(),'Get Directions')]/../../p[2]

    /* Reviews count */
    /html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/div[2]/div[2]/span

    /* Stars */
    div[role=img]

    /* --------- */

    /* Reviews attibutes selectors: */

    /* Parent */
    ul li

    /* User */
    .user-passport-info span.fs-block

    /* Location */
    .user-passport-info div

    /* Stars */
    div[role=img]

    /* Content */
    span[lang=en]
  

Now it's time to prepare the request that will scrape Yelp reviews.

The payload for our scraping request will be:

  
    {
      "url": "https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants",
      "merge_loops": true,
      "premium_proxy": "us",
      "scenario": [
        {
          "loop": [
            { "wait_for": ".user-passport-info span.fs-block" },
            { "wait": 1 },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector('.next-link'); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "name": "h1 >> text",
        "tags": ["/html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/span[3]/span[*] >> text"],
        "phone": "//*[contains(text(),'Phone number')]/../p[2] >> text",
        "address": "//*[contains(text(),'Get Directions')]/../../p[2] >> text",
        "reviews_count": "/html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/div[2]/div[2]/span >> text",
        "stars": "div[role=img] >> aria-label",
        "reviews": [
          {
            "_parent": "ul li",
            "_require": ["user"],
            "user": ".user-passport-info span.fs-block >> text",
            "location": ".user-passport-info div >> text",
            "stars": "div[role=img] >> aria-label",
            "content": "span[lang=en] >> text"
          }
        ]
      },
      "real_browser": true
    }
  

Running the scraping request with cURL

  
    curl -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.yelp.com/biz/north-street-san-marcos?osq=Restaurants",
      "merge_loops": true,
      "premium_proxy": "us",
      "scenario": [
        {
          "loop": [
            { "wait_for": ".user-passport-info span.fs-block" },
            { "wait": 1 },
            { "execute": "parse" },
            { "execute_js": "var next = document.querySelector(\".next-link\"); if(next) { next.click() }" }
          ],
          "iterations": 3
        }
      ],
      "parse": {
        "name": "h1 >> text",
        "tags": ["/html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/span[3]/span[*] >> text"],
        "phone": "//*[contains(text(),\"Phone number\")]/../p[2] >> text",
        "address": "//*[contains(text(),\"Get Directions\")]/../../p[2] >> text",
        "reviews_count": "/html/body/yelp-react-root/div[1]/div[3]/div[1]/div[1]/div/div/div[2]/div[2]/span >> text",
        "stars": "div[role=img] >> aria-label",
        "reviews": [
          {
            "_parent": "ul li",
            "_require": ["user"],
            "user": ".user-passport-info span.fs-block >> text",
            "location": ".user-passport-info div >> text",
            "stars": "div[role=img] >> aria-label",
            "content": "span[lang=en] >> text"
          }
        ]
      },
      "real_browser": true
    }' 'https://www.page2api.com/api/v1/scrape' | python3.10 -mjson.tool
  

The result

  
    {
      "result": {
        "name": "North Street",
        "tags": [
          "Indian,",
          "Beer Bar,",
          "Coffee & Tea"
        ],
        "phone": "(512) 667-7094",
        "address": "216 North St San Marcos, TX 78666",
        "reviews_count": "139 reviews",
        "stars": "4.5 star rating",
        "reviews": [
          {
            "user": "Jeri T.",
            "location": "Elite 2021",
            "stars": "5 star rating",
            "content": "My mom and I were looking for a new place in San Marcos ..."
          },
          {
            "user": "Maribel D.",
            "location": "Elite 2021",
            "stars": "5 star rating",
            "content": "Visiting San Marcos and North street popped up on my Yelp top restaurants ..."
          }, ...
        ]
      }, ...
    }
  

Conclusion

Done!
In this article, you've learned how to scrape business information and reviews from Yelp with Page2API - a Web Scraping API that turns pages into JSON with ease.

What customers are saying

Superb support
Superb, reliable support, even out of hours, patient and polite plus educational.
October 21, 2023
Very effective and trustworthy
Very effective and trustworthy!
I had some challenges which were addressed right away.
October 12, 2023
Page2API is without fail my favorite scraping API
Not only does Page2API work without fail constantly, but their customer support team is on a new level.
If i ever have issues integrating or have errors in my code they've always been responsive almost instantly and helped fix any errors.
I've never seen customer service like this anywhere, so massive thanks to the Page2API team.
July 14, 2023
Amazing product and support!
I have tried a lot of different scraping solutions and Page2Api is definitely the best one. It's very developer-friendly and Nick is extremely innovative in coming up with new ideas to solve problems.
The support is unreal as well.
I have sent Nick a request that I have trouble scraping and he's helped me fix all of them. Can highly recommend.
April 13, 2023
This API is amazing and the support was GREAT
This API is amazing and I am very excited to keep using it.
I'm writing this review because I was stumped on a very hard scrape for youtube transcripts, I brought my issue to support and in no time they had written what looks like a very tailored and complicated API call for me, I tested it and it worked perfect! Great great support.
April 19, 2023
Excellent service, super technical support!
I have been looking for such a quality for a long time, I have never met such an individual approach to clients.
Everything is at the highest level!
Nick very quickly helped to deal with all my questions, I am very grateful to him!
Recommend!
February 08, 2023
Fantastic Product and Customer Service
I'm a no-code guy trying to hack it in an API world... so I was pretty apprehensive about what I would be getting into with this.
I'm please to say that the customer service is so fantastic that they got me a solution in under 30 seconds that worked instantly in my application.
They did a great job and it works exactly as advertised.
Highly recommend them!
March 24, 2023
Surprisingly great service and support
I have certainly not come across any other internet initiative in the internet world that provides such good technical support and tries to help even if they are not related to them.
I will take as an example the approach of page2api to the customer in the startups I have founded.
February 16, 2023
Perfect for webcrapping javascript generated webpages
Page2API is perfect to be use from bubble or any other nocode tool.
It works submitting forms, scrapping info, and loading javascript generated content in webpages.
January 22, 2023
Best scraping service - tried them all
Hands down the best scraping service there is for a no-coder (...and I've tried them all).
Fast, easy to use, great documentation and stellar support.
Wish I'd found this months and months ago of waisting time at others. Highly recommend!
May 05, 2023
The best web scraper API for Bubble apps
Having tried several web scraper APIs I have found that Page2API is the best web scraper API for integrating with the Bubble API connector.
If you're a Bubble app developer Page2API is the web scraper you've been looking for!
November 30, 2022
Customer service is WORLD CLASS
Nick is serious about his business -- super knowledgeable and helpful whenever we have the slightest problem.
Honestly, the best customer service of any SaaS I've had the pleasure of working with.
10/10.
December 02, 2022
It's a perfect product
This team has a very high sense of responsibility for the product.
They let me know the part I don't know so kindly.
I didn't feel any discomfort when I used it in Korea
June 12, 2023
Highly professional support!
Amazing quick support!
But more than that, an actual relevant and pro help which solved my issue.
April 19, 2023
Incredible
Nick was incredible.
He helped me so much.
Need it for a research project and I highly highly recommend this service.
December 21, 2022
Great product, great support
I was searching for a scraping tool which fits to different types of needs and found Page2API.
The support is amazing and the product, too!
We will use Page2API also for our agency clients now.
Thank you for this great tool!
March 07, 2023
Really good provider for web-scraping…
Really good provider for web-scraping services, their customer service is top notch!
January 25, 2023
Great service with absolutely…
Great service with absolutely outstanding support
December 01, 2022

You might also like

Nicolae Rotaru
Nicolae Rotaru
2021-11-22 - 7 min read

How to Scrape Real Estate Data from Zillow (Code & No code)

Learn how to scrape real estate data from Zillow with Page2API in no time

Nicolae Rotaru
Nicolae Rotaru
2021-10-31 - 4 min read

How to Scrape eBay Data: Products, Prices, and more

This article will describe the easiest way to scrape eBay products with Page2API

Nicolae Rotaru
Nicolae Rotaru
2021-10-19 - 6 min read

How to Scrape Amazon Data: Products, Pricing, Reviews, etc.

This article will describe the easiest way to scrape amazon product data and reviews with Page2API

Ready to Scrape the Web like a PRO?

1000 free API calls.
Based on all requests made in the last 30 days. 99.85% success rate.
No-code-friendly.
Trustpilot stars 4.6