How to Scrape Reddit Data: Posts, Images, Comments, and more.

Introduction

Reddit.com is a social news website and forum where content is socially curated and promoted by site members through voting.

In this article, you will read about the easiest way to scrape Reddit posts and comments with Page2API.

You will find code examples for Ruby, Python, PHP, NodeJS, cURL, and a No-Code solution that will import Reddit posts into Google Sheets.

To scrape Reddit, we will use Page2API - a powerful and delightful API that will help you scrape the web like a PRO.

Prerequisites

To start scraping Reddit posts, we will need the following things:

A Page2API account
The link to a subreddit that we are about to scrape.
To make the scraping process easier - we will use the old Reddit UI since it has a simpler HTML structure.
In this article, we will scrape the BMW subreddit.

How to scrape Reddit Posts

The first thing we need is to open the subreddit URL:

  
    https://old.reddit.com/r/BMW/

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

From this page, we will scrape the following attributes from each Reddit post:

Title
URL
Thumbnail
Submitted
Author
Author URL
Comments
Rank
Score

Now, let's define the selectors for each attribute.

  
    /* Parent: */
    .thing[data-promoted=false]

    /* Title */
    a.title

    /* URL */
    a.title

    /* Thumbnail */
    a.thumbnail img

    /* Submitted */
    time

    /* Author */
    a.author

    /* Author URL */
    a.author

    /* Comments */
    (parent div, attribute: data-comments-count)

    /* Rank */
    (parent div, attribute: data-rank

    /* Score */
    (parent div, attribute: data-score

Now, let's handle the pagination.
We will iterate through the Reddit pages by clicking on the Next page button.

To go to the next page, we must click on the next page link if it's present on the page:

  
    document.querySelector('.next-button a').click()

The scraping will continue while the Next link is present on the page, and stop if it disappears.
The stop condition for the scraper will be the following javascript snippet:

  
    document.querySelector('.next-button a') === null

Let's build the request that will scrape the needed pages from this subreddit.

The following examples will show how to scrape 3 pages of posts from Reddit.com

Our payload will look like this:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/",
      "merge_loops": true,
      "real_browser": true,
      "javascript": false,
      "scenario": [
        {
          "loop": [
            { "wait_for": ".thing" },
            { "execute": "parse" },
            { "click": ".next-button a" }
          ],
          "stop_condition": "document.querySelector('.next-button a') === null",
          "iterations": 3
        }
      ],
      "parse": {
        "posts": [
          {
            "_parent": ".thing[data-promoted=false]",
            "title": "a.title >> text",
            "url": "a.title >> href",
            "thumbnail": "a.thumbnail img >> src",
            "submitted": "time >> datetime",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "comments": "_parent >> data-comments-count",
            "rank": "_parent >> data-rank",
            "score": "_parent >> data-score"
          }
        ]
      }
    }

Code examples:

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape 5 pages of posts from Reddit.com

    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: "https://old.reddit.com/r/BMW/",
      merge_loops: true,
      real_browser: true,
      javascript: false,
      scenario: [
        {
          loop: [
            { wait_for: ".thing" },
            { execute: "parse" },
            { click: ".next-button a" }
          ],
          stop_condition: "document.querySelector('.next-button a') === null",
          iterations: 3
        }
      ],
      parse: {
        posts: [
          {
            _parent: ".thing[data-promoted=false]",
            title: "a.title >> text",
            url: "a.title >> href",
            thumbnail: "a.thumbnail img >> src",
            submitted: "time >> datetime",
            author: "a.author >> text",
            author_url: "a.author >> href",
            comments: "_parent >> data-comments-count",
            rank: "_parent >> data-rank",
            score: "_parent >> data-score"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

      
    import requests
    import json

    api_url = 'https://www.page2api.com/api/v1/scrape'

    # The following example will show how to scrape 5 pages of posts from Reddit.com

    payload = {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/",
      "merge_loops": True,
      "real_browser": True,
      "javascript": False,
      "scenario": [
        {
          "loop": [
            { "wait_for": ".thing" },
            { "execute": "parse" },
            { "click": ".next-button a" }
          ],
          "stop_condition": "document.querySelector('.next-button a') === null",
          "iterations": 3
        }
      ],
      "parse": {
        "posts": [
          {
            "_parent": ".thing[data-promoted=false]",
            "title": "a.title >> text",
            "url": "a.title >> href",
            "thumbnail": "a.thumbnail img >> src",
            "submitted": "time >> datetime",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "comments": "_parent >> data-comments-count",
            "rank": "_parent >> data-rank",
            "score": "_parent >> data-score"
          }
        ]
      }
    }

    headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
    response = requests.post(api_url, data=json.dumps(payload), headers=headers)
    result = json.loads(response.text)

    print(result)

      
    <?php


    $api_url = 'https://www.page2api.com/api/v1/scrape';

    // The following example will show how to scrape 5 pages of posts from Reddit.com

    $payload = [
      'api_key' => 'YOUR_PAGE2API_KEY',
      'url' => 'https://old.reddit.com/r/BMW/',
      'merge_loops' => true,
      'real_browser' => true,
      'javascript' => false,
      'scenario' => [
        0 => [
          'loop' => [
            0 => [ 'wait_for' => '.thing' ],
            1 => [ 'execute' => 'parse' ],
            2 => [ 'click' => '.next-button a' ]
          ],
          'stop_condition' => 'document.querySelector(\'.next-button a\') === null',
          'iterations' => 3
        ]
      ],
      'parse' => [
        'posts' => [
          0 => [
            '_parent' => '.thing[data-promoted=false]',
            'title' => 'a.title >> text',
            'url' => 'a.title >> href',
            'thumbnail' => 'a.thumbnail img >> src',
            'submitted' => 'time >> datetime',
            'author' => 'a.author >> text',
            'author_url' => 'a.author >> href',
            'comments' => '_parent >> data-comments-count',
            'rank' => '_parent >> data-rank',
            'score' => '_parent >> data-score'
          ]
        ]
      ]
    ];

    $postdata = json_encode($payload);
    $ch = curl_init($api_url);
    curl_setopt($ch,CURLOPT_POST, true);
    curl_setopt($ch,CURLOPT_POSTFIELDS, $postdata);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:application/json'));
    curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
    $result = curl_exec($ch);
    curl_close($ch);

    echo $result;


    ?>

      
    const axios = require('axios');
    const api_url = 'https://www.page2api.com/api/v1/scrape';

    // The following example will show how to scrape 5 pages of posts from Reddit.com

    const payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: 'https://old.reddit.com/r/BMW/',
      merge_loops: true,
      real_browser: true,
      javascript: false,
      scenario: [
        {
          loop: [
            { wait_for: '.thing' },
            { execute: 'parse' },
            { click: '.next-button a' }
          ],
          stop_condition: 'document.querySelector(\'.next-button a\') === null',
          iterations: 3
        }
      ],
      parse: {
        posts: [
          {
            _parent: '.thing[data-promoted=false]',
            title: 'a.title >> text',
            url: 'a.title >> href',
            thumbnail: 'a.thumbnail img >> src',
            submitted: 'time >> datetime',
            author: 'a.author >> text',
            author_url: 'a.author >> href',
            comments: '_parent >> data-comments-count',
            rank: '_parent >> data-rank',
            score: '_parent >> data-score'
          }
        ]
      }
    };

    axios.post(api_url, payload)
      .then((res) => {
        console.log(JSON.stringify(res.data, null, "  "));
      }).catch((err) => {
         console.error(err);
      });

      
    curl -XPOST -H "Content-type: application/json" -d '{
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/",
      "merge_loops": true,
      "real_browser": true,
      "javascript": false,
      "scenario": [
        {
          "loop": [
            { "wait_for": ".thing" },
            { "execute": "parse" },
            { "click": ".next-button a" }
          ],
          "stop_condition": "document.querySelector(\".next-button a\") === null",
          "iterations": 3
        }
      ],
      "parse": {
        "posts": [
          {
            "_parent": ".thing[data-promoted=false]",
            "title": "a.title >> text",
            "url": "a.title >> href",
            "thumbnail": "a.thumbnail img >> src",
            "submitted": "time >> datetime",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "comments": "_parent >> data-comments-count",
            "rank": "_parent >> data-rank",
            "score": "_parent >> data-score"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python3.10 -mjson.tool

The result

  
    {
      "result": {
        "posts": [
          {
              "title": "Thought you would enjoy this",
              "url": "https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/",
              "thumbnail": "https://b.thumbs.redditmedia.com/1OMVuKzvOeVi6IkOMc1K94HzPOLs3InozhCPlxU8K7E.jpg",
              "submitted": "2022-05-20T18:55:11+00:00",
              "author": "L9FanboyXD",
              "author_url": "https://old.reddit.com/user/L9FanboyXD",
              "comments": "206",
              "rank": "1",
              "score": "1234"
          },
          {
              "title": "Totalled..",
              "url": "https://old.reddit.com/r/BMW/comments/uu7yih/totalled/",
              "thumbnail": "https://b.thumbs.redditmedia.com/QQ7_MbJqhB5t4QfKJjbTFU4sEK0Cd9Cy5uFCSzzeLzM.jpg",
              "submitted": "2022-05-20T21:16:40+00:00",
              "author": "Conscious_Champion35",
              "author_url": "https://old.reddit.com/user/Conscious_Champion35",
              "comments": "130",
              "rank": "2",
              "score": "375"
          },
          {
              "title": "Joined the squad. Yes there was a flat bed at the dealer as I was closing this deal today.",
              "url": "https://www.reddit.com/gallery/uucgb5",
              "thumbnail": "https://b.thumbs.redditmedia.com/g8vAVq0N59R0a-IX1mBtM1YEy2gouhhGTTS7m4Nr8us.jpg",
              "submitted": "2022-05-21T01:16:08+00:00",
              "author": "restloy",
              "author_url": "https://old.reddit.com/user/restloy",
              "comments": "22",
              "rank": "3",
              "score": "157"
          }, ...
        ]
      }, ...
    }

How to scrape Reddit Comments

We need to open any post URL from the previous step.

This will change the browser URL to something similar to:

  
    https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/

This URL is the first parameter we need to scrape the Reddit post data and comments.

The comments section that you see must look like the following one:

From this page, we will scrape the following attributes:

Post data:

Title
Image

Comments data:

Author
Author URL
Posted
Score
ID
Content

Now, let's define the selectors for each attribute.

Post data

  
    /* Title */
    a.title

    /* Image */
    .media-preview-content img.preview

Comments data:

  
    /* Parent */
    .entry

    /* Author */
    a.author

    /* Author URL */
    a.author

    /* Posted */
    time

    /* Score */
    .score.unvoted

    /* ID */
    input[name=thing_id]

    /* Content */
    .usertext-body

It's time to prepare the request that will scrape the data from a Reddit post page.

The payload for our scraping request will be:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
      "parse": {
        "title": "a.title >> text",
        "image": ".media-preview-content img.preview >> src",
        "comments": [
          {
            "_parent": ".entry",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "posted": "time >> datetime",
            "score": ".score.unvoted >> text",
            "id": "input[name=thing_id] >> value",
            "content": ".usertext-body >> text"
          }
        ]
      }
    }

Code examples

      
    require 'rest_client'
    require 'json'

    api_url = 'https://www.page2api.com/api/v1/scrape'
    payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
      parse: {
        title: "a.title >> text",
        image: ".media-preview-content img.preview >> src",
        comments: [
          {
            _parent: ".entry",
            author: "a.author >> text",
            author_url: "a.author >> href",
            posted: "time >> datetime",
            score: ".score.unvoted >> text",
            id: "input[name=thing_id] >> value",
            content: ".usertext-body >> text"
          }
        ]
      }
    }

    response = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    result = JSON.parse(response)

    puts(result)

      
    import requests
    import json

    api_url = 'https://www.page2api.com/api/v1/scrape'
    payload = {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
      "parse": {
        "title": "a.title >> text",
        "image": ".media-preview-content img.preview >> src",
        "comments": [
          {
            "_parent": ".entry",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "posted": "time >> datetime",
            "score": ".score.unvoted >> text",
            "id": "input[name=thing_id] >> value",
            "content": ".usertext-body >> text"
          }
        ]
      }
    }

    headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
    response = requests.post(api_url, data=json.dumps(payload), headers=headers)
    result = json.loads(response.text)

    print(result)

      
    <?php

    $api_url = 'https://www.page2api.com/api/v1/scrape';
    $payload = [
      'api_key' => 'YOUR_PAGE2API_KEY',
      'url' => 'https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/',
      'parse' => [
        'title' => 'a.title >> text',
        'image' => '.media-preview-content img.preview >> src',
        'comments' => [
          0 => [
            '_parent' => '.entry',
            'author' => 'a.author >> text',
            'author_url' => 'a.author >> href',
            'posted' => 'time >> datetime',
            'score' => '.score.unvoted >> text',
            'id' => 'input[name=thing_id] >> value',
            'content' => '.usertext-body >> text'
          ]
        ]
      ]
    ];


    $postdata = json_encode($payload);
    $ch = curl_init($api_url);
    curl_setopt($ch,CURLOPT_POST, true);
    curl_setopt($ch,CURLOPT_POSTFIELDS, $postdata);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:application/json'));
    curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
    $result = curl_exec($ch);
    curl_close($ch);

    echo $result;

    ?>

      
    const axios = require('axios');
    const api_url = 'https://www.page2api.com/api/v1/scrape';
    const payload = {
      api_key: 'YOUR_PAGE2API_KEY',
      url: 'https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/',
      parse: {
        title: 'a.title >> text',
        image: '.media-preview-content img.preview >> src',
        comments: [
          {
            _parent: '.entry',
            author: 'a.author >> text',
            author_url: 'a.author >> href',
            posted: 'time >> datetime',
            score: '.score.unvoted >> text',
            id: 'input[name=thing_id] >> value',
            content: '.usertext-body >> text'
          }
        ]
      }
    };

    axios.post(api_url, payload)
      .then((res) => {
        console.log(JSON.stringify(res.data, null, "  "));
      }).catch((err) => {
         console.error(err);
      });

      
    curl -XPOST -H "Content-type: application/json" -d '{
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
      "parse": {
        "title": "a.title >> text",
        "image": ".media-preview-content img.preview >> src",
        "comments": [
          {
            "_parent": ".entry",
            "author": "a.author >> text",
            "author_url": "a.author >> href",
            "posted": "time >> datetime",
            "score": ".score.unvoted >> text",
            "id": "input[name=thing_id] >> value",
            "content": ".usertext-body >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python3.10 -mjson.tool

The result

  
    {
      "result": {
        "title": "I saw it in prague. the red is much prettier in real life",
        "image": "https://preview.redd.it/91x8qz36f3091.jpg?width=1023&auto=png&s=9c0aced50d6743989f053cf9cc78c896f928d8a7",
        "comments": [
          {
            "author": "DarkMeta81",
            "author_url": "https://old.reddit.com/user/DarkMeta81",
            "posted": "2022-05-17T20:12:31+00:00",
            "score": "15 points",
            "id": "t1_i8zlp77",
            "content": "i was on vacation and prague is full of beautiful bmw this m5 is great"
          },
          {
            "author": "Id-atl",
            "author_url": "https://old.reddit.com/user/Id-atl",
            "posted": "2022-05-18T01:06:15+00:00",
            "score": "5 points",
            "id": "t1_i90on16",
            "content": "Wow imagine being in Prague ripping this around!"
          }, ...
        ]
      }, ...
    }

How to export Reddit posts to Google Sheets

To export our Reddit posts to a Google Spreadsheet, we will need to slightly modify our request to receive the data in CSV format instead of JSON.

According to the documentation, we need to add the following parameters to our payload:

  
    "raw": {
      "key": "posts", "format": "csv"
    }

Now our payload will look like this:

{ "api_key": "YOUR_PAGE2API_KEY", "url": "https://old.reddit.com/r/BMW/", "raw": { "key": "posts", "format": "csv" }, "merge_loops": true, "real_browser": true, "scenario": [ { "loop": [ { "wait_for": ".thing" }, { "execute": "parse" }, { "click": ".next-button a" } ], "stop_condition": "document.querySelector('.next-button a') === null", "iterations": 5 } ], "parse": { "posts": [ { "_parent": ".thing[data-promoted=false]", "title": "a.title >> text", "url": "a.title >> href", "thumbnail": "a.thumbnail img >> src", "submitted": "time >> datetime", "author": "a.author >> text", "author_url": "a.author >> href", "comments": "_parent >> data-comments-count", "rank": "_parent >> data-rank", "score": "_parent >> data-score" } ] } }

Now, edit the payload above if needed, and press Encode →

Encode

The URL with encoded payload will be:

  Press 'Encode'

Note: If you are reading this article being logged in - you can copy the link above since it will already have your api_key in the encoded payload.

The final part is adding the IMPORTDATA function, and we are ready to import our Reddit posts into a Google Spreadsheet.

  Press 'Encode'

The result must look like the following one:

Conclusion

That's it!

In this article, you've learned how to scrape the data from Reddit.com with Page2API - a Web Scraping API that handles all the hassle, and lets you get the data you need with ease.

How to Scrape Reddit Data: Posts, Images, Comments, and more.

Introduction

Prerequisites

How to scrape Reddit Posts

How to scrape Reddit Comments

How to export Reddit posts to Google Sheets

Conclusion

You might also like

How to Scrape Glassdoor Reviews (Code & No code)

How to Scrape Yelp Data: Business Info, Reviews and more.

How to Scrape Real Estate Data from Zillow (Code & No code)

What customers are saying

Ready to Scrape the Web like a PRO?