How to Scrape News Articles and Summarize the Content with AI


2023-05-10 - 9 min read

Nicolae Rotaru
Nicolae Rotaru

Introduction

In today's digital age, the internet provides us with an overwhelming amount of information, including news articles from various sources.
While staying informed is important, it can be time-consuming and overwhelming to sift through numerous articles and websites to get a complete picture of a story.

That's where web scraping and AI-powered summarization come in. Web scraping allows us to extract data from websites, including news articles, while AI-powered summarization algorithms can help us to quickly digest and understand the key points of an article.

In this blog post, we'll explore how to scrape news articles from Yahoo News with Page2API and summarize the content with GPT-3.5-turbo to efficiently gather and understand news content.

Prerequisites

To start scraping all the mentioned news websites, we will need the following things:


  • A Page2API account
  • An OpenAI account
  • The URL of an article from Yahoo News. This can also be an URL from any other news website, since the HTML selector will be the same.
  • Some basic Ruby and JavaScript coding skills.

How to scrape the news article content

Before starting, we need to know that the the content we are looking for is usually located inside an article HTML tag, which makes the task a bit easier.

To start scraping we can use any news article URL, for example:

  
    https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html
  

The article URL is the first parameter we need to start the scraping process.


From the article page, we will only scrape the content, which is located inside the article tag.

But before we start scraping the article content we need to make sure that we clean up the page from the HTML tags we don't need, so we don't get any kind of noisy text.

The JavaScript snippet that will clean up the page will be:

  
    let tagsToRemove = ['iframe', 'img', 'pre', 'script', 'style', 'hr', 'option', 'select', 'svg', 'video', 'input', 'nav', 'button', 'header', 'footer'];

    tagsToRemove.forEach(function(tag) {
      var elements = document.querySelectorAll(tag);
      elements.forEach(function(element) {
        element.parentNode.removeChild(element);
      });
    });


    /* and here is the base64 encoded version of the snippet above */

    bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=
  

Next is the selector that will get the article content:

  
    document.querySelector('article').innerText.replace(/\n\n/g, '\n')

    /* and here is the base64 encoded version of the snippet above */

    ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp
  

Now it's time to build the request that will scrape the news article.


This is the payload we are looking for:

  
    {
      "api_key": "YOUR_PAGE2API_KEY",
      "url": "https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html",
      "real_browser": true,
      "raw": {
        "key": "article"
      },
      "scenario": [
        { "wait_for": "article" },
        { "execute_js": "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
        { "execute": "parse" }
      ],
      "parse": {
        "article": "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
      }
    }
  

Code examples

      
    require 'rest_client'
    require 'json'

    api_url = "https://www.page2api.com/api/v1/scrape"
    payload = {
      api_key: "YOUR_PAGE2API_KEY",
      url: "https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html",
      real_browser: true,
      raw: {
        key: "article"
      },
      scenario: [
        { wait_for: "article" },
        { execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
        { execute: "parse" }
      ],
      parse: {
        article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
      }
    }

    result = RestClient::Request.execute(
      method: :post,
      payload: payload.to_json,
      url: api_url,
      headers: { "Content-type" => "application/json" },
    ).body

    puts(result)
      
    

The result

  
    Chris Baraniuk - Technology reporter
    Tue, May 9,
    2023 at 3: 23 AM PDT·6 min read
    Hurricane force winds can damage even the sturdiest wind turbines
    The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
    Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
    According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
    MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
    The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
    While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.

    ...
  

How to summarize the article with AI (GPT-3.5-turbo)

In the following part of the article, we will:


  • Collect the scraped article.
  • Build a GPT prompt.
  • Send the article content and the prompt to GPT.
  • Receive the article summary.

From the code perspective, we will:


  • Switch to Ruby.
  • Separate the code into two classes to enhance the readability.
  • Provide the possibility to change the article URL and the number of sentences for the summary.
Let's start by creating a new file (gpt.rb) with the following structure

  
  require 'rest_client'
  require 'json'

  class Page2APIParser
    def initialize(url)
    end

    def perform
    end
  end

  class GPTSummarizer
    def initialize(content, sentences)
    end

    def perform
    end
  end

  article_url = ARGV[0] || raise('The article URL was not provided!')
  sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences

  page2api = Page2APIParser.new(article_url)
  page2api.perform

  gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
  gpt.perform

  puts gpt.result
  

This is our main script
It receives 2 arguments: the news article URL, and the number of total sentences for our summary.
The script can be called from the terminal like in the following example:

  
    $ ruby gpt.rb https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html 5
  

Now let's use the code from the first part of the article and build the parser

  
  require 'rest_client'
  require 'json'

  class Page2APIParser
    API_KEY = ''

    attr_reader :url, :article_content

    def initialize(url)
      @url = url
    end

    def perform
      @article_content = RestClient::Request.execute(
        method: :post,
        payload: payload.to_json,
        url: 'https://www.page2api.com/api/v1/scrape',
        headers: { "Content-type" => "application/json" },
      ).body
    end

    private

    def payload
      {
        api_key: API_KEY,
        url: url,
        real_browser: true,
        scenario: [
          { wait_for: "article" },
          { execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
          { execute: "parse" }
        ],
        parse: {
          article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
        },
        raw: {
          key: "article"
        }
      }
    end
  end
  

You can test the parser by updating the API_KEY

  
    API_KEY = 'Your Page2API API key'
  
and running

  
    page2api = Page2APIParser.new('https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html')
    page2api.perform

    puts page2api.article_content
  

The parser will generate the following content

  
    Chris Baraniuk - Technology reporter
    Tue, May 9,
    2023 at 3: 23 AM PDT·6 min read
    Hurricane force winds can damage even the sturdiest wind turbines
    The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
    Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
    According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
    MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
    The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
    While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.

    ...
  

Now let's build the GPT summarizer.

The working principle is similar, but instead of article URL, the class will receive the article content and the number of sentences, build a payload, send it to GPT API and print the result.

We will use the following GPT prompt for our request

  
    "Summarize this article in #{sentences} sentences or less."
  

Here is our GPT class

  
  require 'rest_client'
  require 'json'

  class GPTSummarizer
    API_KEY = ''

    attr_reader :article_content, :sentences, :result

    def initialize(article_content, sentences)
      @article_content = article_content
      @sentences = sentences
    end

    def perform
      response = RestClient::Request.execute(
        method: :post,
        payload: payload.to_json,
        url: 'https://api.openai.com/v1/chat/completions',
        headers: {
          "Content-type" => "application/json",
          "Authorization" => "Bearer #{API_KEY}"
        },
      ).body

      summary = JSON.parse(response)

      @result = summary.dig('choices', 0, 'message', 'content')
    end

    private

    def payload
      {
        model: "gpt-3.5-turbo",
        messages: [
          {
            role: "system",
            content: "Summarize this article in #{sentences} sentences or less."
          },
          {
            role: "user",
            content: article_content
          }
        ]
      }
    end
  end
  

You can test the GPT summarizer by updating the API_KEY

  
    API_KEY = 'Your OpenAI API key'
  
and running

  
    article_content = <<-TEXT
      Chris Baraniuk - Technology reporter
      Tue, May 9,
      2023 at 3: 23 AM PDT·6 min read
      Hurricane force winds can damage even the sturdiest wind turbines
      The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
      Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
      According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
      MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
      The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
      While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
      ...
    TEXT

    gpt = GPTSummarizer.new(article_content, 5)
    gpt.perform

    puts gpt.result
  

Now let's glue everything together

  
  require 'rest_client'
  require 'json'

  class Page2APIParser
    API_KEY = 'Your Page2API API key'

    attr_reader :url, :article_content

    def initialize(url)
      @url = url
    end

    def perform
      @article_content = RestClient::Request.execute(
        method: :post,
        payload: payload.to_json,
        url: 'https://www.page2api.com/api/v1/scrape',
        headers: { "Content-type" => "application/json" },
      ).body
    end

    private

    def payload
      {
        api_key: API_KEY,
        url: url,
        real_browser: true,
        scenario: [
          { wait_for: "article" },
          { execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
          { execute: "parse" }
        ],
        parse: {
          article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
        },
        raw: {
          key: "article"
        }
      }
    end
  end

  class GPTSummarizer
    API_KEY = 'Your OpenAI API key'

    attr_reader :article_content, :sentences, :result

    def initialize(article_content, sentences)
      @article_content = article_content
      @sentences = sentences
    end

    def perform
      response = RestClient::Request.execute(
        method: :post,
        payload: payload.to_json,
        url: 'https://api.openai.com/v1/chat/completions',
        headers: {
          "Content-type" => "application/json",
          "Authorization" => "Bearer #{API_KEY}"
        },
      ).body

      summary = JSON.parse(response)

      @result = summary.dig('choices', 0, 'message', 'content')
    end

    private

    def payload
      {
        model: "gpt-3.5-turbo",
        messages: [
          {
            role: "system",
            content: "Summarize this article in #{sentences} sentences or less."
          },
          {
            role: "user",
            content: article_content
          }
        ]
      }
    end
  end



  article_url = ARGV[0] || raise('The article URL was not provided!')
  sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences

  page2api = Page2APIParser.new(article_url)
  page2api.perform

  gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
  gpt.perform

  puts gpt.result
  

Let's run the script

  
    $ ruby gpt.rb https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html 5
  

The result must look like the following one

  
    Wind farm operators are increasingly using turbines designed to withstand tropical cyclones.
    One of the latest examples is a "typhoon-resistant" floating wind turbine, installed at a facility off the coast of China, which can survive wind speeds of up to 134 mph for 10 minutes.
    It is expected that the expansion of wind energy will occur in regions where tropical cyclones are a familiar threat, including Southeast Asia and the Gulf of Mexico.
    Some of the most dangerous forces to trouble turbine blades are torsion, or twisting, loads, which can induce difficult-to-spot fractures.
    While current testing and industry standards are not sufficient to prove that the largest turbine blades can withstand these stresses, new designs, such as a turbine with tall, vertical blades that spin around a central tower, could help.
  

Conclusion

In conclusion, it is now simpler than ever to scrape news articles and summarize their content thanks to advancements in AI technology. The way we consume news and keep informed could be completely changed by the ability to extract useful information from large amounts of data.

You can use AI tools to scrape news articles and produce succinct and accurate summaries by following the instructions provided in this post.

The methods covered here can help you save time and effort while keeping you informed, whether you're a researcher, journalist, or simply someone who wants to stay current on events.

You might also like:

Nicolae Rotaru
Nicolae Rotaru
2023-04-30 - 10 min read

How to Scrape Trustpilot Reviews and Perform Sentiment Analysis with AI

In this blog post, we will explore the step-by-step process of scraping Trustpilot reviews using Page2API, and then performing sentiment analysis on the extracted data using GPT-3.5-turbo.

Nicolae Rotaru
Nicolae Rotaru
2023-04-10 - 4 min read

How to Scrape IMDB Movies Data (Code & No Code)

In this article, you will find an easy way to scrape IMDB movies with Page2API using one of your favorite programming languages or a no-code solution that will import IMDB movies data to Google Sheets

Nicolae Rotaru
Nicolae Rotaru
2023-01-07 - 5 min read

How to Scrape Youtube Data: Video and Channel Details (Code & No Code)

In this article, you will find an easy way to scrape Youtube with Page2API using one of your favorite programming languages or a no-code solution that will import Youtube channel videos to Google Sheets

Ready to Scrape the Web like a PRO?

1000 free API calls.
Based on all requests made in the last 30 days. 99.85% success rate.
No-code-friendly.
Trustpilot stars 4.5