Introduction
In today's digital age, the internet provides us with an overwhelming amount of information, including news articles from various sources.
While staying informed is important, it can be time-consuming and overwhelming to sift through numerous articles and websites to get a complete picture of a story.
That's where web scraping and AI-powered summarization come in. Web scraping allows us to extract data from websites, including news articles, while AI-powered summarization algorithms can help us to quickly digest and understand the key points of an article.
In this blog post, we'll explore how to scrape news articles from Yahoo News with Page2API and summarize the content with GPT-3.5-turbo to efficiently gather and understand news content.
Prerequisites
To start scraping all the mentioned news websites, we will need the following things:
- A Page2API account
- An OpenAI account
- The URL of an article from Yahoo News. This can also be an URL from any other news website, since the HTML selector will be the same.
- Some basic Ruby and JavaScript coding skills.
How to scrape the news article content
Before starting, we need to know that the the content we are looking for is usually located inside an article HTML tag, which makes the task a bit easier.
To start scraping we can use any news article URL, for example:
https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html
The article URL is the first parameter we need to start the scraping process.
From the article page, we will only scrape the content, which is located inside the article tag.
But before we start scraping the article content we need to make sure that we clean up the page from the HTML tags we don't need, so we don't get any kind of noisy text.
From the article page, we will only scrape the content, which is located inside the article tag.
But before we start scraping the article content we need to make sure that we clean up the page from the HTML tags we don't need, so we don't get any kind of noisy text.
The JavaScript snippet that will clean up the page will be:
let tagsToRemove = ['iframe', 'img', 'pre', 'script', 'style', 'hr', 'option', 'select', 'svg', 'video', 'input', 'nav', 'button', 'header', 'footer'];
tagsToRemove.forEach(function(tag) {
var elements = document.querySelectorAll(tag);
elements.forEach(function(element) {
element.parentNode.removeChild(element);
});
});
/* and here is the base64 encoded version of the snippet above */
bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=
Next is the selector that will get the article content:
document.querySelector('article').innerText.replace(/\n\n/g, '\n')
/* and here is the base64 encoded version of the snippet above */
ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp
Now it's time to build the request that will scrape the news article.
This is the payload we are looking for:
{
"api_key": "YOUR_PAGE2API_KEY",
"url": "https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html",
"real_browser": true,
"raw": {
"key": "article"
},
"scenario": [
{ "wait_for": "article" },
{ "execute_js": "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ "execute": "parse" }
],
"parse": {
"article": "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
}
}
Code examples
require 'rest_client'
require 'json'
api_url = "https://www.page2api.com/api/v1/scrape"
payload = {
api_key: "YOUR_PAGE2API_KEY",
url: "https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html",
real_browser: true,
raw: {
key: "article"
},
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
}
}
result = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: api_url,
headers: { "Content-type" => "application/json" },
).body
puts(result)
The result
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
How to summarize the article with AI (GPT-3.5-turbo)
In the following part of the article, we will:
- Collect the scraped article.
- Build a GPT prompt.
- Send the article content and the prompt to GPT.
- Receive the article summary.
From the code perspective, we will:
- Switch to Ruby.
- Separate the code into two classes to enhance the readability.
- Provide the possibility to change the article URL and the number of sentences for the summary.
Let's start by creating a new file (gpt.rb) with the following structure
require 'rest_client'
require 'json'
class Page2APIParser
def initialize(url)
end
def perform
end
end
class GPTSummarizer
def initialize(content, sentences)
end
def perform
end
end
article_url = ARGV[0] || raise('The article URL was not provided!')
sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences
page2api = Page2APIParser.new(article_url)
page2api.perform
gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
gpt.perform
puts gpt.result
This is our main script
It receives 2 arguments: the news article URL, and the number of total sentences for our summary.
The script can be called from the terminal like in the following example:
The script can be called from the terminal like in the following example:
$ ruby gpt.rb https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html 5
Now let's use the code from the first part of the article and build the parser
require 'rest_client'
require 'json'
class Page2APIParser
API_KEY = ''
attr_reader :url, :article_content
def initialize(url)
@url = url
end
def perform
@article_content = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://www.page2api.com/api/v1/scrape',
headers: { "Content-type" => "application/json" },
).body
end
private
def payload
{
api_key: API_KEY,
url: url,
real_browser: true,
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
},
raw: {
key: "article"
}
}
end
end
You can test the parser by updating the API_KEY
API_KEY = 'Your Page2API API key'
and running
page2api = Page2APIParser.new('https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html')
page2api.perform
puts page2api.article_content
The parser will generate the following content
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
Now let's build the GPT summarizer.
The working principle is similar, but instead of article URL, the class will receive the article content and the number of sentences, build a payload, send it to GPT API and print the result.
We will use the following GPT prompt for our request
"Summarize this article in #{sentences} sentences or less."
Here is our GPT class
require 'rest_client'
require 'json'
class GPTSummarizer
API_KEY = ''
attr_reader :article_content, :sentences, :result
def initialize(article_content, sentences)
@article_content = article_content
@sentences = sentences
end
def perform
response = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://api.openai.com/v1/chat/completions',
headers: {
"Content-type" => "application/json",
"Authorization" => "Bearer #{API_KEY}"
},
).body
summary = JSON.parse(response)
@result = summary.dig('choices', 0, 'message', 'content')
end
private
def payload
{
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "Summarize this article in #{sentences} sentences or less."
},
{
role: "user",
content: article_content
}
]
}
end
end
You can test the GPT summarizer by updating the API_KEY
API_KEY = 'Your OpenAI API key'
and running
article_content = <<-TEXT
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
TEXT
gpt = GPTSummarizer.new(article_content, 5)
gpt.perform
puts gpt.result
Now let's glue everything together
require 'rest_client'
require 'json'
class Page2APIParser
API_KEY = 'Your Page2API API key'
attr_reader :url, :article_content
def initialize(url)
@url = url
end
def perform
@article_content = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://www.page2api.com/api/v1/scrape',
headers: { "Content-type" => "application/json" },
).body
end
private
def payload
{
api_key: API_KEY,
url: url,
real_browser: true,
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
},
raw: {
key: "article"
}
}
end
end
class GPTSummarizer
API_KEY = 'Your OpenAI API key'
attr_reader :article_content, :sentences, :result
def initialize(article_content, sentences)
@article_content = article_content
@sentences = sentences
end
def perform
response = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://api.openai.com/v1/chat/completions',
headers: {
"Content-type" => "application/json",
"Authorization" => "Bearer #{API_KEY}"
},
).body
summary = JSON.parse(response)
@result = summary.dig('choices', 0, 'message', 'content')
end
private
def payload
{
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "Summarize this article in #{sentences} sentences or less."
},
{
role: "user",
content: article_content
}
]
}
end
end
article_url = ARGV[0] || raise('The article URL was not provided!')
sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences
page2api = Page2APIParser.new(article_url)
page2api.perform
gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
gpt.perform
puts gpt.result
Let's run the script
$ ruby gpt.rb https://news.yahoo.com/wind-turbine-handle-hurricane-speed-233028698.html 5
The result must look like the following one
Wind farm operators are increasingly using turbines designed to withstand tropical cyclones.
One of the latest examples is a "typhoon-resistant" floating wind turbine, installed at a facility off the coast of China, which can survive wind speeds of up to 134 mph for 10 minutes.
It is expected that the expansion of wind energy will occur in regions where tropical cyclones are a familiar threat, including Southeast Asia and the Gulf of Mexico.
Some of the most dangerous forces to trouble turbine blades are torsion, or twisting, loads, which can induce difficult-to-spot fractures.
While current testing and industry standards are not sufficient to prove that the largest turbine blades can withstand these stresses, new designs, such as a turbine with tall, vertical blades that spin around a central tower, could help.
Conclusion
In conclusion, it is now simpler than ever to scrape news articles and summarize their content thanks to advancements in AI technology.
The way we consume news and keep informed could be completely changed by the ability to extract useful information from large amounts of data.
You can use AI tools to scrape news articles and produce succinct and accurate summaries by following the instructions provided in this post.
The methods covered here can help you save time and effort while keeping you informed, whether you're a researcher, journalist, or simply someone who wants to stay current on events.
