Introduction
In today's digital age, the internet provides us with an overwhelming amount of information, including news articles from various sources.
While staying informed is important, it can be time-consuming and overwhelming to sift through numerous articles and websites to get a complete picture of a story.
That's where web scraping and AI-powered summarization come in. Web scraping allows us to extract data from websites, including news articles, while AI-powered summarization algorithms can help us to quickly digest and understand the key points of an article.
In this blog post, we'll explore how to scrape news articles from BBC News with Page2API and summarize the content with GPT-3.5-turbo to efficiently gather and understand news content.
You can use the code from this post to start building your own AI news scraper.
This will allow you to automate the process of collecting and analyzing news articles, providing you with a streamlined and efficient way to stay updated on current events and topics of interest.
With customization, you can tailor the article scraper to focus on specific news sources, topics, or types of content, making it a valuable tool for both personal use and professional research.
Prerequisites
To start scraping the articles from the news websites, we will need the following things:
- A Page2API account
- An OpenAI account
- The URL of an article from BBC News. This can also be an URL from any other news website, since the HTML selector will be the same.
- Some basic Ruby and JavaScript coding skills.
How to scrape the news article content
Before starting, we need to know that the the content we are looking for is usually located inside an article HTML tag, which makes the task a bit easier.
https://www.bbc.com/news/business-65261147
From the article page, we will only scrape the content, which is located inside the article tag.
But before we start scraping the article content we need to make sure that we clean up the page from the HTML tags we don't need, so we don't get any kind of noisy text.
let tagsToRemove = ['iframe', 'img', 'pre', 'script', 'style', 'hr', 'option', 'select', 'svg', 'video', 'input', 'nav', 'button', 'header', 'footer'];
tagsToRemove.forEach(function(tag) {
var elements = document.querySelectorAll(tag);
elements.forEach(function(element) {
element.parentNode.removeChild(element);
});
});
/* and here is the base64 encoded version of the snippet above */
bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=
document.querySelector('article').innerText.replace(/\n\n/g, '\n')
/* and here is the base64 encoded version of the snippet above */
ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp
Now it's time to build the request that will scrape the news article.
{
"api_key": "YOUR_PAGE2API_KEY",
"url": "https://www.bbc.com/news/business-65261147",
"real_browser": true,
"raw": {
"key": "article"
},
"scenario": [
{ "wait_for": "article" },
{ "execute_js": "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ "execute": "parse" }
],
"parse": {
"article": "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
}
}
require 'rest_client'
require 'json'
api_url = "https://www.page2api.com/api/v1/scrape"
payload = {
api_key: "YOUR_PAGE2API_KEY",
url: "https://www.bbc.com/news/business-65261147",
real_browser: true,
raw: {
key: "article"
},
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
}
}
result = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: api_url,
headers: { "Content-type" => "application/json" },
).body
puts(result)
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
How to summarize the article with AI (GPT-3.5-turbo)
In the following part of the article, we will:
- Collect the scraped article.
- Build a GPT prompt.
- Send the article content and the prompt to GPT.
- Receive the article summary.
From the code perspective, we will:
- Switch to Ruby.
- Separate the code into two classes to enhance the readability.
- Provide the possibility to change the article URL and the number of sentences for the summary.
require 'rest_client'
require 'json'
class Page2APIParser
def initialize(url)
end
def perform
end
end
class GPTSummarizer
def initialize(content, sentences)
end
def perform
end
end
article_url = ARGV[0] || raise('The article URL was not provided!')
sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences
page2api = Page2APIParser.new(article_url)
page2api.perform
gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
gpt.perform
puts gpt.result
The script can be called from the terminal like in the following example:
$ ruby gpt.rb https://www.bbc.com/news/business-65261147 5
require 'rest_client'
require 'json'
class Page2APIParser
API_KEY = ''
attr_reader :url, :article_content
def initialize(url)
@url = url
end
def perform
@article_content = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://www.page2api.com/api/v1/scrape',
headers: { "Content-type" => "application/json" },
).body
end
private
def payload
{
api_key: API_KEY,
url: url,
real_browser: true,
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
},
raw: {
key: "article"
}
}
end
end
API_KEY = 'Your Page2API API key'
page2api = Page2APIParser.new('https://www.bbc.com/news/business-65261147')
page2api.perform
puts page2api.article_content
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
"Summarize this article in #{sentences} sentences or less."
require 'rest_client'
require 'json'
class GPTSummarizer
API_KEY = ''
attr_reader :article_content, :sentences, :result
def initialize(article_content, sentences)
@article_content = article_content
@sentences = sentences
end
def perform
response = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://api.openai.com/v1/chat/completions',
headers: {
"Content-type" => "application/json",
"Authorization" => "Bearer #{API_KEY}"
},
).body
summary = JSON.parse(response)
@result = summary.dig('choices', 0, 'message', 'content')
end
private
def payload
{
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "Summarize this article in #{sentences} sentences or less."
},
{
role: "user",
content: article_content
}
]
}
end
end
API_KEY = 'Your OpenAI API key'
article_content = <<-TEXT
Chris Baraniuk - Technology reporter
Tue, May 9,
2023 at 3: 23 AM PDT·6 min read
Hurricane force winds can damage even the sturdiest wind turbines
The world's biggest storms, which whip the high seas into a frenzy or flatten buildings on land, have long daunted wind farm developers. But that is changing.
Operators are increasingly adopting turbines designed to withstand tropical cyclones. One of the latest examples is a "typhoon-resistant" floating wind turbine, which will soon help to power an offshore oil platform in China.
According to the manufacturer, MingYang Smart Energy, this 7.25 megawatt (MW) turbine can survive wind speeds of up to 134mph for 10 minutes. It has been installed at a facility 136km off the coast of the island province of Hainan.
MingYang did not respond to a BBC request for comment but theirs is not the first turbine designed to face down such an onslaught. In 2021, US firm GE received typhoon-certification for its mammoth Haliade-X turbine. It is fixed, not floating, and has a capacity of up to 13MW.
The blistering growth of the wind energy industry is pushing turbines to their limits and some question whether the pace of the rollout is wise.
While components such as turbine blades are remarkably strong, they are not indestructible. And the forces of nature, especially out at sea, are notoriously unpredictable, meaning the pressure is on to prove that wind turbines really are hurricane-ready.
...
TEXT
gpt = GPTSummarizer.new(article_content, 5)
gpt.perform
puts gpt.result
require 'rest_client'
require 'json'
class Page2APIParser
API_KEY = 'Your Page2API API key'
attr_reader :url, :article_content
def initialize(url)
@url = url
end
def perform
@article_content = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://www.page2api.com/api/v1/scrape',
headers: { "Content-type" => "application/json" },
).body
end
private
def payload
{
api_key: API_KEY,
url: url,
real_browser: true,
scenario: [
{ wait_for: "article" },
{ execute_js: "bGV0IHRhZ3NUb1JlbW92ZSA9IFsnaWZyYW1lJywgJ2ltZycsICdwcmUnLCAnc2NyaXB0JywgJ3N0eWxlJywgJ2hyJywgJ29wdGlvbicsICdzZWxlY3QnLCAnc3ZnJywgJ3ZpZGVvJywgJ2lucHV0JywgJ25hdicsICdidXR0b24nLCAnaGVhZGVyJywgJ2Zvb3RlciddOwoKdGFnc1RvUmVtb3ZlLmZvckVhY2goZnVuY3Rpb24odGFnKSB7CiAgdmFyIGVsZW1lbnRzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCh0YWcpOwogIGVsZW1lbnRzLmZvckVhY2goZnVuY3Rpb24oZWxlbWVudCkgewogICAgZWxlbWVudC5wYXJlbnROb2RlLnJlbW92ZUNoaWxkKGVsZW1lbnQpOwogIH0pOwp9KTs=" },
{ execute: "parse" }
],
parse: {
article: "js >> ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignYXJ0aWNsZScpLmlubmVyVGV4dC5yZXBsYWNlKC9cblxuL2csICdcbicp"
},
raw: {
key: "article"
}
}
end
end
class GPTSummarizer
API_KEY = 'Your OpenAI API key'
attr_reader :article_content, :sentences, :result
def initialize(article_content, sentences)
@article_content = article_content
@sentences = sentences
end
def perform
response = RestClient::Request.execute(
method: :post,
payload: payload.to_json,
url: 'https://api.openai.com/v1/chat/completions',
headers: {
"Content-type" => "application/json",
"Authorization" => "Bearer #{API_KEY}"
},
).body
summary = JSON.parse(response)
@result = summary.dig('choices', 0, 'message', 'content')
end
private
def payload
{
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "Summarize this article in #{sentences} sentences or less."
},
{
role: "user",
content: article_content
}
]
}
end
end
article_url = ARGV[0] || raise('The article URL was not provided!')
sentences = ARGV[1].to_i.nonzero? || 5 # default summary length: 5 sentences
page2api = Page2APIParser.new(article_url)
page2api.perform
gpt = GPTSummarizer.new(page2api.article_content[0..20_000], sentences) # 20.000 characters max length
gpt.perform
puts gpt.result
$ ruby gpt.rb https://www.bbc.com/news/business-65261147 5
Wind farm operators are increasingly using turbines designed to withstand tropical cyclones.
One of the latest examples is a "typhoon-resistant" floating wind turbine, installed at a facility off the coast of China, which can survive wind speeds of up to 134 mph for 10 minutes.
It is expected that the expansion of wind energy will occur in regions where tropical cyclones are a familiar threat, including Southeast Asia and the Gulf of Mexico.
Some of the most dangerous forces to trouble turbine blades are torsion, or twisting, loads, which can induce difficult-to-spot fractures.
While current testing and industry standards are not sufficient to prove that the largest turbine blades can withstand these stresses, new designs, such as a turbine with tall, vertical blades that spin around a central tower, could help.
Conclusion
In conclusion, it is now simpler than ever to scrape news articles and summarize their content thanks to advancements in AI technology.
The way we consume news and keep informed could be completely changed by the ability to extract useful information from large amounts of data.
You can use AI tools to scrape news articles and produce succinct and accurate summaries by following the instructions provided in this post.
The methods covered here can help you build your own article scraper to save time and effort while keeping you informed, whether you're a researcher, journalist, or simply someone who wants to stay current on events.
While our focus here is on harnessing AI to scrape and summarize news articles, the same principles of AI-driven data extraction and analysis can be applied in various other contexts.
A prime example is our detailed exploration in How to Scrape TripAdvisor Reviews and Perform Sentiment Analysis with AI.
This guide illustrates how AI can effectively analyze customer feedback from platforms like TripAdvisor, providing valuable insights into public opinion and customer satisfaction.
By following the methods outlined in our complementary blog post, businesses and researchers can gain a deeper understanding of consumer sentiment, leveraging AI's power to transform raw data into actionable knowledge.