Page2API Documentation

Wait

Parameter: wait scenario.wait loop.wait

This parameter allows the browser to give the web page some time (seconds) to render before capturing the HTML.
The use case usually occurs when interacting with the web page via scenario parameter or when some content is rendered asynchronously after the page load.
Maximum value: 10 (seconds).

    "wait": 2
  

Note: this parameter is available only when real_browser is set to true and javascript is not disabled.

Wait for element

Parameter: wait_for scenario.wait_for loop.wait_for

This parameter allows the browser to give the web page some time (seconds) to render a particular element before capturing the HTML.
Maximum timeout: 10 (seconds).

    "wait_for": "li.next"
  

Note: this parameter is available only when real_browser is set to true and javascript is not disabled.

Wait until

Parameter: wait_until scenario.wait_until loop.wait_until

This parameter allows the browser to wait until a JavaScript snippet will return a Truthy value before capturing the HTML.
Maximum timeout: 10 (seconds).

    
      "wait_until": "document.querySelectorAll('.element').length == 10"

      // or base64 encoded:

      "wait_until": "ZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgnLmVsZW1lbnQnKS5sZW5ndGggPT0gMTA="
    
  

Note: this parameter is available only when real_browser is set to true and javascript is not disabled.

Cookies

Parameter: cookies

This parameter allows you to send custom cookies that will be used for the request.
Format: an object with string values.

    "cookies": { "test": "success", "session": "123asdqwe" }
  

When using this parameter with real_browser set to true, the response will contain the information about all cookies that were set, like in the example below:

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "http://httpbin.org/cookies?json",
      "cookies": { "testing": "true" },
      "real_browser": true,
      "parse": {
        "cookies": "body >> json"
      }
    }
  

  
    {
      "result": {
        "cookies": {
          "cookies": {
            "testing": "true"
          }
        }
      },
      "request": {
        "parse": {
          "cookies": "body >> json"
        },
        "url": "http://httpbin.org/cookies?json",
        "cookies": {
          "testing": "true"
        },
        "real_browser": true,
        "scenario": [
          {
            "execute": "parse"
          }
        ]
      },
      "id": 12345,
      "pages_parsed": 1,
      "cost": 0.002,
      "success": true,
      "extra": {
        "cookies": [
          {
            "name": "testing",
            "value": "true",
            "path": "/",
            "domain": "httpbin.org",
            "expires": null,
            "secure": false
          }
        ]
      },
      "duration": 5.22
    }
  

Execute JavaScript

Parameter: scenario.execute_js loop.execute_js

Page2API can execute custom JavaScript code during the scraping session.
This is useful when you need to interact with the web page while or before parsing.

It is performed via scenario parameter that was described earlier.

  
    document.querySelector('.morelink').click()
  

  
    ZG9jdW1lbnQucXVlcnlTZWxlY3RvcignLm1vcmVsaW5rJykuY2xpY2soKQ==
  

  
    "scenario" : [
      { "execute_js": "document.querySelector('.morelink').click()" },
      { "wait": 0.1 },
      { "execute": "parse" }
    ]
  

Note: this parameter is available only when real_browser is set to true.

JavaScript selectors

Parameter: parse

Page2API can execute custom JavaScript code during the scraping session.
The executed code can be used as a selector to ease the data extraction process.
This is useful when you need to access some data that is stored in the javascript code from the page.

  
    js >> raw_or_base64_js_snippet
  

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://www.page2api.com/",
      "real_browser": true,
      "parse": {
        "title": "js >> $('h1').text().trim()",
        "location": "js >> document.location.href",
        "js_variable": "js >> let object = { arr: [1, 2] }; object",
        "base64_js": "js >> bGV0IG9iaiA9IHsgb25lX3R3bzogWzEsIDJdIH07IG9iag=="
      }
    }
  

  
    {
      "result": {
        "title": "The Ultimate Web Scraping API",
        "location": "https://www.page2api.com/",
        "js_variable": {
          "arr": [1, 2]
        },
        "base64_js": {
          "one_two": [1, 2]
        }
      }
    }
  

Note: this parameter is available only when real_browser is set to true

Fill input [native]

Parameter: scenario.fill_in loop.fill_in

Page2API can fill inputs natively by using the fill_in scenario command.
The format is an array, when the first attribute represents a css/xpath selector, and the second one - the value.

  
    "scenario" : [
      { "fill_in": ["input#search", "Page2API"] },
      { "wait_for": ".search-results" },
      { "execute": "parse" }
    ]
  

Note: this parameter is available only when real_browser is set to true.

Click [native]

Parameter: scenario.click loop.click

Page2API can click natively on visible elements from the page by using the click scenario command.
The format is a string that represents a css/xpath selector of the element that must be clicked.

  
    "scenario" : [
      { "fill_in": ["input#search", "Page2API"] },
      { "click": "button[type=submit]" },
      { "execute": "parse" }
    ]
  

Note: this parameter is available only when real_browser is set to true.

Handle pagination

Parameter: scenario.loop

Page2API can handle paginated views, such as the classic ones with links to the pages, as well as infinite scrolls.
This is made via a loop command from the inside of a scenario.

Note: a loop is just a collection of instructions that are executed in cycle.

  
    "scenario" : [
      {
        "loop" : [
          { "wait_for": "li.next a" },
          { "execute": "parse" },
          { "execute_js": "document.getElementById('proxylisttable_next').click()" }
        ],
        "iterations": 10, // in this case - this parameter is optional
        "stop_condition": "document.getElementById('proxylisttable_next').classList.contains('disabled')"
      }
    ]
  

Note: this parameter is available only when real_browser is set to true.

Batch scraping

Parameter: batch

1. Basic batching (same payload, different URLs)

  
    "urls": [
      "https://companiesmarketcap.com/page/1/",
      "https://companiesmarketcap.com/page/2/",
      "https://companiesmarketcap.com/page/3/",
    ]
  

  
    "urls": "https://companiesmarketcap.com/page/[1, 3, 1]/"
  

  
    [START, END, STEP]
  

  
    {
      "api_key": "YOUR_API_KEY",
      "batch": {
        "concurrency": 3,
        "urls": [
          "https://www.ebay.com/itm/334297414333",
          "https://www.ebay.com/itm/392912936671",
          "https://www.ebay.com/itm/174045421299"
        ]
      },
      "parse": {
        "title": "h1 >> text",
        "price": "#prcIsum >> text",
        "url": "link[rel=canonical] >> href"
      }
    }
  

  
    {
      "api_key": "YOUR_API_KEY",
      "batch": {
        "merge_results": true,
        "concurrency": 3,
        "urls": "https://companiesmarketcap.com/page/[1, 3, 1]/"
      },
      "parse": {
        "data": "table"
      }
    }
  

Note: when performing batch scraping, the url parameter that is usually used to scrape a single page - is optional. (see the examples above)

2. Advanced batching (different payloads, different URLs)

  
    {
      "payloads": [
        {
          "url": "https://httpbin.org/anything?a=1",
          "request_method": "POST",
          "post_body": { "post": true }
        },
        {
          "url": "https://httpbin.org/anything?a=2",
          "request_method": "PUT",
          "post_body": { "put": true }
        }
      ], ...
    }
  

  
    {
      "api_key": "YOUR_API_KEY",
      "batch": {
        "payloads": [
          {
            "url": "https://httpbin.org/anything?a=1",
            "request_method": "POST",
            "post_body": { "post": true }
          },
          {
            "url": "https://httpbin.org/anything?a=2",
            "request_method": "PUT",
            "post_body": { "put": true }
          }
        ],
        "concurrency": 1,
        "merge_results": false
      },
      "parse": {
        "data": "body >> json"
      }
    }
  

  
    {
      "api_key": "YOUR_API_KEY",
      "batch": {
        "payloads": [
          {
            "url": "https://www.page2api.com",
            "parse": {
              "title": "h1 >> text"
            },
            "real_browser": true
          },
          {
            "url": "https://www.example.com",
            "parse": {
              "description": "p >> text"
            }
          }
        ],
        "concurrency": 1,
        "merge_results": false
      }
    }
  

Async scraping

Parameter: async

Usually, any request that takes more than 120 seconds will be interrupted.
In order to handle long running scraping requests (up to 240 seconds), Page2API has the ability of scraping the web pages in the background.
This is made by adding the async parameter to the request and setting it to true.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://www.example.com",
      "async": true,
      "parse": {
        "title": "h1",
      }
    }
  

  
    {
      "id": 123456,
      "performed_async": true
    }
  

After the scraping is done, the result will be sent to the Callback url that you set on your profile page.

  
    {
      "result": {
        "title": "<h1>Example Domain</h1>",
      },
      "request": {
        "parse": {
          "title": "h1"
        },
        "url": "https://www.example.com",
        "callback_url": "https://www.userapplication.com/callback"
      },
      "id": 123456,
      "pages_parsed": 1,
      "cost": 0.00025,
      "success": true,
      "duration": 1.85
    }
  

You can set a custom Callback URL for each of your asynchronous requests

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://www.example.com",
      "callback_url": "https://www.userapplication.com/custom_callback"
      "async": true,
      "parse": {
        "title": "h1",
      }
    }
  

You can also use the passthrough field for your asynchronous requests and this field will be returned within the callback request.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://www.example.com",
      "passthrough": {
        "custom_field": "qwe123asd",
        "passthrough_can_be_integer_string_or_object": true
      },
      "async": true,
      "parse": {
        "title": "h1",
      }
    }
  

  
    {
      "result": {
        "title": "<h1>Example Domain</h1>",
      },
      "request": {
        "parse": {
          "title": "h1"
        },
        "url": "https://www.example.com",
        "passthrough": {
          "custom_field": "qwe123asd",
          "passthrough_can_be_integer_string_or_object": true
        },
        "callback_url": "https://www.userapplication.com/callback"
      },
      "id": 123456,
      "pages_parsed": 1,
      "cost": 0.00025,
      "success": true,
      "duration": 1.85
    }
  

Scheduled scraping

Parameter: refresh_interval

Page2API can create a schedule and scrape web pages automatically in background.
To create a schedule - just add refresh_interval (minutes) parameter to the request with a value between 1 and 2592000 (30 days).
We will run the schedule according to the interval you specified and send the results to your Callback url
or to a custom callback url that you can set per request.

After creating a schedule, you will be able to visualize it, update the interval and delete the schedule intirely.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://www.example.com",
      "async": true,
      "refresh_interval": 5,
      "parse": {
        "title": "h1 >> text",
      }
    }
  

  
    {
      "id": 123456,
      "performed_async": true,
      "schedule_id": 1234
    }
  

  
    {
      "result": {
        "title": "Example Domain",
      },
      "request": {
        "parse": {
          "title": "h1 >> text"
        },
        "url": "https://www.example.com",
        "callback_url": "https://www.userapplication.com/callback"
      },
      "id": 123456,
      "schedule_id": 1234,
      "pages_parsed": 1,
      "cost": 0.00025,
      "success": true,
      "duration": 1.85
    }
  

Note: The schedule_id will be returned in the response, regardless of the async parameter value.

1. Visualize all schedules

  https://www.page2api.com/api/v1/schedules

  GET

  
    [
        {
            "id": 1234,
            "refresh_interval": 5,
            "last_refresh_at": "2021-08-01T09:52:33Z",
            "next_refresh_at": "2021-08-01T09:53:33Z",
            "active": true,
            "options": {
                "url": "https://www.example.com",
                "parse": {
                    "title": "h1 >> text"
                },
                "callback_url": "https://www.userapplication.com/callback"
                "refresh_interval": 5
            },
            "created_at": "2021-08-01T09:51:21Z",
            "updated_at": "2021-08-01T09:52:33Z",
            "scrape_records_count": 14
        }
    ]
  

2. Update a schedule

For a specific Schedule, you can update any of the following parameters:

• parse
• batch
• wait_for
• wait_until
• wait
• scenario
• url
• refresh_interval
• callback_url
• user_agent
• javascript
• cookies
• passthrough
• merge_loops
• log_requests
• raw
• headers
• request_method
• post_body
• premium_proxy
• datacenter_proxy
• real_browser
• absolute_urls
• import_jquery
• custom_proxy
• locale
• sanitize

  https://www.page2api.com/api/v1/schedules/:id

  PUT

  
    {
      "api_key": "YOUR_API_KEY",
      "refresh_interval": 1,
      "callback_url": "https://www.userapplication.com/new_callback_url"
    }
  

  
    {
      "id": 1234,
      "refresh_interval": 1,
      "last_refresh_at": "2021-08-01T09:52:33Z",
      "next_refresh_at": "2021-08-01T09:53:33Z",
      "active": true,
      "options": {
          "url": "https://www.example.com",
          "parse": {
              "title": "h1 >> text"
          },
          "callback_url": "https://www.userapplication.com/new_callback_url"
          "refresh_interval": 1
      },
      "created_at": "2021-08-01T09:51:21Z",
      "updated_at": "2021-08-01T09:52:33Z",
      "scrape_records_count": 14
    }
  

3. Delete a schedule

  https://www.page2api.com/api/v1/schedules/:id

  DELETE

  
    {
      "api_key": "YOUR_API_KEY",
    }
  

  
    {
        "message": "Schedule with id: '1234' was deleted successfully."
    }
  

Datacenter Proxy

Parameter: datacenter_proxy

The Datacenter Proxy is the default proxy used to scrape the web.
The default value is auto. With this value - the API will try to use the most suitable datacenter location for the scraped website.
You can also choose a specific location or set the value to random to pick a random datacenter proxy from the available locations.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://whatismycountry.com",
      "datacenter_proxy": "de",
      "parse": {
        "country": "h2#country >> text"
      }
    }
  

  
    {
      "result": {
        "country": "Your country is Germany"
      },
      "request": {
        "parse": {
            "country": "h2#country >> text"
        },
        "url": "https://whatismycountry.com",
        "premium_proxy": "de",
        "scenario": [
          {
            "execute": "parse"
          }
        ]
      },
      "id": 190,
      "pages_parsed": 1,
      "cost":0.00025,
      "success": true,
      "duration": 1.55
    }
  

Supported locations (6)

Premium Proxy

Parameter: premium_proxy

For hard-to-scrape websites, we offer the possibility to use Premium Proxy, also known as Residential proxy.
Premium Proxy allow you to choose a specific country (or a random one) and surf the web as a real user in that area.

If you set the value to auto - the API will try to use the most suitable premium location for the scraped website.
You can also choose a specific location or set the value to random to pick a random premium proxy from the available locations.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://whatismycountry.com",
      "premium_proxy": "de",
      "parse": {
        "country": "h2#country >> text"
      }
    }
  

  
    {
      "result": {
        "country": "Your country is Germany"
      },
      "request": {
        "parse": {
            "country": "h2#country >> text"
        },
        "url": "https://whatismycountry.com",
        "premium_proxy": "de",
        "scenario": [
          {
            "execute": "parse"
          }
        ]
      },
      "id": 190,
      "pages_parsed": 1,
      "cost":0.0025,
      "success": true,
      "duration": 1.55
    }
  

Supported locations (139)

Custom Proxy

Parameter: custom_proxy

You can provide your own proxy for scraping the web with Page2API.

  
    {
      "api_key": "YOUR_API_KEY",
      "url": "https://api.ipify.org?format=json",
      "custom_proxy": "http://username:[email protected]:1234",
      "parse": {
        "ip": "body >> json"
      }
    }
  

  
    {
      "result": {
         "data": {
            "ip": "192.168.13.14"
          }
        }
      },
      "request": {
        "parse": {
          "ip": "body >> json"
        },
        "url": "https://api.ipify.org?format=json",
        "custom_proxy": "http://username:[email protected]:1234",
        "scenario": [
          {
            "execute": "parse"
          }
        ]
      },
      "id": 190,
      "pages_parsed": 1,
      "cost":0.00025,
      "success": true,
      "duration": 1.55
    }