Universal Scraping API User Guide
阅读目录
Last Updated: March 17, 2025
This document is intended for developers to efficiently and conveniently integrate data and quickly connect to the API.
API Name
General Collection API
API Description
Used for convenient collection of public network data.
Request URL
http://xscrape.pangolinfo.com/scrape/v2
Request Methods
POST / GET
Supports both POST and GET request methods.
Parameters
POST Request Parameters
| Parameter Name | Parameter Type | Description |
|---|---|---|
token | String | User authentication information, please contact the administrator to obtain it. |
GET Request Parameters
| Parameter Name | Parameter Type | Description |
|---|---|---|
token | String | User authentication information, please contact the administrator to obtain it. |
task | String | Task information, note that URLEncoder encoding is required. The task format is as follows: |
{
"url": "https://x.com/elonmusk",
"responseFilter": {
"name": "urlRuleFilter",
"values": ["UserTweets"]
}
}
Task Attribute Description
| Parameter Name | Parameter Type | Required | Description |
|---|---|---|---|
url | String | Yes | Target page URL. |
responseFilter.name | String | No | Response filter name, enumerated values: |
urlRuleFilter(URL filter)resourceTypeFilter(Resource type filter) | |responseFilter.values| String Array | No | |
Response Parameters
{
"code": 0,
"message": "ok",
"completedTime": 1741749438354,
"data": {
"taskId": "239b1e398c1f472180e4e016568d014f",
"xhrs": [
],
"documents": [
],
"imgs": [
]
}
}
Error Codes
1001
- Meaning: Parameter is empty / Incorrect parameter
- Solution: Check if the request parameters are correct
1004
- Meaning: Access denied / Invalid token / Exceeded trial limit
- Solution: Please check the token
Example Requests
- Google Search and Amazon front-end data collection are pre-built and optimized for easier user access. More mainstream sites will be added gradually.
- Twitter and Walmart collection use a general collection solution.
Details are as follows:
1. Google Search
Trial URL:
http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&task=%7B%22url%22%3A%20%22https%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Desk%22%7D
# Request
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{"url": "https://www.google.com/search?q=desk"}'
2、Amazon
Trial：
Parameter Name | Parameter Type | Required | Description |
|---|---|---|---|
url | String | Yes | Amazon target page URL. |
parserName | String | No | Parser name, includes the following enumerated values: – amzKeyword: Keyword search results page – amzProductDetail: Product detail page – amzProductOfCategory: Product list page under a category – amzProductOfSeller: Seller’s product list page – amzBestSellers: Best Sellers ranking – amzNewReleases: New Releases ranking |
bizContext.zipcode | String | No | Postal Code, supports any valid postal code. Formats are as follows: United States: |
Example 1: Scrape https://www.amazon.com/s?k=desk，Request:
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{"url": "https://www.amazon.com/s?k=desk"}'
Example 2: Scrape https://www.amazon.com/s?k=desk with Zipcode 10041,Request:
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{"url": "https://www.amazon.com/s?k=desk","bizContext":{"zipcode":"10041"}}'
Example 3: Scrape https://www.amazon.com/s?k=desk with Zipcode 10041 and Parser amzKeyword，Request:
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{"url": "https://www.amazon.com/s?k=desk","parserName":"amzKeyword","bizContext":{"zipcode":"10041"}}'
Twitter Profile Scraping
POST Request
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '
{
"url": "https://x.com/elonmusk",
"responseFilter": {
"name": "urlRuleFilter",
"values": [
"UserTweets"
]
}
}'
GET Request
http://xscrape.pangolinfo.com/scrape/v1?token=c2f4c686ee2a41369a44db5d7aa0fe09&task=%7B%22url%22%3A%22https%3A%2F%2Fx.com%2Felonmusk%22%2C%22responseFilter%22%3A%7B%22name%22%3A%22urlRuleFilter%22%2C%22values%22%3A%5B%22UserTweets%22%5D%7D%7D
# Walmart Product List Page
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{
"url":"https://www.walmart.com/shop/clothing-and-accessories/mens-new-arrivals?povid=GlobalNav_rWeb_ClothingShoesAccessories_NewArrivals_MensNewArrivals",
"responseFilter": {
"name": "urlRuleFilter",
"values": [
"https://www.walmart.com/shop"
]
},
"blockResources":["font","image","media"]
}'
# Walmart Product Detail Page
curl --location 'http://xscrape.pangolinfo.com/scrape/v1?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data '{
"url":"https://www.walmart.com/ip/George-Men-s-Slim-Chino-Pants/366426548?classType=VARIANT&athbdg=L1600",
"responseFilter": {
"name": "urlRuleFilter",
"values": [
"https://www.walmart.com/ip"
]
},
"blockResources":["font","image","media"]
}'
准备好开始了吗？
无需信用卡，注册即可获得300积分
