Anti-bot screen bypass
Ang ganitong uri ng gawain kung saan nagna-navigate ang aming manggagawa sa isang web page na gusto mo, nilalampasan ang anumang anti-bot na screen, kumukuha ng cookies at ibinabalik ang mga ito sa iyong app. Maaari mong gamitin ang cookies na ito upang malayang mag-navigate sa website na ito gamit ang iyong paboritong programming language. Upang matagumpay na ma-bypass ang ganitong uri ng mga page na nakakakuha ng bot, kakailanganin mo rin ang halaga ng User-Agent ng browser ng aming manggagawa at bigyan kami ng proxy na may magandang kalidad. Kung walang mga proxy, hindi gagana ang pamamaraang ito, dahil ang lahat ng anti-bot na solusyon ay tumutugma sa kanilang cookies sa IP address ng bisita at ng kanilang User-Agent.

Ano ang nakuha namin sa huling pahina:
- Mga cookies
- fingerprint ng browser
- Mga header ng huling kahilingan sa HTTP na ginawa mula sa pangunahing window frame
Ang teknolohiyang ito ay binuo sa aming Mga template ng AntiGate, maliban na kami mismo ang namamahala sa mga template at pinapanatili silang na-update. Ang paggamit ng ganitong uri ng gawain ay mangangailangan ng subscription, na may halagang 5 credits para sa bawat gawain. Ang mga presyo ng subscription ay nagsisimula sa $9.90 sa isang buwan na may kasamang 5,000 bypass. Kung hindi mo gustong bumili ng isa, maaari kang bumuo ng isang template sa iyong sarili gamit ang pagtuturo na ito at patuloy na magbayad mula sa iyong balanse sa halagang humigit-kumulang $0.002 bawat gawain.
Ang lahat ng proseso ay kinokontrol ng isang browser plugin, na ini-install ng aming mga manggagawa bago makatanggap ng mga takdang-aralin. Wala silang ginagawang anumang bagay nang manu-mano, karaniwang isang bagong tab ay awtomatikong nabubuksan nang naka-enable ang iyong proxy, nagna-navigate sa page na iyong pinili, naghihintay hanggang sa maipasa ang anti-bot na screen, kumukuha ng data, isinasara ang tab at ipinapasa ang data sa iyong app sa pamamagitan ng API.
Tanging mga de-kalidad na proxy lang ang tinatanggap, walang hostname, walang "residential proxies", walang shared proxy pool. Palaging gagana ang pagrenta ng VPS sa USA/Europe at pag-install ng SQUID server sa aming mga tagubilin. Ang mga proxy ay sinusuri para sa bilis at pagiging tugma bago ang pagpapatupad ng gawain. Dapat ay mayroon silang mabilis na oras ng pagtugon sa ibaba ng 1 segundo, kung hindi, kakanselahin ng mga manggagawa ang iyong mga gawain.
Suriin kung sinusuportahan ng isang website ang anti-bot na screen:
Tandaan na ang ilang mga anti-bot na screen ay gumagamit ng mga advanced na diskarte sa fingerprinting tulad ng SSL handshake fingerprinting, na maaaring matukoy kung anong uri ng SSL/TLS client ang kumokonekta sa website. Halimbawa, ang Chrome browser ay may isang fingerprint, ang Firefox ay isa pa at ang CURL ay isang ganap na naiiba. Huwag ipagkamali ang iyong sarili sa isang User-Agent, ito ay isang bagay sa mas mababang antas. Sa kasong ito upang magamit ang aming serbisyo, kakailanganin mong gumamit ng parehong halimbawa ng browser upang buuin muli ang session ng aming manggagawa. Dahil karamihan sa aming mga manggagawa ay gumagamit ng Chrome browser, isang bundle ng NodeJS+Puppeteer+Chromium o Selenium+Chromedriver ang gagawa ng trabaho. Kasama ang mga halimbawa.
Bagay ng gawain
Katangian | Uri | Kinakailangan | Layunin |
---|---|---|---|
type | String | Oo | AntiBotCookieTask |
websiteURL | String | Oo | Adres ng puntiryang pahina kung saan pupunta ang aming manggagawa. |
proxyAddress | String | Oo | IP address ng proxy ipv4/ipv6. Walang mga host name o IP address mula sa mga lokal na network. |
proxyPort | Integer | Oo | Port ng proxy |
proxyLogin | String | Oo | Login para sa proxy na kinakailangan ang awtorisasyon (basic) |
proxyPassword | String | Oo | Password ng proxy |
Halimbawa ng kahilingan
curl -i -H "Accept: application/json" \
-H "Content-Type: application/json" \
-X POST -d '{
"clientKey":"YOUR_API_KEY_HERE",
"task":
{
"type":"AntiBotCookieTask",
"websiteURL":"https://somewebsite.com/",
"proxyAddress":"8.8.8.8",
"proxyPort":8080,
"proxyLogin":"proxyLoginHere",
"proxyPassword":"proxyPasswordHere"
},
"softId": 0
}' https://api.anti-captcha.com/createTask
<?php
//git clone git@github.com:AdminAnticaptcha/anticaptcha-php.git
include("anticaptcha.php");
include("antibotcookie.php");
$api = new AntiBotCookie();
$api->setVerboseMode(true);
//your anti-captcha.com account key
$api->setKey("YOUR_API_KEY_HERE");
//target website address
$api->setWebsiteURL("https://somewebsite.com/");
//proxy access parameters
// DO NOT USE PURCHASED/RENTED PROXIES ON PROXY SERVICES!!!
// THEY WILL NOT WORK!
// USE ONLY PROXIES YOU INSTALL YOURSELF ON YOUR OWN SERVER OR FAST VPS
// USE PROPER PROXY SOFTWARE LIKE SQUID !
// INSTALLATION INSTRUCTIONS:
// https://anti-captcha.com/apidoc/articles/how-to-install-squid
$api->setProxyAddress("8.8.8.8");
$api->setProxyPort(1234);
//optional login and password
$api->setProxyLogin("login");
$api->setProxyPassword("password");
//Specify softId to earn 10% commission with your app.
//Get your softId here: https://anti-captcha.com/clients/tools/devcenter
$api->setSoftId(0);
//create task in API
if (!$api->createTask()) {
$api->debout("API v2 send failed - ".$api->getErrorMessage(), "red");
return false;
}
$taskId = $api->getTaskId();
//wait in a loop for max 300 seconds till task is solved
if (!$api->waitForResult(300)) {
echo "could not solve captcha\n";
echo $api->getErrorMessage()."\n";
} else {
$result = $api->getTaskSolution();
echo "\nTask result:\n";
print_r($result);
$assoc = json_decode(json_encode($api->getTaskSolution()), true);
$userAgent = $assoc['fingerprint']['self.navigator.userAgent'];
$cookies = [];
foreach ($assoc["cookies"] as $name => $value) {
$cookies[] = $name . "=" . $value;
}
$cookieStr = implode("; ", $cookies);
echo "Use this User-Agent for your requests: $userAgent\n";
echo "Use this cookies for your requests: $cookieStr\n";
//example, may need to pass more headers with CURLOPT_HTTPHEADER:
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, "https://somewebsite.com/");
curl_setopt($ch,CURLOPT_RETURNTRANSFER, 1);
//set worker's user-agent
curl_setopt($ch,CURLOPT_USERAGENT, $userAgent);
//set cookies
curl_setopt($ch,CURLOPT_HTTPHEADER, array(
'Cookie: ' . $cookieStr
));
//set your proxies
curl_setopt($ch,CURLOPT_PROXY, "8.8.8.8:1234");
curl_setopt($ch, CURLOPT_PROXYUSERPWD, "login:password");
//retrieve the content
$content = curl_exec($ch);
}
#pip3 install anticaptchaofficial
from anticaptchaofficial.antibotcookietask import *
import requests
solver = antibotcookieTask()
solver.set_verbose(1)
solver.set_key("API_KEY_HERE")
solver.set_website_url("https://www.somewebsite.com/")
solver.set_proxy_address("1.2.3.4")
solver.set_proxy_port(3128)
solver.set_proxy_login("login")
solver.set_proxy_password("password")
# Specify softId to earn 10% commission with your app.
# Get your softId here: https://anti-captcha.com/clients/tools/devcenter
solver.set_soft_id(0)
result = solver.solve_and_return_solution()
if result == 0:
print("could not solve task")
exit()
print(result)
cookies, localStorage, fingerprint = result["cookies"], result["localStorage"], result["fingerprint"]
if len(cookies) == 0:
print("empty cookies, try again")
exit()
cookie_string = '; '.join([f'{key}={value}' for key, value in cookies.items()])
user_agent = fingerprint['self.navigator.userAgent']
print(f"use these cookies for requests: {cookie_string}")
print(f"use this user-agent for requests: {user_agent}")
s = requests.Session()
proxies = {
"http": "http://login:password@1.2.3.4:3128",
"https": "http://login:password@1.2.3.4:3128"
}
s.proxies = proxies
content = s.get("https://www.somewebsite.com/", headers={
"Cookie": cookie_string,
"User-Agent": user_agent
}).text
print(content)
//npm install @antiadmin/anticaptchaofficial
//https://github.com/AdminAnticaptcha/anticaptcha-npm
const ac = require("@antiadmin/anticaptchaofficial");
//set API key
ac.setAPIKey('YOUR_API_KEY_HERE');
//Specify softId to earn 10% commission with your app.
//Get your softId here: https://anti-captcha.com/clients/tools/devcenter
ac.setSoftId(0);
(async () => {
let solution = null;
try {
solution = await anticaptcha.solveAntiBotCookieTask(
checkUrl,
proxyAddress,
proxyPort,
proxyLogin,
proxyPassword);
} catch (e) {
console.error("could not solve captcha: "+e.toString());
return;
}
const fingerPrint = solution.fingerprint;
const targetCookies = joinCookies(solution.cookies);
console.log(`joined cookies: ${targetCookies}`);
console.log(`user-agent: ${fingerPrint['self.navigator.userAgent']}`);
try {
let responseText = await axios.request({
url: checkUrl,
httpsAgent: agent,
headers: {
'User-Agent': fingerPrint['self.navigator.userAgent'],
'Cookie': targetCookies,
'Accept-Encoding': 'deflate',
'Accept': 'text/html',
'Accept-Language': 'en'
}
});
console.log(responseText.data);
} catch (e) {
console.error('Could not request page')
console.log(e.toString());
}
})();
function joinCookies(object) {
let resultArray = [];
for (const key in object) {
resultArray.push(key+"="+object[key])
}
return resultArray.join("; ");
}
/*
npm install @antiadmin/anticaptchaofficial puppeteer
* */
const anticaptcha = require("@antiadmin/anticaptchaofficial");
const pup = require('puppeteer');
//address behind cloudflare
const checkUrl = 'https://www.thewebsite.com/';
const domainName = 'www.thewebsite.com';
//Anti-captcha.com API key
const apiKey = 'API_KEY_HERE';
// STOP! IMPORTANT! Shared proxy services won't work!
// Use ONLY self-installed proxies on your own infrastructure! Instruction: https://anti-captcha.com/apidoc/articles/how-to-install-squid
// Again and again people people insist they have best purchased proxies. NO YOU DO NOT!
// Absolutely recommended to read this FAQ about proxies: https://anti-captcha.com/faq/510_questions_about_solving_recaptcha_with_proxy__applies_to_funcaptcha__geetest__hcaptcha_
const proxyAddress = '1.2.3.4';
const proxyPort = 1234;
const proxyLogin = 'mylogin';
const proxyPassword = 'mypass';
let browser = null;
let page = null;
(async () => {
anticaptcha.setAPIKey(apiKey);
const balance = await anticaptcha.getBalance();
if (balance <= 0) {
console.log('Topup your anti-captcha.com balance!');
return;
} else {
console.log('API key balance is '+balance+', continuing');
// anticaptcha.shutUp(); //uncomment for silent captcha recognition
}
let antigateResult = null;
try {
antigateResult = await anticaptcha.solveAntiBotCookieTask(
checkUrl,
proxyAddress,
proxyPort,
proxyLogin,
proxyPassword);
} catch (e) {
console.error("could not solve captcha: "+e.toString());
return;
}
const fingerPrint = antigateResult.fingerprint;
try {
console.log('opening browser ..');
let options = {
headless: true, //disable to see the browser window
devtools: false, //enable to see developers console
args: [
'--window-size='+fingerPrint['self.screen.width']+','+fingerPrint['self.screen.height'],
`--proxy-server=${proxyAddress}:${proxyPort}`
],
};
browser = await pup.launch(options);
console.log('creating new page ..');
page = await browser.newPage();
} catch (e) {
console.log("could not open browser: "+e);
return false;
}
if (proxyPassword && proxyLogin) {
console.log(`setting proxy authentication ${proxyLogin}:${proxyPassword}`);
await page.authenticate({
username: proxyLogin,
password: proxyPassword,
});
}
//screen size
console.log('setting view port to '+fingerPrint['self.screen.width']+'x'+fingerPrint['self.screen.height']);
await page.setViewport({width: fingerPrint['self.screen.width'], height: fingerPrint['self.screen.height']});
//user agent
let userAgent = '';
if (fingerPrint['self.navigator.userAgent']) {
userAgent = fingerPrint['self.navigator.userAgent'];
} else {
if (fingerPrint['self.navigator.appVersion'] && fingerPrint['self.navigator.appCodeName']) {
userAgent = fingerPrint['self.navigator.appCodeName'] + '/' + fingerPrint['self.navigator.appVersion']
}
}
console.log('setting browser user agent to '+userAgent);
await page.setUserAgent(userAgent);
console.log('setting cookies', antigateResult.cookies);
let cookies = [];
for (const name in antigateResult.cookies) {
cookies.push({ name: name, value: antigateResult.cookies[name], domain: domainName })
}
await page.setCookie(...cookies);
try {
await page.goto(antigateResult.url, {
waitUntil: "networkidle0"
});
} catch (e) {
console.log('err while loading the page: '+e);
}
const htmlContent = await page.content();
console.log("page content:\n\n", htmlContent);
})();
Halimbawa ng tugon
{
"errorId": 0,
"taskId": 7654321
}
{
"errorId": 1,
"errorCode": "ERROR_KEY_DOES_NOT_EXIST",
"errorDescription": "Account authorization key not found in the system"
}
Kunin ang solusyon
Bagay ng solusyon ng gawain
Katangian | Uri | Layunin |
---|---|---|
cookies | Bagay | Cookies mula sa pahina sa likod ng anti-bot screen. Samahan silang lahat at gamitin sa iyong mga kahilingan sa HTTP. |
localStorage | Bagay | Kapareho ng mga cookie, ang bagay na may mga localStorage na halaga ay kinuha sa huling pahina. |
fingerprint | Bagay | Mga parametro ng fingerprint ng browser. Gamitin ang mga ito kasama ng mga cookie at localStorage para maisagawa ang sesyon ng browser ng manggagawa sa iyong browser. Gamitin ang halaga ng self.navigator.userAgent bilang isang user-agent sa iyong mga kahilingan sa HTTP. |
url | String | URL ng pahina kung saan isinagawa ang pagtatapos ng template |
lastRequestHeaders | Array | Mga header ng huling kahilingan na ipinadala sa web site mula sa pangunahing window frame ng browser. |
Halimbawa ng tugon
{
"errorId": 0,
"status": "ready",
"solution": {
"cookies": {
"some_antibotcookie": "0A8VO9NX5N1s4LRoS4sJlFTCNzLj0dEfA_2whUh0E6ZjgQtM~I1cV7U2IhQx0~jnowNjg-Oi76b-MjYPd1GQAmIxh5-v~33PI8F",
"maybe_another_id": "join_all_cookies_together"
},
"localStorage": {
"some_value": "Might be used too in the future as a method to 'remember' visitors, so we collect it too.",
"what_is_it": "localStorage is a more complex analogue of cookies, allowing to store larger objects in browser memory"
},
"fingerprint": {
"self.navigator.userAgent": "Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/101.0.0.0 Safari\/537.36",
"self.screen.width": 1280,
"self.screen.height": 768,
"self.screen.availWidth": 1280,
"self.screen.availHeight": 768,
"self.screen.availLeft": 0,
"self.screen.availTop": 25,
"self.navigator.vendorSub": "",
"self.navigator.productSub": "20030107",
"self.navigator.vendor": "Google Inc.",
"self.navigator.maxTouchPoints": 0,
"self.navigator.hardwareConcurrency": 8,
"self.navigator.cookieEnabled": true,
"self.navigator.appCodeName": "Mozilla",
"self.navigator.appName": "Netscape",
"self.navigator.appVersion": "5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/101.0.0.0 Safari\/537.36",
"self.navigator.platform": "MacIntel",
"self.navigator.product": "Gecko",
"self.navigator.language": "en-US",
"self.navigator.onLine": true,
"self.navigator.deviceMemory": 4
},
"url": "https://www.thewebsite.com/some/final/path/after_redirects",
"lastRequestHeaders": [
"sec-ch-device-memory: 8",
"sec-ch-ua: \" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\"",
"sec-ch-ua-mobile: ?0",
"sec-ch-ua-arch: \"x86\"",
"sec-ch-ua-platform: \"macOS\"",
"sec-ch-ua-model: \"\"",
"sec-ch-ua-full-version-list: \" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"101.0.5005.115\", \"Google Chrome\";v=\"101.0.5005.115\"",
"Upgrade-Insecure-Requests: 1",
"User-Agent: Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/101.0.0.0 Safari\/537.36",
"Accept: text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/avif,image\/webp,image\/apng,*\/*;q=0.8,application\/signed-exchange;v=b3;q=0.9",
"Sec-Fetch-Site: same-origin",
"Sec-Fetch-Mode: navigate",
"Sec-Fetch-Dest: document",
"Referer: https:\/\/somewebsite.com\/",
"Accept-Encoding: gzip, deflate, br",
"Accept-Language: en-US,en;q=0.9",
"Cookie: some_antibotcookie=0A8VO9NX5N1s4LRoS4sJlFTCNzLj0dEfA_2whUh0E6ZjgQtM~I1cV7U2IhQx0~jnowNjg-Oi76b-MjYPd1GQAmIxh5-v~33PI8F"
]
},
"cost": "0.00858",
"ip": "5.25.11.114",
"createTime": 1637841143,
"endTime": 1637841189,
"solveCount": 0
}