Is there anybody whom has experience with setting up UI vision on local computer and activate true IP address from website server?
I’m doing web scraping and find that some websites have added many protection layers and can quickly detect if web scraping is done from server. My plan is therefor to setup a local windows computer and then have some sort of program on this computer that will start UI Vision code if it gets contacted true IP/TCP port and returns the full HTML dom posted to server.
Server is linux running with PHP.
If you can help me I’m ready to pay for your coding
Proxy works fine, but my issue is that i need to run UI Vision on 1 computer and remote control this from my linux server running with PHP.
Do you know how to setup remote desktop control or create a program for my windows computer i can use as an API that will then activate my UI Vision script when i call the program with cURL from my server?
Do you know the characteristics that give away your server to websites? If not, how do you know those sites won’t also detect scraping from your non-server computer?
If request speed and volume are the giveaway, just pause between requests.
If request headers are the giveaway, you don’t need a complex scheme to trigger a scrape on another computer; just change the header(s) your server sends.
Maybe it’s cookies not being returned? If so, alter your server app to send the cookies it receives on subsequent requests.
Maybe the sites see your app requesting the html but not one of the referenced graphics? Good luck on that one; you can hack together something the looks for resource refs in the html and makes requests for those, but probably better to automate a browser engine.
Maybe the pages have JavaScript that sends signals to hosts, and the hosts decides an app is a scraper when such signals aren’t received. Browser automation.
If the sites are looking at your IP to see if it belongs to a hosting service, just run your app through a proxy or from home.
Creating an app that listens for and reacts to incoming TCP/IP requests isn’t hard for your average programmer. The bigger issue will be support, as configuring the network on which the PC sits to allow incoming requests to hit the PC can be non-trivial for a non-techie.
Since you’re already using PHP, why not setup IIS or Apache, and PHP, on your Windows box? Then write a simple PHP page that, when hit, kicks off whatever process.