How to import cookies from a website

Ui.Vision
1

Hi,

Is there a way to import cookies from a webpage?

Business Case: We need to make multiple (around 50,000) http calls on a webpage to scrape data and load into our database. To get to that webpage, RPA needs to click a few buttons like login / read T&Cs / agree etc. Once we get to the page, the cookies need to be imported and stored in a file which will be used in a Java program to make HTTP calls.

Any pointers will be really helpful.

2

You can use XRUN to call scripts to import or export cookies.

Question: How would you manually do this task?

3

@ulrich : Below are the steps to do the same task.

  1. Open main website
  2. Login
  3. Agree to T&Cs, click Next.
  4. Go to the webpage.
  5. On chrome, right click and open Inspect.
  6. Refresh the webpage.
  7. Go to network Tab and copy the cookie details from Request header.
  8. Copy the details in the text file.
  9. Run the Java process which will use the details from the text file.

Currently, steps 1 - 4 are automated using UI.Vision RPA.
5 - 8 are still manual and Iā€™m looking for a workaround.

4

Hi @Gaurav_S

Try with some addons to export and import cookies or a simple batch file to move files (cookies are files)-

5

@newuserkantu : Excellent idea mate. I wonder why didnā€™t I think of this. Thanks so much.

6

If you need to automate another extension, see How to trigger a click on a chrome extension button?

7

Thanks so so much mate. This link has been really helpful since I was having so much trouble decrypting binary passwords from the cookie file.

8

@ulrich : Thanks for your help so far. I have managed to create the macro with what I needed. It runs as expected using the windows batch file as well. But when I tried to run that batch file using a scheduled task, its throwing the following error.

Status=Error: Image ā€˜Extension_dpi_96.pngā€™ (conf. = 0.8) not found

This is the step when it tries to click the chromeā€™s cookies extension.

Any way to get around this issue?

9

Do you keep the desktop unlocked? This is required for using XClick. See here:

https://ui.vision/howto/taskscheduler#howtorun

Personally I mostly use the option 1 from this link, with a small Ubuntu 20.04 LTS Virtual Box:

How to run desktop automation on a locked machine

Option 1: Run RPA inside a virtual machine (VM)

The most secure way to run desktop automation unattended is to use a virtual machine. For desktop automation to work, you need to keep the desktop unlocked. This is true for all RPA software tools. The best way to do this securely is to run the automation in an unlocked virtual machine (VM) inside a locked server. For the virtual machine, we recommend to use a Linux machine, since Linux VMs are much smaller and faster then a comparable Windows VM. And there are no license issues with Linux.

10

Thanks @ulrich but when I used the below statement, the XClick commands started failing with the image not found error.

%windir%\System32\tscon.exe RDP-Tcp#NNN /dest:console

Status=Error: Image ā€˜r7e4gl_dpi_96.pngā€™ (conf. = 0.8) not found

Is this expected?

We are not allowed to install virtual machines on our AWS windows servers, so I canā€™t use Option 1 at the moment.

11

This is not expected, but I am not sure what goes wrong. I do not use this ā€œtscon.exeā€ option. If I can not use the VM option, I use one of these two options below:

Two other suggestions:

  • Instead of RDP, connect to your machine with a tool like Anydesk or Teamviewer. Unlike RDP, they do not lock the screen after disconnect!

  • Or use autologin - then you only need to reboot after the RDP disconnect and the screen is open again. That is what I use, too. It keeps everything working even if the system rebooted itself for some reason, e. g. after an OS update.

  • For both solutions, make sure any screensaver is turned off - so the system does not lock the screen by itself.

12

I suggest you to think a easy and fast way to do the same work.

This works seems too confusion and complicated, itā€™s easy to have more others problems with this scenario.

This is not a simple cookies movement.

Open main website
Login
Agree to T&Cs, click Next.
Go to the webpage.
On chrome, right click and open Inspect.
Refresh the webpage.
Go to network Tab and copy the cookie details from Request header.
Copy the details in the text file.
Run the Java process which will use the details from the text file.

Currently, steps 1 - 4 are automated using UI.Vision RPA.
5 - 8 are still manual and Iā€™m looking for a workaround.

13

@newuserkantu : Definitely agree, mate. I havenā€™t been able to do Steps 1-4 programatically hence had to take the RPA route. Iā€™m now reading and decrypting the cookies using a python code (link below) which has been working as expected the past couple of days. Chrome 80+ cookie decrypter

Now I donā€™t need to use RPA for rest of my steps.

1 Like