How to use VisualLimitSearch, XMove efficiently

Hi all,

I try to make a login in a website similar to facebook.com. My goal is do the process using Visual and X Commands (XMove, Xclick, Xtype, etc) to simulate human since it has an antibot system. Then if I use html elements like Click|Xpath won’t work.

I’ve tried for hours to get a working code using XMove but simply is no locating the image selected due that finds some similar areas. Then I’d like to limit the search area but I don’t know how it works that command and I haven´t found an example.

My idea/thought is to proceed like this:
1-) Limit Search Area to focus in this square
image
2-) Move (with XMove) to “Email or phone number” box and XType the email address
3-) Move (with XMove) to “Create new account” area
3-) Move (with XMove) to “Password” box and Xtype the password
4-) Move (with XMove) to “Forgot password” area
5-) Move (with XMove) to “Login” button and XClick the button

I’d like to get that movement behavior to have more possibilities to fool the antibot.

Maybe some expert could help me with an example on how to do this, since this macro will be used in more than one machine and by experience, the image taken in one machine is not found in another one by Visual commands.

Thanks in advance

My sample code is like this

{
  "Name": "test",
  "CreationDate": "2024-10-20",
  "Commands": [
    {
      "Command": "comment",
      "Target": "open // https://www.facebook.com/",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "visionLimitSearchAreaRelative",
      "Target": "limit_area_dpi_90.png",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XMoveRelative",
      "Target": "email_or_phone_dpi_90.png@0.4",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClick",
      "Target": "email_or_phone_dpi_90.png",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XType",
      "Target": "email@example.com",
      "Value": "",
      "Description": ""
    }
    ]
}

That surprises me. Can you post a video of your macro running? Maybe you selected the image area to narrow?

Example:

  • Bad XClick image (too small, not many unique features, just mostly white space and some light grey text):

image

  • Good XClick image. It will also click directly in the middle of the password box. But the image has many unique structures in it, which make it stable to find:

image

I automate many logins with XClick, and it works fine. I can help.

PS: Usually XClick alone is sufficient. No need to use XMove here.

Yeah, that can be a challenge. There are some tricks to improve the image portability, but often it is easier to redo the image on the new machine. But the best method is to use XClickText. This uses OCR and is very portable across all kinds of screens and resolutions!

Here my test with Facebook. I use relative OCR twice, to use a better (easer to read) anchor word. For example, the white on blue “Log In” text is more difficult to OCR then the word “Forgot” just below the button. So I use the word “Forgot” and the RPA software to click above this word.

{
  "Name": "facebook",
  "CreationDate": "2024-10-21",
  "Commands": [
    {
      "Command": "store",
      "Target": "98",
      "Value": "!ocrengine",
      "Description": "Javascript OCR"
    },
    {
      "Command": "XClickTextRelative",
      "Target": "Email#R0,-9",
      "Value": "",
      "Description": "Find Email, then click below in the Password field. So the Email text is still there."
    },
    {
      "Command": "XType",
      "Target": "123456",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClickText",
      "Target": "Email",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XType",
      "Target": "Email@Test.com",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClickTextRelative",
      "Target": "Forgot#R1,5",
      "Value": "",
      "Description": ""
    }
  ]
}

Thanks so much @ulrich for your help. I think I could use XclickText in some parts but I use XClick or XMove since sometimes there is no text in the webpage to search. Regarding your example, I copy and paste your code and not sure why is not working. You can see the video below.

The video is helpful. The click in the email field, is that you or the software?

Also, try to change

"Target": "Email#R0,-9",

to

"Target": "Email#R1,-9",

Does it work now?

I’ve tried changing to “Target”: “Email#R1,-9” but doesn´t work either.

The click in email field happens automatically (by default) if you add the “Open|https://www.facebook.com/” at the beginning.

Additionally, If I select some text at the bottom, for example “for a celebrity, brand or business” and after that I run the macro, happens in that case, that the macro completes all commands without error but is not filling the email and password fields.

Confirmed. => The solution is to click somewhere else on the website to move the focus away from the input box. Otherwise the word “Email” is not readable. The below macro does this. I added a XClickText | Face*, that is the only change.

Also, tested on on a completely different system, and it works fine, there, too. OCR is really very portable.

{
  "Name": "facebook",
  "CreationDate": "2024-10-22",
  "Commands": [
    {
      "Command": "store",
      "Target": "98",
      "Value": "!ocrengine",
      "Description": "Javascript OCR"
    },
    {
      "Command": "open",
      "Target": "https://www.facebook.com/",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClickText",
      "Target": "Face*",
      "Value": "",
      "Description": "Set focus away from entry book so words remain readable for OCR"
    },
    {
      "Command": "XClickTextRelative",
      "Target": "Email#R1,-9",
      "Value": "",
      "Description": "Find Email, then click below in the Password field"
    },
    {
      "Command": "XType",
      "Target": "123456",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClickText",
      "Target": "Email",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XType",
      "Target": "Email@Test.com",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XMoveTextRelative",
      "Target": "Forgot*@pos=1#R1,8",
      "Value": "",
      "Description": ""
    }
  ]
}

Hi @ulrich thanks for the help. Not sure why in my case the macro completes all command without doing anything. See video please

Version installed is 9.3.2 trying in windows10 and in these 2 Chrome versions
Version 129.0.6668.101
Version 130.0.6723.59

In this other video you can see that “facebook” is found since appears a square in pink line but doesn´t fill email and password boxes.

For the second video with the sidebar, one issue can be that that sidebar is on the left. If so, you need to check this box:

sidebar is on the left.

But for the first video… I am not sure. It seems it clicks on the wrong areas.

If you run the DemoXMove macro, does this work for you? And is this on Mac, Windows or Linux?

Hi @ulrich, I’ve checked now both boxes you have checked related with sidepanel. For me none of them was checked. Then I run the macro again resizing manually ui vision interface and chrome to see both side to side, having UI vision to left (exactly as in my last video) and the result was exactly in my last video, didn´t work. Then I closed UI vision and opened again and the sidepanel appeared to the right. This time writes “12345” in email box and nothing else (see the video below)

As mentioned before, I’m using UI version 9.3.2 trying in windows10 and in these 2 Chrome versions
Version 129.0.6668.101
Version 130.0.6723.59

And I don´t have a lot of Demo macros, including DemoXMove and not sure why. Where can I download the demo macros?

So it seems the XClickTextRelative | Email#R1,-9 mouse click goes to the wrong place!

As a test, please select the line and click the FIND button. => Where is the pink square box displayed? In my test it is on top of the password entry field - that is where the 123456 should be entered:

Good morning @ulrich

When I try to test the command with “find” the error

[Line 4] invalid API key appears

The macro includes setting the !OCREngine value to 98, which is the Javascript OCR engine. But the “FIND” button uses whatever is set as the default OCR engine in the OCR tab => So, you need to set the default OCR Engine to the Javascript OCR engine:

set the default OCR Engine to the Javascript OCR engine

Selecting Javascript OCR engine now the “find” button shows this

But still is typing 12345 in email box (see video below).

Additionally, when the error that shows this video is when is active side panel, but the behaviour is different when the IDE is in some place to the left over the screen or even if I minimize the IDE, the macro completes all commands without doing anything.

Interesting. Text search/OCR finds the right spot, but the click itself seems to hit the “Facebook” word on the left. That is why the text entry fails for you! For me it hits the password box as expected:

image

@admin - any idea why this could be?

please se my latest post, I’ve updated with a video and I mention that the macro behavior is different when the IDE is over or minimized or side panel. Shouldt be the same result right? no matter where is the IDE or side panel.

@Rasec I find this mysterious. Can you maybe try on another machine? Or maybe use Firefox instead of Chrome?

I tested the macro on a Mac Sequoia 15.0.1 today, and it works flawlessly there, too:

@ulrich I’ve tried in same machine in Firefox and using the same config and settings you said. In this case the “Find” button behaves different as shown in image below.

With all these issues I understand why some things that are supposed to work, don´t work in my case and why some macros that worked some time ago, now begin to fail.

Besides that, do you know where to download the Demo macros to test the demo macros that have been suggested?

Hi @admin

Have you had chance to see this issue? @ulrich have been helping me and even my question was related with search by image and how to limit visual search area, the solution provided by ulrich was using XClickText and XClickTextRelative, but doesn´t work in my system Windows 10 in Chrome and Firefox. Actually, I opened the question since something that should work with XMove, XClick using images don´t work or suddenly stopped working.

Thanks for the help in advance