How to use XClickText and XClickTextRelative?

GerCas · May 23, 2024, 3:33am

Hi all,

I’m trying to replace some code that works fine with XClick/XMove in one machine but fails in other due to the screen resolution.

In this thread they suggest to these two OCR commands as a workaround, but I’m not clear in how to select the correct position since if it is resolution independent, is not by pixels or X/Y coordinates.

May someone explain how is the process to know which value move to left/right, up/down?

Additionally, I’ve been trying to make some simple tests with this site but the string is not found.

Only 2 steps:
1-) Locate “example X0C2” with XClickText to write something there
2-) Locate “Log in Another Way” with XClickTextRelative in order to understand how much move to up and right to click “Find button”.

I 've seen command descriptions and video explanation but I’m still have doubts. Thanks

This is my 4 commands macro that is stopping in 2nd command since text is not found.

Line 2: OCR for ‘example X0C2’ not found

{
  "Name": "test2",
  "CreationDate": "2024-5-22",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.examlogin.com/#/login",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "XClickText",
      "Target": "example X0C2",
      "Value": "",
      "Description": "Locate Code box"
    },
    {
      "Command": "XClick",
      "Target": "ABCD",
      "Value": "",
      "Description": "Write ABCD"
    },
    {
      "Command": "XClickTextRelative",
      "Target": "Log In Another Way#R20,5",
      "Value": "",
      "Description": "To click in Find button"
    }
  ]
}

TechSupport · May 23, 2024, 12:23pm

This is a good question, and a very good timing. In the last weeks we have been working on improving the XClickText and XClickTextRelative commands. The next RPA update brings more local OCR options (even on Linux), more OCR languages and wildcard (* and ?) support for more control over text matches.

The following demo already uses the new version:

The video also shows that text recognition works better than image recognition if the screen resolution is dramatically different between system.

macro that is stopping in 2nd command since text is not found.

I found this issue. In the macro you use: "Target": "example X0C2",

This is formally correct but in order to make text recognition commands more robust, we recommend to use the least number of words that uniquely identify a position. In this example, just using the word “example” will do.

The found mouse click position is stored in the ${!ocrX} and ${!ocrY} variables. This works the same as the ${!imageX} and ${!imageY} variables for image recognition. (This is not needed here, I am just mentioning it for completeness).

I will update this post again once the new version is released. It should not take long

GerCas · May 23, 2024, 2:38pm

Hi, thanks you for your answer and video. I still have the doubt how did you know that in 4th command the value to search “Find” button should be 35,4. How do you identify that? Test and error or how?

I understand about to use minimum number of words to make OCR search precise, but when I was trying to select a practice site for you to show, I tried to search a word that appears more than once in the page. Then I needed to search by 2 works separated by space and the result was found in another side of the page even the 2 strings were not present.

Thanks for you help again

admin · May 23, 2024, 5:23pm

Yes, test and error. I start with a good guess (or simply e. g. 10,20), press find and see where it “hits”. Then I adjust the values as needed, and press “Find” again. Usually, after 3-4 rounds I have a good value.

It is important to note that these values (10,20 etc) are not x/y values, they are multiples of the “average length of a character” that Ui.Vision calculates internally. The important part is this value stays constant when the screen scaling changes. So once you found good parameters for XClickTextRelative on one machine, the command work on every machine without changes.

The same applies to all other …TextRelative commands:

XClickTextRelative,
XMoveTextRelative and
OCRExtractTextRelative.

GerCas · May 23, 2024, 5:47pm

Thanks for your explanation. I’ll delete my other question related with this.

Regards

admin · May 27, 2024, 1:19pm

V9.2.0 is available now - it brings multi-word OCR search and wildcard support (*, ?) for all OCR command: XClickText, XMoveText, XClickTextRelative, XMoveTextRelative and OCRSearch.

admin · July 25, 2024, 3:36pm

From our daily Enterprise tech support work: Here is a nice example of how to XClickTextRelative can be a more stable replacement for XClick if the images to find are small and not very unique.

In this example, the first approach was to use

XClick |

to click the GPS icon. However, all icons on this website are small and very similar (small blue buttons with some white symbol inside). So when moving between different machines and screen resolutions, sometimes the wrong icon was selected.

The solution was to replace XClick with XClickTextRelative. Now detect the word “GPS-Position” with OCR, and then click on the icon to the right of it. The screenshots explain this:

XClick Solution for GPS icon click:

XClickTextRelative Solution for GPS icon click: