Issues with Desktop Mode on Linux

I used RPA on Chrome under Linux in the browser and was very impressed. Next step: desktop. I got everything installed and it just didn’t want to work. I am using Chrome beta, but I did copy over the Native messaging bits to the right place. The program took screen shots for select, but on find or when running the script it would either hang or timeout on small normal size targets. If I made a large target it would find many many hits none of which were correct at 0.8 confidence and less.

So I tried Windows and Chrome. With two monitors the program didn’t work. First, the select button shows only the main monitor. And when running, the mouse moves to the other monitor to about where it should have been on the second monitor. I suspect it might have worked if my main monitor’s origin was at 0,0 but it isn’t.

So with a single monitor on Windows 10 it does work well. So I know I have everything installed and I know how to use it. But multimonitors no, and Linux (multi or single) doesn’t seem to work well at all.

Just letting you know.

The lack of multi-monitor support is a known issue, see [issue #41] XClick fails if second monitor is connected

But desktop automation for Linux should work well! What flavor of Linux are you using?

Thanks for the reply. This is on two different computers both running KDE Neon. I wonder if it is a Chrome extension issue?

I tried today on Firefox and have the same problem. Shut off the second monitor just in case.

AT 0.8 the find button does nothing. At 0.5 it shows dozens or hundreds of hits, none of them correct.

Oh, I think there is a problem on some Linux systems. I just tested on Ubuntu 20.04 LTS and found that in desktop mode the UIVision integrated input image selector (= when you click the SELECT button) does not work - all saved images are empty => then of course XClick and all other visual UI testing commands do not find anything.

Workaround: Use an external screenshot app, for the example the one that ships with your system. Use it to create the input images and store the images in the UiVision /image directory, then all works fine!

=> Can you confirm that this solution works for KDE, too?

Example:

  • app_dpi_96.png was taken by UIVision => empty :frowning:
  • ubuapp_dpi_96.png was taken by Ubuntu’s screenshot app => normal :slight_smile:

ubuapp_dpi_96.png works with FIND button:

ubuapp_dpi_96.png works when macro runs: (it finds the calc app header and clicks on it):

This doesn’t seem to work. First, I had to switch to file system mode, I think. So I did that. I will say that even using your capture, I see the image when I float over the code line.But your image or mine, both either have no hits at high confidence level or many spurious hits (including NO HITS on the actual target) when set to a low confidence.

I will try on the other computer soon.

Here’s an example from your capture:
ngeujt_dpi_168

Here is a search at 0.5 confidence

At 0.8 it logs “not found”

I will say these are all high dpi displays and probably scaled. Maybe that has something to do with it?

Ok it isn’t scaling but I think I know now what it is.

If I try to do it on anything that is part of Chrome, it shows up on the capture but not on the find or macros execution. However, if I do something in the non-client area (e.g. a menu) or in another non-chrome app, it works OK.

I imagine it is Hardware Acceleration in Chrome but I’m tired and will try that another time.

No, actually I think it is totally scaling. On one machine I now have it working as long as I have one monitor scaled at 100%. Oddly, higher scaling doesn’t work the way you’d think. But at 100% it even works with Chrome.

Will try on the other machine later.

This should be fixed with the V2021-2 XModule AppImage.