How to Scrape Pop-Up

Win7, Chrome, UI.Vision RPA 5.5.7 and XModules downloaded and installed last week.

Newbie here… need to read and control home thermostat using Lennox windows web dashboard for iComfort thermostats. Have successfully created macros to capture and store data from and click on most clickable items but:

  1. clicking on “program settings” causes dropdown window to appear with more clickable selections. With the RPA “click/select”, I can get UIV to click on “program Settings” to cause the dropdown to appear but as soon as I returnto/click on the ui-vision programming panel to select from the dropdown, the dropdown disappears so have nothing to steer the click “Select” mouse to to identify the field on the pop-up to be clicked on. Same problem for the Fan drop-down selections.
  2. also cannot capture the current program setting that is displayed on the “program settings” button… e.g. will not capture “winter” when storeText/select on attached picture w/o popup window showing.

Screen capture 1 shows the dashboard and capture 2 shows the popup window open.

Is there a way to get RPA to storeText/select data from the program settings button?
Is there a way to get RPA to click/select selections from the drop-down window?

iComfort popup showing

There is no direct url link to the .aspx page… only to the login page… don’t know how to provide access to the control page without providing my login information which would expose HVAC control of my house. If helpful, I can provide the source html with sensitive info removed… very large file to wade though.

What button is this? I can not find it. I mean, I guess see the button what there is nothing interesting (to scrape) on it, or? What info do you need to scrape?

Can you please add a screencast of this? Normally the standard web recording feature should capture this sequence.

If that fails, use XClick (se )

Thanks, ulrich for the comeback.

The “button” is in the “Mode Settings” area, Program Tab, labeled on top “Program Settings” and displaying on the button the current setting: “save energy” in the attached captures (was “winter” in the previous post). When do click command / select and click on this button, RPA does not capture the current setting “Save Energy” which is what I need to read.

In addition to capturing the current setting, I need to be able to change the setting which requires clicking on the new setting that is in the drop-down box that appears when one clicks on the Program Settings button. Not many upload options for the forum… I attached a .png file showing clips of the process. I also uploaded an mp4 video that shows the normal usage of the web dashboard (the RPA is not on the PC that my screen capture software is on): https://drive.google.com/file/d/1kGw5on9v0FoaSd4NyBWvdyRAmAtDBJAc/view?usp=sharing

Bump… would really appreciate help here. Also emailed problem to UIV support 2 weeks ago but received no reply…

  1. To get the text, screen-scrape it with OCR with OCRExtractRelative. Your input image could be like this:

xclickrelative

Whatever is inside the pink box gets OCRed.

  1. To automate it, I am sure that XClick and Xtype will work. But without website access, it is tricky to see where exactly it currently hangs.

Thanks, ulrich!
Think (?) I did the screen-scrape correctly but the OCR did not find or did not read the Program Setting (summer) and the value winds up blank. I note that the full screen OCR overlay does not read/see the setting (summer) as there is no overlay for that field… perhaps that is the problem?


2 Pgm Commands 3 OCR Anchor and Target Selection 4 Saved Anchor and Target 5 Log of Macro Execution 6 Variables Captured (summer not read) 7 last screen shot summer 8 csv output - Mode blank

Exactly! This is the problem. Luckily the solution is easy. => Use OCR Engine2 then it it works.

You change the OCR engine in the macro or globally:

In macro:
store | 2 | !OCRengine

Global default:
e2

How did I find this out? I used your screenshot and tested it with the online ocr api, which uses the same two OCR engines (no surprise, same company :slight_smile: ): There you see that with Engine 2 it works fine, and “Summer” is detected ok:

Still got blank…
1OCREngine2 2OCRoverlay 3SelectArea 4Anchor&Target 5Commands 6AfterSaveAndRunMacro-Log (2)

As a test, please make the pink box larger. This way it will contain other text. At least this text you should see.

Also: Check the _lastscreenshot image in the Screenshots tab after the OCRExtractRelative command was executed. It should contain the content of the pink box, that is send to the OCR API for processing. This way you can check the the green/pink frame detection and image matching worked ok.

I would also use an ECHO right after OCRExtractRelative to check on the content of ${ProgramCode}, just to be sure the issue is with the OCR and not the CSV.

Larger pink box example:
text ocr

Scraped this time! But variable is “Program Settings\nsummer” and csv winds up with “Program Settings”
Does this provide you with clue on how to scrape just summer?
Or (perhaps easier) show me how to parse out the mode from the scraped variable?
Thanks so much for your help!

1Commands 2Visual 3Log 4Variables 5Screenshot 6CSV

1 Like

Great! => This shows that the first pink box was too small. Often OCR results are better if the OCR engine has more text then just a tiny snippet. And/or maybe even the “Summer” text was partially cut off (looking at the _lastscreenshot image could be used to check on this, but it is not important now.)

Yes, that will be much easier and a larger OCR area is usually also better/more stable.

How to parse the “summer” text? => Use “Split” function or check for a substring. See here:

1 Like

Split worked… had to add !Timeout_Wait extension… now all is working thanks to you!

Going to work on clicking on the dropdown box now.