How to work with a chrome extension that produces html popup messages

I’m using a chrome extension for web security which presents html popup messages which is generated locally. How can I search/scan these popup messages for content. In the example below I open and get the following popup message. I’ve tried several approaches but always get error messages since it doesn’t open the actual url that was entered. Current log showing errors is at the bottom. Note: I had to change the url’s in the log to get around the new user restriction of two links in a post.

  • [status]

Playing macro Cyberlandia - Secure

  • [info]

Executing: | store | 5 | !TIMEOUT_PAGELOAD |

  • [info]

Executing: | store | true | !errorignore |

  • [info]

Executing: | bringBrowserToForeground | | |

  • [info]

Executing: | open | https://google_com | |

  • [info]

Executing: | sourceSearch | Google Search | text |

  • [info]

Executing: | open | https://google_ir | |

  • [error][ignored]

:grinning:: Error #230: Page load 5 seconds time out

  • [info]

Executing: | onError | #goto | TakeScreenshot |

  • [info]

Executing: | label | TakeScreenshot | |

  • [info]

Executing: | captureEntirePageScreenshot | googleir.png | |

  • [error][ignored]

:grinning:: ipcPromise: onAsk timeout 3000 for cmd “START_CAPTURE_FULL_SCREENSHOT”, args {}

  • [info]

Executing: | echo | Checking for Block Page | |

  • [echo]

Checking for Block Page

  • [info]

Macro completed (Runtime 14.45s)

If this dialog is from a web extension (not a website) you need to use desktop automation to automate a browser extension.

But I am a bit unsure what the goal here is. Do you want to extract/scrape the text from the “Web page blocked” dialog?

Yes, it is a popup from a web extension that is intercepting my request to go to the original url. I want to scan the popup for ‘Web Page Blocked’ for this particular url. For other url’s the page might say something different. This is the beginnings of a Policy Test script to ensure security policies are working.

I had everything working in iMacros but had to sit and watch it for the desired results. Same here, but eventually I would like to have this script run on several machines and gather the results for display on an Ops dashboard (Selenium?).

If the goal is to extract text from the popup (or any other user interface) you can use the commands

  • OCRExtractRelative + Area defined by IMAGE
  • OCRExtractRelativeTEXT + Area defined by TEXT

Both will work here. In my demo I use the OCRExtractRelativeTEXT screen scraping command. This is a very robust method for screen scraping.


Note that during test and macro run I move the IDE to the left (outside desktop screen) so that the OCR does not pick up the word “RISK” in the IDE. I want it to find the work RISK inside the popup screenshot. In production, you can avoid that by minimizing the IDE with BringIDEandBrowserToBackground

OCRExtractRelativeText explained:


The values Risk#R-8,25H10W35 mean to search for the word “Risk” and then, from the center of this word, go 8 to the left and 25 up. At this position take a screenshot of a rectangle area with height=10 and width=35. Then Ui.Vision runs local OCR on this image, and stores the result in variable a.

How to find these values? => Start with an educated guess, and press “FIND” to test it. Based on the result, refine the values. Rinse and repeat until you have moved the green screen scraping rectangle to the right place :wink:

Sorry, very new to RPA coming from iMacros.

So I switched to Desktop Automation and got the search box perfectly defined with Find as shown in your video.

However, now the script errors on all the open url commands? When in Desktop mode can you not open url’s? Also, if I have the extension Block Page displayed in the browser (like in my screenshot) the echo only shows [echo] text=$(a) and not Web Page Blocked.

I changed the parenthesis to curly brackets and now get [echo] text=

Welcome :smiley:

OPEN works fine in desktop mode.

Note that the OPEN command (and any other web browser only command) is not displayed in the command dropdown in “Desktop mode”, but the commands itself work fine.

That could mean that the OCR extraction returned “” (empty string). To debug this, have a look at the last_screenshot image in the “Screenshots” tab. It shows the input that is OCR processed. Is it the right area?

Even though the Find is good this is what the last_screenshot image shows. Looks like it’s below the target area where gray meets white.

I reran the Find and it looks perfect like in the screenshot above.


Are you hiding/covering the Ui.Vision IDE during macro run? Otherwise it could be the the OCR word is found inside the IDE instead of on the desktop. The last_screenshot image hints that this is happening…

I saw that in my screenshot when you pointed it out. I ran it again making sure the IDE was further over to the left so that Risk wasn’t showing and it did the same thing.

As a test, when you use the “FIND” button, how does the __last_desktop_screenshot.png image look then?

Note that I say last_desktop_screenshot.png, that is the image to look at, since we are using desktop automation. I mistyped that above.

It looks okay to me :laughing:


If I adjust the extract area as seen in the screenshot below it works. Not understanding why this is happening.