How to automate a custom select box visually with OCR

Ui.Vision HowTo
, ,
1

Standard HTML select boxes can be automated with the select command. But some select boxes only look like a select box, technically they are custom elements. Here the selenium IDE select command fails. Often then at least keyboard shortcuts (XType) work. But the custom select box in this does not accept any keyboard entry. So we are limited mouse control.

We use a visual automation approach:

  • Restrict the computer vision area to the select box area
  • Then use XCLICK | OCR= to search the select box area for the keyword
  • If not found: Click in the “Scroll down” area of the scroll bar to check the next entries
  • Repeat this until the OCR result includes the search string
  • If the string is found XCLICK | OCR= automatically clicks it.

Input image for visionLimitSearchAreaRelative:

image

Macro running:

In the video below we use cloud OCR. With the optional local OCR engine it will be faster.

{
  "Name": "Visual Select Box",
  "CreationDate": "2021-2-17",
  "Commands": [
    {
      "Command": "XClick",
      "Target": "changelink_dpi_144.png",
      "Value": ""
    },
    {
      "Command": "store",
      "Target": "2",
      "Value": "!ocrengine"
    },
    {
      "Command": "store",
      "Target": "Columbia",
      "Value": "entry_to_click"
    },
    {
      "Command": "visionLimitSearchAreaRelative",
      "Target": "lx_limitarea_dpi_144.png",
      "Value": ""
    },
    {
      "Command": "store",
      "Target": "true",
      "Value": "!errorignore"
    },
    {
      "Command": "do",
      "Target": "",
      "Value": ""
    },
    {
      "Command": "store",
      "Target": "true",
      "Value": "!statusOK"
    },
    {
      "Command": "XClick",
      "Target": "ocr=${entry_to_click}",
      "Value": ""
    },
    {
      "Command": "XClick",
      "Target": "lx_scrollbox_dpi_144.png",
      "Value": ""
    },
    {
      "Command": "repeatIf",
      "Target": "${!statusOK} == false",
      "Value": ""
    },
    {
      "Command": "store",
      "Target": "false",
      "Value": "!errorignore"
    },
    {
      "Command": "echo",
      "Target": "Entry found & clicked!",
      "Value": "green"
    },
    {
      "Command": "XDesktopAutomation",
      "Target": "false",
      "Value": ""
    },
    {
      "Command": "XClick",
      "Target": "xpath=//input",
      "Value": ""
    },
    {
      "Command": "type",
      "Target": "xpath=//input",
      "Value": "Selection OK!"
    }
  ]
}