How to extract a value within iframes? some advice

Ui.Vision General Discussion
1

Hi all, please some advice.

I’m trying to extract the value of an element that is inside an iframe. When I record a macro to do the steps, it records selectFrame | index=1 twice and sometime I recorded the macro appeared selectFrame | index=0 followed by selectFrame | index=1, but when I run both macro versions, the command storeText | xpath=//some_element gives the error

timeout reached when looking for element 'xpath=//some_element'

The website it looks exactly the same as this URL https://www.wayfair.com/ when you do click on Cart icon and you see that appears an iframe from the right. The thing with this other URL is that when I record the macro the selectFrame commands are not recorded even an iframe appears and I cannot share the actual URL since it needs a login first. Only what I now is that uses nextjs technology.

The other thing is I don’t use OCRExtractRelative since the macro when it runs in another machine the image taken doesn´t work. So, I don’t know which best portable option between machines could I use.

Thanks for any help.

2

Try OCRExtractByTextRelative - this version uses text (OCR) to mark the area of interest to read the text from. Unlike images, OCR is much more stable between platforms, even between Mac, Linux and Windows.

web scraping with OCR

:point_up_2: I used desktop mode to take this image. In web mode, the frames prevent the Ui.Vision from drawing the green and pink box overlay on the website. But that is just an overlay display issue. The command itself works just fine, since it works on screenshots, not HTML source.

Macro running:

{
  "Name": "cart1_local",
  "CreationDate": "2024-11-21",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.wayfair.com/",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "pause",
      "Target": "3000",
      "Value": "",
      "Description": "We need to wait for page to load completely, otherwise cart does not work on click"
    },
    {
      "Command": "store",
      "Target": "99",
      "Value": "!ocrengine",
      "Description": "xmodule engine"
    },
    {
      "Command": "XClickText",
      "Target": "*Cart*",
      "Value": "",
      "Description": "open shopping cart"
    },
    {
      "Command": "OCRExtractbyTextRelative",
      "Target": "Cart#R-30,-5H70W50",
      "Value": "s",
      "Description": "This marks the area below the word \"Cart\" and OCR it"
    },
    {
      "Command": "echo",
      "Target": "Scraped Text=${s}",
      "Value": "green",
      "Description": ""
    }
  ]
}
3

Bonus answer: I prefer the local solution above. But just to mention it. Instead of OCRExtractbyTextRelative you can also scrape the shopping cart content with the new aiPrompt command.

This solution is super easy, but running it often will cost you a few cents in Anthropic AI credits.

My prompt was: cart.png#attached a screenshot. Extract the text from inside the shopping cart area

Macro running:

{
  "Name": "cart1_ai",
  "CreationDate": "2024-11-21",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.wayfair.com/",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "pause",
      "Target": "3000",
      "Value": "",
      "Description": "We need to wait for page to load completely, otherwise cart does not work on click"
    },
    {
      "Command": "store",
      "Target": "99",
      "Value": "!ocrengine",
      "Description": "xmodule engine"
    },
    {
      "Command": "XClickText",
      "Target": "*Cart*",
      "Value": "",
      "Description": "open shopping cart"
    },
    {
      "Command": "pause",
      "Target": "2000",
      "Value": "",
      "Description": "Wait for cart to open (or use VisualAssert)"
    },
    {
      "Command": "captureScreenshot",
      "Target": "cart.png",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "aiPrompt",
      "Target": "cart.png#attached a screenshot. Extract  the text from inside the shopping cart area",
      "Value": "s",
      "Description": ""
    },
    {
      "Command": "echo",
      "Target": "AI Scraped Text=${s}",
      "Value": "green",
      "Description": ""
    }
  ]
}
4

PS: There might be also a solution that works on the HTML only using XPath and storeText. I did not look into this since the approach is specific for every website, and the wayfair website is just an example.

5

Hi @ulrich Thanks so much for your 3 answers. I didn´t know about the new ai commands, I’ll update the UI to test them. After several test and errors I was able to make it work the OCRExtractByTextRelative but the word (Wallet) I like to use as anchor is not found, I think due its size is smaller than others since when I use as anchor the word Payment is found. The issue is there are 4 occurences of Payment in the screen and I don´t know how to select the first one (the one at the top). My goal is to get the Wallet balance (in this case $0.00)

If I use this command below using as anchor Wallet, I get this error

OCRExtractbyTextRelative | Wallet#R-5,10H5W10 | wt_balance
[error] Line 1: Cannot read properties of null (reading '1')

This is the print of the area I’m working with

image
image549×962 30 KB

and this is how it looks when I click on “find” with this command

OCRExtractbyTextRelative | Payment#R-22,-18H5W10 | wt_balance

image
image567×596 41.8 KB

6

The box overlay looks good/correct. Since the first match is the one you need, there is nothing else to do. It picks the first match by default. If you would like the use e. g. the second match, add @pos=2. Example:

OCRExtractbyTextRelative | Wallet@pos=2#R-5,10H5W10

As a side note, I would make the green box a bit bigger. It is my experience that more white space around a text/number improves the OCR results.

[error] Line 1: Cannot read properties of null (reading ‘1’)

Hmm… do you always get it? It sounds like a problem with the OCR engine. Can you select another one just to see if this error goes away?

7

Hi @ulrich yes. I always get the same error when I try to search “Wallet” having selected JavaScript OCR engine or the other OCR engine. The only thing I can think is that is not found because the font size is smaller than Payment word. The weird thing is I tested in the other website(Wayfair) that has a similar font size and the word was found.

8

Another test idea: Instead of
OCRExtractbyTextRelative | Wallet@pos=2#R-5,10H5W10
try something simpler, like
XMoveText | Wallet

Do you get the same error?

9

XMoveText | Wallet it works

It works this command too

XClickTextRelative | Wallet#R0,-4

image

10

Interesting result. → Just to confirm:

OCRExtractbyTextRelative | Wallet#R0,-4H3W3 | s

gives you an error? I used the same 0,-4 as in the working XClickTextRelative as a test.

11

Hi @ulrich, the error is the same using the command you suggested

OCRExtractbyTextRelative | Wallet#R0,-4H3W3 | s