How to web scrape search results

Ui.Vision HowTo
1

hi everyone
i want to scrape data from a web page:

On this web page, there are search results
this XPath //h3 get all search result

problem is:

ui.vision scrape the first result only [does not recognize all XPath element just the first one]

2

For RPA web scraping results from a list, you need to loop over it. Here is a small example for scraping URLs from a result. Note that I replaced a part of the XPath with a variable and increase it in each loop:

image
image675Ɨ807 100 KB

To find out what part of your XPath need to be replaced, just record the click of some H3 titles and then compare the recorded XPaths.

My macro code:

{
  "Name": "scrape",
  "CreationDate": "2021-2-21",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://duckduckgo.com/?q=solar&t=h_&va=d&ia=web",
      "Value": ""
    },
    {
      "Command": "times",
      "Target": "5",
      "Value": ""
    },
    {
      "Command": "storeText",
      "Target": "xpath=//*[@id=\"r1-${!times}\"]/div/div/div/a/span",
      "Value": "url"
    },
    {
      "Command": "echo",
      "Target": "Result ${!times}: ${url}",
      "Value": ""
    },
    {
      "Command": "end",
      "Target": "",
      "Value": ""
    }
  ]
}
1 Like
3

You Just Made My Day Thank You a Lot

4

What if the list is of variable length?

5

I can actually set the times variable to higher than the max I expect and then loop through. This works fine. I get #LNF when I run out of elements. I thought I could break out by search for #LNF but having problems how to reference. I store my values in EXP1, EXP2 etc using !times as the counter. Also works fine. But how do I chec if the variable named EXP!Times contains ā€œ#LNFā€? I store Exp${!Times} in a new variabel called newtimes. This works. So now I have a variable newtimes which contains the latest EXP. If #LNF was found in element 6 then newtimes contains the string EXP6 which is the name of the variable. But how do I reference this. I have tried if_v2 {{newtimes} == ā€œ#LNFā€ but no luck (Unepected token ā€˜{ā€™

6

I mean I can just let it run through all the elements in the list takes lots of time searching for element it cannot find.

7

I try to do something the same as this, but iā€™m getting the same result 5 times
how can i code it to go to the next target?

thanks