How to web scrape search results

hi everyone
i want to scrape data from a web page:

On this web page, there are search results
this XPath //h3 get all search result

problem is:

ui.vision scrape the first result only [does not recognize all XPath element just the first one]

For RPA web scraping results from a list, you need to loop over it. Here is a small example for scraping URLs from a result. Note that I replaced a part of the XPath with a variable and increase it in each loop:

To find out what part of your XPath need to be replaced, just record the click of some H3 titles and then compare the recorded XPaths.

My macro code:

{
  "Name": "scrape",
  "CreationDate": "2021-2-21",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://duckduckgo.com/?q=solar&t=h_&va=d&ia=web",
      "Value": ""
    },
    {
      "Command": "times",
      "Target": "5",
      "Value": ""
    },
    {
      "Command": "storeText",
      "Target": "xpath=//*[@id=\"r1-${!times}\"]/div/div/div/a/span",
      "Value": "url"
    },
    {
      "Command": "echo",
      "Target": "Result ${!times}: ${url}",
      "Value": ""
    },
    {
      "Command": "end",
      "Target": "",
      "Value": ""
    }
  ]
}
1 Like

You Just Made My Day Thank You a Lot

What if the list is of variable length?

I can actually set the times variable to higher than the max I expect and then loop through. This works fine. I get #LNF when I run out of elements. I thought I could break out by search for #LNF but having problems how to reference. I store my values in EXP1, EXP2 etc using !times as the counter. Also works fine. But how do I chec if the variable named EXP!Times contains ā€œ#LNFā€? I store Exp${!Times} in a new variabel called newtimes. This works. So now I have a variable newtimes which contains the latest EXP. If #LNF was found in element 6 then newtimes contains the string EXP6 which is the name of the variable. But how do I reference this. I have tried if_v2 {{newtimes} == ā€œ#LNFā€ but no luck (Unepected token ā€˜{ā€™

I mean I can just let it run through all the elements in the list takes lots of time searching for element it cannot find.

I try to do something the same as this, but iā€™m getting the same result 5 times
how can i code it to go to the next target?

thanks