How to web scrape search results

For RPA web scraping results from a list, you need to loop over it. Here is a small example for scraping URLs from a result. Note that I replaced a part of the XPath with a variable and increase it in each loop:

To find out what part of your XPath need to be replaced, just record the click of some H3 titles and then compare the recorded XPaths.

My macro code:

{
  "Name": "scrape",
  "CreationDate": "2021-2-21",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://duckduckgo.com/?q=solar&t=h_&va=d&ia=web",
      "Value": ""
    },
    {
      "Command": "times",
      "Target": "5",
      "Value": ""
    },
    {
      "Command": "storeText",
      "Target": "xpath=//*[@id=\"r1-${!times}\"]/div/div/div/a/span",
      "Value": "url"
    },
    {
      "Command": "echo",
      "Target": "Result ${!times}: ${url}",
      "Value": ""
    },
    {
      "Command": "end",
      "Target": "",
      "Value": ""
    }
  ]
}
1 Like