Trying to extract a link

Ui.Vision
, ,
2

I confirmed that the simple storeAttribute @href link web scraping does not work on this website. I assume this is because Connectedpapers.com is a highly complex website with plenty of Javascript and the link is built on the fly at mouse over.

But what always works is to automate the browser itself:

  • Simulate right-click on the “DOI” icon with XClick | icon.png | #right
  • Select “Copy link address” with XType => URL is now in clipboard
  • In the macro, the value is now inside the ${!clipboard} variable

Test run:

{
  "Name": "get-url",
  "CreationDate": "2021-4-13",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.connectedpapers.com/",
      "Value": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": ""
    },
    {
      "Command": "type",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": "test"
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/button/div",
      "Value": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/article/div/div/span",
      "Value": ""
    },
    {
      "Command": "comment",
      "Target": "click // css=#open-in-container > div > div > a.search-result-icon.doi-icon > svg > rect:nth-child(2)",
      "Value": ""
    },
    {
      "Command": "storeAttribute",
      "Target": "css=#open-in-container > div > div > a.search-result-icon.doi-icon > svg > rect:nth-child(1)@href",
      "Value": "url"
    },
    {
      "Command": "echo",
      "Target": "URL is ${url}",
      "Value": "blue"
    },
    {
      "Command": "XClick",
      "Target": "doi_dpi_168.png",
      "Value": "#right"
    },
    {
      "Command": "XType",
      "Target": "${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}",
      "Value": ""
    },
    {
      "Command": "XType",
      "Target": "${KEY_ENTER}",
      "Value": ""
    },
    {
      "Command": "echo",
      "Target": "URL is ${!clipboard}",
      "Value": "green"
    }
  ]
}