Trying to extract a link

Ui.Vision
, ,
1

I am trying to extract a link from https://www.connectedpapers.com. Ive tried to use storeAttribute but it seems I cant get my head on how to operate it. I also tried store text with the xpath but I cant get it right.

So far I have

{
  "Name": "copy",
  "CreationDate": "2021-4-12",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.connectedpapers.com/",
      "Value": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": "",
      "Targets": [
        "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
        "xpath=//div[2]/div/div/div/div/div/div/input",
        "css=#desktop-app > div.basic-page.main-router > div.main-container.landing-page-container > div.landing_search_bar > div > div > div.flexrow.autocomplete-searchbar-round-cont > div > input"
      ]
    },
    {
      "Command": "type",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": "water",
      "Targets": [
        "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
        "xpath=//div[2]/div/div/div/div/div/div/input",
        "css=#desktop-app > div.basic-page.main-router > div.main-container.landing-page-container > div.landing_search_bar > div > div > div.flexrow.autocomplete-searchbar-round-cont > div > input"
      ]
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/button/div",
      "Value": "",
      "Targets": [
        "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/button/div",
        "xpath=//div[2]/div/div/div/div/div/button/div",
        "css=#desktop-app > div.basic-page.main-router > div.main-container.landing-page-container > div.landing_search_bar > div > div > div.flexrow.autocomplete-searchbar-round-cont > button > div"
      ]
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/article/div/div/div/div",
      "Value": "",
      "Targets": [
        "xpath=//*[@id=\"desktop-app\"]/div[2]/div/article/div/div/div/div",
        "xpath=//article/div/div/div/div",
        "css=#desktop-app > div.basic-page.main-router > div > article:nth-child(2) > div > div > div.meta-data > div.search-result-authors"
      ]
    },
    {
      "Command": "click",
      "Target": "id=circle_60fa3053f2db749a11a55755348293991c84a1a2",
      "Value": "",
      "Targets": [
        "id=circle_60fa3053f2db749a11a55755348293991c84a1a2",
        "xpath=//*[@id=\"circle_60fa3053f2db749a11a55755348293991c84a1a2\"]",
        "css=#circle_60fa3053f2db749a11a55755348293991c84a1a2"
      ]
    },
    {
      "Command": "storeText",
      "Target": "xpath=//*[@id="open-in-container"]/div/div/a[2]]/",
      "Value": "LINK"
    }
  ]
}

I know the problem is in the last step, Iver tried Xpath, full Xpath, and storeAttribute but I am sure I am doing something wrong.

Edit:
The link I am trying to get is here:

image
image1015×538 100 KB

2

I confirmed that the simple storeAttribute @href link web scraping does not work on this website. I assume this is because Connectedpapers.com is a highly complex website with plenty of Javascript and the link is built on the fly at mouse over.

But what always works is to automate the browser itself:

  • Simulate right-click on the “DOI” icon with XClick | icon.png | #right
  • Select “Copy link address” with XType => URL is now in clipboard
  • In the macro, the value is now inside the ${!clipboard} variable

Test run:

{
  "Name": "get-url",
  "CreationDate": "2021-4-13",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://www.connectedpapers.com/",
      "Value": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": ""
    },
    {
      "Command": "type",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/div/input",
      "Value": "test"
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/div/div/div/div/button/div",
      "Value": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"desktop-app\"]/div[2]/div/article/div/div/span",
      "Value": ""
    },
    {
      "Command": "comment",
      "Target": "click // css=#open-in-container > div > div > a.search-result-icon.doi-icon > svg > rect:nth-child(2)",
      "Value": ""
    },
    {
      "Command": "storeAttribute",
      "Target": "css=#open-in-container > div > div > a.search-result-icon.doi-icon > svg > rect:nth-child(1)@href",
      "Value": "url"
    },
    {
      "Command": "echo",
      "Target": "URL is ${url}",
      "Value": "blue"
    },
    {
      "Command": "XClick",
      "Target": "doi_dpi_168.png",
      "Value": "#right"
    },
    {
      "Command": "XType",
      "Target": "${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}${KEY_DOWN}",
      "Value": ""
    },
    {
      "Command": "XType",
      "Target": "${KEY_ENTER}",
      "Value": ""
    },
    {
      "Command": "echo",
      "Target": "URL is ${!clipboard}",
      "Value": "green"
    }
  ]
}