Scraping a list: xpath element changes when scrolling

UI.Vision RPA
1

Hello,

I have the following scraping task for this webpage:

  1. click on the company’s name to access its “detail” page
  2. store the company’s website as a csv line
  3. go back to the “list” page of all companies
  4. repeat for all companies (approx. 15,000)

(Unfortunately, you will need to create a free account in order to have access to the full list of companies)

The following (very simple) script works properly for the first 4 companies and then breaks:

“Name”: “FrenchTech2”,
“CreationDate”: “2020-5-4”,
“Commands”: [
{
“Command”: “open”,
“Target”: “https://ecosystem.lafrenchtech.com/companies.startups/f/launch_year_min/anyof_2015/locations/allof_France?showGrid=true”,
“Value”: “”
},
{
“Command”: “click”,
“Target”: “xpath=//[@id="list-map-list"]/div/div/div[1]/div[${!loop}]/div/section/div/div[1]/a/h3",
“Value”: “”
},
{
“Command”: “storeText”,
“Target”: "xpath=//[@id="window-scrollbar"]/div[1]/main/div/div/section/div/div/div[2]/div[2]/div[3]/a”,
“Value”: “!csvLine”
},
{
“Command”: “csvSave”,
“Target”: “url6”,
“Value”: “”
}
]
}

This is due to the fact that the element “xpath=//*[@id="list-map-list"]/div/div/div[1]/div[${!loop}]/div/section/div/div[1]/a/h3” (which allows access to the detail page for each company) is unstable for each company: every time I scroll up or down, the number in the 4th “div” (which is the relevant variable - cf. !loop) changes.

Can someone help with this? I was thinking that perhaps using the XClick or XMove commands could help but I have not managed to make the macro work.

Thanks a lot for your help!

2

Try another xpath there are more xpaths for every web element

3

Thanks for replying - I have tried alternative xpaths but the issue is the same: for every “company card” all xpaths that lead to the company’s detail page have the same variability (meaning that the number in the relevant div changes when you scroll up or down on the website, and this is the case for all xpaths)

4

There are several types of xpath some do not examine the div and use other elements I regularly identify dynamic xpath without problems you have to look for a better xpath otherwise there is no solution, often create good automation can also take days or weeks of work and tests.

Exist the partial xpath or xpath that start or end with a part and more and more types of xpath

Look this image this is a minimal part of xpath available for my username in this forum

xpath

5

Thank you very much, this worked well.