Here is the converted macro. Essentially we
- replaced all TAG… EXTRACT=TXT with storeText,
- and TAG…EXTRACT=HREF with storeAttributte | ID=…@href, just as mentioned on our iMacros conversion page and web scraping page.
- In the storeText/storeAttributte xpaths we replace the element numbers with a variable, so instead of _ctrl0, _ctrl1,… we use
_ctrl${i}. This way we can loop over it by increasing the value if “i” each loop. - We used Do…RepeatIf to replace the iMacros “Loop” button use.
In the customer’s words:
That worked perfectly! I am now back to where I was with the Imacros!!! But better, especially in terms of speed. Much more elegant macro.
I have used that Imacro system for 15 to 20 years! This is an improvement.
Ui.Vision Web Scraping Video:
Macro source code:
{
"Name": "iMacros2UI.Vision",
"CreationDate": "2024-2-15",
"Commands": [
{
"Command": "store",
"Target": "0",
"Value": "!timeout_wait",
"Description": "Fast scraping: Just skip if element is missing. No need to wait for it to appear. Page is already fully loaded."
},
{
"Command": "store",
"Target": "fast",
"Value": "!replayspeed",
"Description": ""
},
{
"Command": "store",
"Target": "1",
"Value": "page",
"Description": "Page counter"
},
{
"Command": "do",
"Target": "",
"Value": "",
"Description": ""
},
{
"Command": "store",
"Target": "0",
"Value": "i",
"Description": "data set counter 0....99"
},
{
"Command": "do",
"Target": "",
"Value": "",
"Description": ""
},
{
"Command": "storeAttribute",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_HyperLink1@href",
"Value": "!csvline",
"Description": "instrument"
},
{
"Command": "storeText",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_HyperLink1",
"Value": "!csvline",
"Description": "link"
},
{
"Command": "storeText",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_LinkAuthor",
"Value": "!csvline",
"Description": "author"
},
{
"Command": "storeText",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_LabelKWordAndSTitle",
"Value": "!csvline",
"Description": "main text"
},
{
"Command": "click",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_LabelKWordAndSTitle",
"Value": "",
"Description": "This CLICK is NOT needed. We use it just for the video to scroll the page and highlight the scraped content. Remove for production use. Then the scraping runs faster."
},
{
"Command": "storeText",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_LabelStatus",
"Value": "!csvline",
"Description": "status"
},
{
"Command": "storeText",
"Target": "xpath=//*[@id=\"ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl${i}_LabelConsidered\"]/a",
"Value": "!csvline",
"Description": ""
},
{
"Command": "echo",
"Target": "${page}|${i}|${!csvline}",
"Value": "blue",
"Description": "Show scraped data in log"
},
{
"Command": "csvSave",
"Target": "result1.csv",
"Value": "",
"Description": "Save scraped data to CSV"
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number (${i}) + 1;",
"Value": "i",
"Description": "Add 1 to i"
},
{
"Command": "repeatIf",
"Target": "${i} < 100",
"Value": "",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number (${page}) + 1;",
"Value": "page",
"Description": "Add 1 to page"
},
{
"Command": "executeScript",
"Target": "__doPostBack('ctl00$ctl00$PageBody$PageContent$DataPager1$ctl02$ctl00','')",
"Value": "",
"Description": "NEXT button is tricky. Use XClick or simply this JS code"
},
{
"Command": "echo",
"Target": "Starting page ${page}",
"Value": "green",
"Description": ""
},
{
"Command": "pause",
"Target": "10000",
"Value": "",
"Description": "Wait for new page to load"
},
{
"Command": "store",
"Target": "true",
"Value": "!statusOK",
"Description": "Reset !status OK"
},
{
"Command": "storeText",
"Target": "id=ctl00_ctl00_PageBody_PageContent_ListViewSearchResults_ctrl0_HyperLink1",
"Value": "test",
"Description": "TEST web scraping to check if a new page of data is loaded. If not, we are done!"
},
{
"Command": "repeatIf",
"Target": "${test} != \"#LNF\"",
"Value": "",
"Description": "Make sure we are on a page with results. If no anchor is found, storeText returns #LNF"
},
{
"Command": "echo",
"Target": "All done!",
"Value": "green",
"Description": ""
}
]
}