Is there a limit to the number of lines UiVision can extract and save to a file?
I’m running an automation that extracts many lines (100 thousand or more), but there comes a time when the Macro breaks, crashes and no longer works. I believe this could be the problem.
extract less, like 70 thousand then jump to another macro and extract another 70 thousand
I tried again and when it reaches the line approximately at line 1,500 the extension breaks and stops running.
If you need a macro to run a very long time (hours, days or even weeks) I recommend that you start it from the command line. This way you have full control over the browser and ui.vision.
If that happens, I recommend to loop the extracting in your calling script. So you would write a macro that e. g. extracts 1000 lines at once, and then stops.
Now your calling script closes the browser & ui.vision and then starts both fresh for the next 1000 lines. Do this 100 times and your web scraping is completed.
You find sample code for Node.JS, Python, Powershell etc here: RPA/command-line at master · A9T9/RPA · GitHub
I understand, but it is not a viable option. I believe the reason the macro stops is due to browser memory. I switched to the system save option, it even saves in the folder, but the macro still breaks. And it’s not even that big, just the repetition cycle is very big, like I said, about 25 to 100 thousand lines.
{
"Name": "064 - VAGAS POR POLO - V 10.0",
"CreationDate": "2024-11-7",
"Commands": [
{
"Command": "store",
"Target": "medium",
"Value": "!REPLAYSPEED",
"Description": ""
},
{
"Command": "store",
"Target": "true",
"Value": "!errorignore",
"Description": "Ignore errors and continue execution"
},
{
"Command": "store",
"Target": "180",
"Value": "!TIMEOUT_WAIT",
"Description": ""
},
{
"Command": "store",
"Target": "1",
"Value": "SOMA",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id='listar-ies-cadastro']/tfoot/tr/td/div/div[1]",
"Value": "TEXT_CONTEUDO",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return ${TEXT_CONTEUDO}.match(/\\d+$/)[0];",
"Value": "TOTAL_CURSO",
"Description": ""
},
{
"Command": "label",
"Target": "checkElement_1",
"Value": "",
"Description": ""
},
{
"Command": "store",
"Target": "true",
"Value": "!errorignore",
"Description": "Ignore errors and continue execution"
},
{
"Command": "store",
"Target": "180",
"Value": "!TIMEOUT_WAIT",
"Description": ""
},
{
"Command": "store",
"Target": "medium",
"Value": "!REPLAYSPEED",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"listar-ies-cadastro\"]/tbody[${SOMA}]/tr/td[1]",
"Value": "COD_CURSO",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number(${SOMA}) + 1;",
"Value": "COD_NUM_CURSO",
"Description": ""
},
{
"Command": "verifyElementPresent",
"Target": "id=${COD_CURSO}",
"Value": "",
"Description": ""
},
{
"Command": "if_v2",
"Target": "${!LastCommandOK} == true",
"Value": "",
"Description": ""
},
{
"Command": "click",
"Target": "id=${COD_CURSO}",
"Value": "",
"Targets": [
"id=${COD_CURSO}"
],
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id='lista-polos']/table/tfoot/tr/td/div/div[1]",
"Value": "textoCompleto",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return ${textoCompleto}.match(/\\d+$/)[0];",
"Value": "TOTAL_POLO",
"Description": ""
},
{
"Command": "waitForElementVisible",
"Target": "//*[@id=\"paginationItemCountItemlista-polos\"]",
"Value": "",
"Description": ""
},
{
"Command": "select",
"Target": "//*[@id=\"paginationItemCountItemlista-polos\"]",
"Value": "label=1000",
"Description": ""
},
{
"Command": "executeScript",
"Target": "window.scrollTo(0, 0);",
"Value": "",
"Description": ""
},
{
"Command": "waitForElementNotPresent",
"Target": "//*[@id='paginationItemCountItemlista-polos']",
"Value": "",
"Description": ""
},
{
"Command": "store",
"Target": "0",
"Value": "CONTADOR",
"Description": ""
},
{
"Command": "while",
"Target": "true",
"Value": "",
"Description": ""
},
{
"Command": "store",
"Target": "true",
"Value": "!errorignore",
"Description": "Ignore errors and continue execution"
},
{
"Command": "store",
"Target": "5",
"Value": "!TIMEOUT_WAIT",
"Description": ""
},
{
"Command": "store",
"Target": "fast",
"Value": "!REPLAYSPEED",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id='divTitle']/font[2]",
"Value": "FULL_TEXT",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return ${FULL_TEXT}.match(/\\((.*?)\\)/)[1];",
"Value": "IES",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"dd_curso\"]/tbody/tr[2]/td",
"Value": "NOME_CURSO",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"dd_curso\"]/tbody/tr[3]/td",
"Value": "STATUS_CURSO",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"dd_curso\"]/tbody/tr[4]/td",
"Value": "VAGAS_AUTORIZADAS",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"dd_curso\"]/tbody/tr[5]/td",
"Value": "VAGAS_DISTRIBUIDAS",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"lista-polos\"]/table/tbody/tr[${SOMA}]/td[1]",
"Value": "COD_POLO",
"Description": ""
},
{
"Command": "storeText",
"Target": "//*[@id=\"lista-polos\"]/table/tbody/tr[${SOMA}]/td[2]/strong",
"Value": "NOME_POLO",
"Description": ""
},
{
"Command": "storeAttribute",
"Target": "xpath=//*[@id='dt_funcionamento_${COD_POLO}']@value",
"Value": "DATA_FUNCIONAMENTO",
"Description": ""
},
{
"Command": "storeAttribute",
"Target": "xpath=//*[@id='nu_vagas_${COD_POLO}']@value",
"Value": "NUMERO_VAGAS",
"Description": ""
},
{
"Command": "store",
"Target": "IES",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${IES}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "COD_CURSO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${COD_CURSO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "NOME_CURSO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${NOME_CURSO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "STATUS_CURSO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${STATUS_CURSO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "VAGAS_AUTORIZADAS",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${VAGAS_AUTORIZADAS}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "VAGAS_DISTRIBUIDAS",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${VAGAS_DISTRIBUIDAS}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "COD_POLO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${COD_POLO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "NOME_POLO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${NOME_POLO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "DATA_FUNCIONAMENTO",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${DATA_FUNCIONAMENTO}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "NUMERO_VAGAS",
"Value": "!csvline",
"Description": ""
},
{
"Command": "store",
"Target": "${NUMERO_VAGAS}",
"Value": "!csvline",
"Description": ""
},
{
"Command": "csvSave",
"Target": "064 - IES ${IES} - CURSO - ${NOME_CURSO}.csv",
"Value": "",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number(${CONTADOR}) + 1;",
"Value": "CONTADOR",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return parseInt('${COD_NUM_CURSO}') - 1;",
"Value": "RESULTADO_SUBTRAIDO",
"Description": ""
},
{
"Command": "echo",
"Target": "CURSO ${RESULTADO_SUBTRAIDO} DE ${TOTAL_CURSO} | POLO: ${COD_POLO} - ${CONTADOR} DE ${TOTAL_POLO} ",
"Value": "",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number(${SOMA}) + 1;",
"Value": "SOMA",
"Description": ""
},
{
"Command": "verifyElementPresent",
"Target": "//*[@id='lista-polos']/table/tbody/tr[${SOMA}]/td[1]",
"Value": "",
"Description": ""
},
{
"Command": "if_v2",
"Target": "${!LastCommandOK} == false",
"Value": "",
"Description": ""
},
{
"Command": "store",
"Target": "slow",
"Value": "!REPLAYSPEED",
"Description": ""
},
{
"Command": "executeScript_Sandbox",
"Target": "return Number(${COD_NUM_CURSO});",
"Value": "SOMA",
"Description": ""
},
{
"Command": "gotoLabel",
"Target": "checkElement_1",
"Value": "",
"Description": ""
},
{
"Command": "end",
"Target": "",
"Value": "",
"Description": ""
},
{
"Command": "end",
"Target": "",
"Value": "",
"Description": ""
},
{
"Command": "end",
"Target": "",
"Value": "",
"Description": ""
},
{
"Command": "exit",
"Target": "",
"Value": "",
"Description": ""
}
]
}
Why not? Regular restarting of the browser will clear all memory issues.
The macro takes information from a Brazilian government website that requires an authenticator to enter. I log in manually. Then it takes the basic information from the screen itself to proceed. If you restart the browser, it will always return to the first point. therefore it is not viable.
Oh, I see. In this case I recommend to close the Ui.Vision extension only (and not the browser). You can do this by adding &closeRPA=1&closeBrowser=0 to the command line. This will close the IDE once the macro has finished, but not the browser. At the next call of the command line, a fresh new Ui.Vision instance is opened - but it uses the same browser that is already opened and logged in.