HTML table extraction slow and times out

Ui.Vision
1

Hi,
I fill a form and on submission I get a HTML web table that has 20 columns and 2877 rows. If I would have increased the date range in the form then this table would have a lot more rows. My problems is that extracting cell by cell (!replayspeed is FAST) results in a timeout after 900s.
[error] macro timeout 900s (change the value in the settings if needed)
My issue is, if it takes 15 minutes only for 70 rows then I will never be able to use this method to get 2877 rows! Can the team help me use a faster and reliable method to extract the entire table in hopefully under 5 minutes.
I am attaching the code for your reference.

{
  "CreationDate": "2018-11-22",
  "Commands": [
    {
      "Command": "store",
      "Target": "fast",
      "Value": "!replayspeed"
    },
    {
      "Command": "store",
      "Target": "180",
      "Value": "!TIMEOUT_WAIT"
    },
    {
      "Command": "open",
      "Target": "https://www.indiabondinfo.nsdl.com/bds-web/advancedSearchRequest.do",
      "Value": ""
    },
    {
      "Command": "XClick",
      "Target": "//*[@id=\"typeOfInstrument\"]",
      "Value": ""
    },
    {
      "Command": "XType",
      "Target": "${KEY_CTRL+KEY_A}",
      "Value": ""
    },
    {
      "Command": "XClick",
      "Target": "//*[@id=\"instrumentStatus\"]",
      "Value": ""
    },
    {
      "Command": "XType",
      "Target": "${KEY_CTRL+KEY_A}",
      "Value": ""
    },
    {
      "Command": "type",
      "Target": "id=allotmentDateFrom",
      "Value": "15/11/2017"
    },
    {
      "Command": "type",
      "Target": "id=allotmentDateTo",
      "Value": "15/11/2018"
    },
    {
      "Command": "click",
      "Target": "id=searchButtonID",
      "Value": ""
    },
    {
      "Command": "store",
      "Target": "1",
      "Value": "colcntr"
    },
    {
      "Command": "store",
      "Target": "1",
      "Value": "rowcounter"
    },
    {
      "Command": "label",
      "Target": "STARTLOOP",
      "Value": ""
    },
    {
      "Command": "verifyElementPresent",
      "Target": "//*[@id=\"rows\"]/tbody//child::tr[${rowcounter}]",
      "Value": ""
    },
    {
      "Command": "if",
      "Target": "${!LastCommandOK}",
      "Value": ""
    },
    {
      "Command": "echo",
      "Target": "Row present at counter ${rowcounter}",
      "Value": "green"
    },
    {
      "Command": "store",
      "Target": "1",
      "Value": "colcntr"
    },
    {
      "Command": "while",
      "Target": "( ${colcntr} <= 20 )",
      "Value": ""
    },
    {
      "Command": "storeText",
      "Target": "//*[@id=\"rows\"]/tbody//child::tr[${rowcounter}]/td[${colcntr}]",
      "Value": "!csvLine"
    },
    {
      "Command": "storeEval",
      "Target": "${colcntr} + 1",
      "Value": "colcntr"
    },
    {
      "Command": "endWhile",
      "Target": "",
      "Value": ""
    },
    {
      "Command": "csvSave",
      "Target": "NSDL_Bond_Debentures.csv",
      "Value": ""
    },
    {
      "Command": "storeEval",
      "Target": "${rowcounter} + 1",
      "Value": "rowcounter"
    },
    {
      "Command": "gotoLabel",
      "Target": "STARTLOOP",
      "Value": ""
    },
    {
      "Command": "else",
      "Target": "",
      "Value": ""
    },
    {
      "Command": "echo",
      "Target": "Row absent at counter ${rowcounter}",
      "Value": "red"
    },
    {
      "Command": "gotoLabel",
      "Target": "ENDLOOP",
      "Value": ""
    },
    {
      "Command": "endif",
      "Target": "",
      "Value": ""
    },
    {
      "Command": "label",
      "Target": "ENDLOOP",
      "Value": ""
    },
    {
      "Command": "localStorageExport",
      "Target": "NSDL_Bond_Debentures.csv",
      "Value": "#DeleteAfterExport"
    }
  ]
}

Thanks & Regards,
Bharat

2

Can you send this table to us (team AT a9t9.com)? Maybe there is indeed a performance issue for huge tables.

Question: Do you need to extract the complete table or only certain cells?

3

Hi,

The website was down last night. I did not understand how will it be possible to send only the table across to the support team since the website is essentially a form that generates the table when the proper inputs are submitted. There are only 4 fields that need to be filled. Below are the details of how to generate the table.

Open 
https://www.indiabondinfo.nsdl.com/bds-web/advancedSearchRequest.do
1. Type of Instrument:   select both Bonds + Debentures
2. Instrument Status:    select both Active + Matured
3. Date of Allotment     From: 15/11/2017   To:   15/11/2018
4. Click [Search] button

The complete table of 20 columns x 2877+ rows needs to be extracted.

4

Hi,
Were you able to pin down the cause of the issue, as to why Kantu is unable to extract very large tables and times out. I have again faced the same issue with another (similar) usecase too.

5

Hi, sorry for my bad english,
you can get the entire table, this way

{
  "Command": "storeText",
  "Target": "//table[i]/tbody",
  "Value": "c2"
},

all the table is stored as text(tab delimited), when you import the csv you can view the formatting

6

Another interesting way to capture table data automatically is to automate a 3rd party Chrome extension like “Table Capture”.