[Solved]-Scraping data from Highcharts using selenium-Highcharts

I noticed one thing.

It seems true that:

"when I use custom search option, csv file that automatically generated by the website is not updated".

But actually it is not true. It is updated, but the maximum "custom data range" seems to be 1 year.

For example, if you set from 2013-09-23 to 2017-10-23 the .csv(.json) generated has max the data of 1 year (in this example from 22/10/2016 to 21/10/2017).

You can better notice this if you play with the "extremes".

For example with:

https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/downloads-day-20131023-20141023.json

• first element: {"date": "2014-10-23", "count": 212730, "end": "2014-10-23"}
• last element: {"date": "2013-10-24", "count": 163094, "end": "2013-10-24"}

if you change with:

https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/downloads-day-20131023-20141024.json

• first element: {"date": "2014-10-24", "count": 215105, "end": "2014-10-24"}
• last element: {"date": "2013-10-25", "count": 168018, "end": "2013-10-25"}

Or with:

https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/downloads-day-20131022-20141023.json

will be again :

• first element: {"date": "2014-10-23", "count": 212730, "end": "2014-10-23"}
• last element: {"date": "2013-10-24", "count": 163094, "end": "2013-10-24"}

So, in order to get the data of the last 5 years you could do:

import subprocess
interestedYears=5;
year=1
today="2017-10-23"
tokenDataToday= today.split("-")
dateEnd=tokenDataToday[0]+tokenDataToday[1]+tokenDataToday[2]
url= "https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/downloads-day-"

while year <= interestedYears:
     yearStart= str(int(float(tokenDataToday[0]))-year)
     dateStart=yearStart+tokenDataToday[1]+tokenDataToday[2]
     #print("dateStart: " + dateStart)
     #print("dateEnd: " + dateEnd)
     tmpUrl=url+dateStart+"-"+dateEnd+".csv"
     cmd = 'curl -O ' + tmpUrl
     print(cmd)
     args = cmd.split()
     process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     stdout, stderr = process.communicate()
     dateEnd=dateStart
     year = year+1
     print("-----------------------------")