Machine-to-Machine Access to the HelioClim3 Irradiation Data

A problem of units ?
of terminology ?
Consult the Education service.

Framework and Principle
Build a HC3 WGET command
Build a HC1 WGET command
Another option: CURL
FAQ

Framework and Principle

Who?
Several of our Customers offer a electricity production monitoring device and data logger, and use the HelioClim data as an alternative source of irradiation values in a purpose of a more objective assessment. The machine-to-machine access to the data with the wget command is adequate you have to proceed to a large number of repetitive requests. This command enables you to retrieve one time series per site.

Principle: The Customer creates an application on this machine that is capable of launching a series of "wget" commands, one per site, that are executed on his computer. It is made through the protocol HTTP. Download here the executable wget.exe version 1.14.4.
SoDa grants an access to a well-identified Customer's machine through his IP address: SoDa checks the IP of the Customer's machine, the subscribed services, as well as counts the daily requests of each Customer.

Which IP?
The Customers need to provide us their public IP address (proxy, firewall, or more generally the IP address of the machine which is connected to the web). This address can be obtained using http://whatismyipaddress.com/ in a browser. This IP must be fixed, which is generally the case with Companies.

Output Column Content (OCC): identical to the one of free services. You can see here the list of the OCC per service. NB: the output is textfile, which means that the separator is "(space)" instead of a ";" as in a csv file. Please click here to obtain a solution to transform automatically the space character into a semi-colon.

Cost:

All annual subscriptions includes an average of 200 wget automatic requests per day. This number is averaged over the month, i.e. you are allowed to do 300 requests one day, and 100 next day.
For more than 200 requests per day, an additional fee is applied, depending on the number of requests. To receive a quotation, contact the sales support.

NB: Please note that all annual subscriptions also include a wget access to the HC1 database.

Leadtime: All details about leadtime are available here. To summarize, the data of day d are available with a minimum of delay of 24 h, more precisely at 11:30 PM UT the next day. If you want to retrieve the HelioClim irradiation data as soon as they are available (i.e. at 11:30 PM UT at night), you need to subscribe our "ner real time option". To receive a quotation, contact the sales support.

Reliability: We remind you that since January 2011, the acquisition of the MSG images and the HelioClim databases first available at Mines ParisTech, Sophia-Antipolis, have been duplicated in Transvalor, Mougins. The system is now fully operational. If you face any difficulty in getting back your data using the wget command with "www.helioclim.org", you can easily access the data using our backup site in Transvalor with "pro.soda-is.com" in your wget and curl requests.

Click here to read the "General Terms and Conditions" of our contract.

Top of page

Build the HC3 wget command

Three examples of scenario:

Paris: you would like to have a HCday, i.e. retrieve daily values over the horizontal plane. You are interested by all the day during the month of May 2006, with the default digital terrain elevation (DTE) database and albedo values.
Madrid: HChourDNI, i.e. hourly irradiation values on a plane normal to sun rays, time period: the last 15 days of August 2010, default DTE and albedo values.
Berlin: HC15Incl, i.e. 15 minute irradiation values on a tilted plane. The plane has a tilt of 35° and is facing the Southern direction (180° in the SoDa conventions). Default DTE and albedo values. You have subscribed the "near real time option", so if we admit we are on December 31^th 2010 at 1:30 AM UT, you are allowed to retrieved the data of the previous day, on December 30^th 2010.

Step Action

Listing of the input parameters:
cmd="http://www.helioclim.org/pub/hc3v2_request.php?"

`cmd=$cmd"geopoint=40,0"`	# in this example, latitude=40 longitude=0
`cmd=$cmd"&elevation=-999"`	# elevation of the site (or -999 to let the system find the altitude of a point in our elevation database)
`cmd=$cmd"&firstday=2009-10-01"`	# start date
`cmd=$cmd"&lastday=2009-10-31"`	# end date
`cmd=$cmd"&duration=m"`	# h (hourly) d (daily) m (monthly) w (weekly) or 15 (minutes)
`cmd=$cmd"&time=TU"`	# TU (universal time) ou TST (true solar time)
`cmd=$cmd"&slope=15"`	# tilt of the plane in degrees from 0° (horizontal) to 90°, or -999 for the normal plane
`cmd=$cmd"&azimuth=180"`	# azimuth of the plane in degrees from 0° (North), 90° (East), 180° (South) to 270° (West)
`cmd=$cmd"&albedo=0.2"`	# albedo (from 0.0 to 1.0)
`wget -O test_wget.csv $cmd`	# Final Command

NB1: the parameter -O with the name of the output file is optional.
NB2: from a Linux platform, add the command:#!/bin/bash before these command elements.

1 The three Examples:
wget -O Paris_HCday_May2006.csv "http://www.helioclim.org/pub/hc3v2_request.php?geopoint=48.86,2.33&elevation=-999&firstday=2006-05-01&lastday=2006-05-31&duration=d&time=TU&slope=0&azimuth=0&albedo=0.2"

wget -O Madrid_HChourDNI_August15_312010.csv "http://www.helioclim.org/pub/hc3v2_request.php?geopoint=40.4,-3.68&elevation=-999&firstday=2010-08-15&lastday=2010-08-31&duration=h&time=TU&slope=-999&azimuth=-999&albedo=0.2"

wget -O Berlin_HC15Incl_August15_312010.csv "http://www.helioclim.org/pub/hc3v2_request.php?geopoint=52.51,13.4&elevation=-999&firstday=2010-09-07&lastday=2010-09-07&duration=15&time=TU&slope=30&azimuth=180&albedo=0.2"

NB: Beware that if you have a tilted plane facing the southern direction, in our convention for the azimuth angle is north => 0°, east => 90°, south => 180°, and West => 270°
Change the extension of the text file .txt into .bat

2 Download wget.exe from internet

3 By double clicking on the file.bat, you should obtain in return from our servers 3 output files named:
- Paris_HCday_May2006.csv
- Madrid_HChourDNI_August15_312010.csv
- Berlin_HC15Incl_August15_312010.csv

NB: you can also use the script hc3v3_request.php instead of hc3v2_request.php. However, as MINES ParisTech had not yet published on this latest version of the database, we cannot base our discussion on peer-reviewed publications.

Top of page

Build the HC1 wget command

As soon as you have an automatic access to any of the HC3 annual subscription, you can automatically access HC1 following the command below:

Step Action

Listing of the input parameters:
cmd="http://www.helioclim.org/pub/hc3v2_request.php?"

`cmd=$cmd"geopoint=40,0"`	# lat, lon
`cmd=$cmd"&firstday=2009-10-01"`	# start date
`cmd=$cmd"&lastday=2009-10-31"`	# end date
`cmd=$cmd"&duration=m"`	# m, w, or d
`wget -O test_wget.csv $cmd`	# final command

NB: the parameter -O with the name of the output file is optional.
NB2: from a Linux platform, add the command:#!/bin/bash before these command elements.

1 _Copy these lines into a text file,
wget -O Nice.csv "http://www.helioclim.org/pub/hc1_request.php?geopoint=43.7,7.25&firstday=1985-01-01&lastday=1985-01-31&duration=d"

Change the extension of the text file .txt into .bat

2 Download wget.exe from internet

3 By double clicking on the file.bat, you should obtain in return from our servers a file named:
- Nice.csv

Top of page

Another option: the CURL request

The wget command implies one file.bat per station. But in the case of numerous stations (more than 100), it turns out to be interested to use another machine-to-machine command, named "curl". The command is built similarly than wget except the parameter -F "stations=@stations.txt" with is used as the list of inputs parameters, as follows:

Command

curl -F "stations=@stations.txt" -o result_curl.txt http://www.helioclim.org/pub/hc3v2_request.php

Content of the stations.txt file:

#id lat lon elevation UT time_step first_date last_date slope azimut

st_Paris 48.86 2.33 -999 UT d 2006-05-01 2006-05-31 0.0 0.0 0.2

st_Madrid 40.4 -3.68 -999 UT h 2010-08-15 2010-08-31 -999 -999 0.2

st_Berlin 52.51 13.4 -999 UT 15 2010-09-07 2010-09-07 30 180 0.2

Limit of curl: the timeout for the requests is fixed to 15 min and the maximum output file size is 5 Mega. This corresponds approximately to 2500 stations with a one-day request of daily or hourly values. Anyway, we deeply advise our Customers to send in parallel not more that 1000 stations per Curl request. The parallelization divides by the same factor the time of retrieval. We recommand as well not to exceed 5 parallel requests.
Download here the executable curl.zip

Top of page

FAQ

Q1_ "I did all what you explain in this page, but it is not working! What is the pb?"

Case 1

When you launch the "http..." command into your internet navigator, you get the error message: "Unregistered IP address".

Solution 1: We haven't allowed your IP to access automatically the HelioClim data, or the IP sent is private (valid only in your internal network), whereas we need your public IP. Please contact us if you need a quotation or to communicate us the correct IP.

Case 2

When you launch the "http..." command into your internet navigator, you get the correct result. The IP address you sent us is allowed to enter our system and retrieve the data you purchase. Good news! But...

Solution 2.1: The problem is that you need now to deal with internal network constraints. Your internet browser accesses our database because he knows which "gate" (proxy, firewall...) to use to get out and enter in your internal network. It is not obvious that your wget executable knows this information. You can try to specify your proxy in your file.bat, just before the wget commands.
For instance, our proxy is: http://proxy.scep/cache.pac. If you place this URL in your navigator, a function is displayed on the screen, which returns the address of your proxy(ies). In our case, it is returned: PROXY proxy.ensmp.fr:3129; which corresponds to PROXY name-or-IP-add:port-number;. You simply have to put the line: set http_proxy=http://proxy.ensmp.fr:3129 or set http_proxy=proxy.ensmp.fr:3129 before your list of wget commands in your .bat file. 1-2-3... launch! it should work :o).

In a Windows environment, test in a dos environment, the command line:
>wget -O toto.html http://www.helioclim.org (capital letter "o" like oliver)
The IP address of helioclim.org should appear in your dos environment (picture 1: 194.167.55.80), and if you open toto.html, you should retrieve the main main of the helioclim.org website without the images as shown in picture 2.
(click on each image to magnify)

1_ command dos

2_ html page

Solution 2.2: The second one is that your file explorer can hide by default the known extensions of your files. Instead of creating a file.bat to launch your wget command, you create a file.bat.txt without being aware of it!! To check this point, if you double-click on your file.bat file, your text editor must not open. if this is the case, you need to rename the file selecting "all files format" before saving.

Q.2 Why two requests inside the same pixel MSG return 2 different irradiation values?

Click here to get the answer.

Q3_ I would like to change the spaces in the output file by semi colons, to make it csv compatible. How can I do that?

Click here to download the sed.exe file and associated libraries. To replace automatically the " " by ";" in your test.txt file obtained using a wget command, you can use the following command:
sed "s/ /;/g" test.txt > test.csv
As a consequence, you can create a .bat file with the following command lines: wget -O test.txt "http://www.helioclim.org/pub/hc3v2_request.php?geopoint=45,5&elevation=-999&firstday=2005-01-01&lastday=2005-05-31&duration=m&time=TU&slope=-999&azimuth=-999&albedo=0.2" sed "s/ /;/g" test.txt > test.csv del test.txt
=> the file test.txt will be automatically deleted, and replaced by the test.csv file with ";" as separator.

Top of page