I'm trying to download some particle cutouts (file size about 4.5G), whatever tools I used (python requests or wget), I'll always lost connection after I download part of the file. What do you suggest I do to overcome this problem?
This is unfortunately a limitation of the current system which is hard to pin down.
The practical solution now is to split the cutout into parts. As you're requested it, it is all fields of all particle types. Do you really need this all? If so, you can maybe make 4 separate requests, one for each type (gas, dm, stars, bhs). If not, you can make even smaller requests of just the fields you want.
I'm trying to download some particle cutouts (file size about 4.5G), whatever tools I used (python requests or wget), I'll always lost connection after I download part of the file. What do you suggest I do to overcome this problem?
wget --content-disposition --header="API-Key: mykey" http://www.illustris-project.org/api/Illustris-1/snapshots/135/subhalos/16937/cutout.hdf5
--2017-02-23 17:08:55-- http://www.illustris-project.org/api/Illustris-1/snapshots/135/subhalos/16937/cutout.hdf5 Resolving www.illustris-project.org... 23.239.9.40 Connecting to www.illustris-project.org|23.239.9.40|:80... connected. HTTP request sent, awaiting response... 302 FOUND Location: http://data.illustris-project.org/app/cutout/subhalo/L75n1820FP/135/16937/2097151.127.0.0.16383.2097151/?token=bb8d5e2fd4a7c5d1d486 [following] --2017-02-23 17:08:55-- http://data.illustris-project.org/app/cutout/subhalo/L75n1820FP/135/16937/2097151.127.0.0.16383.2097151/?token=bb8d5e2fd4a7c5d1d486 Resolving data.illustris-project.org... 140.247.151.71 Connecting to data.illustris-project.org|140.247.151.71|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4874960896 (4.5G) [application/octet-stream] Saving to: ‘cutout_16937.hdf5’
cutout_16937.hdf5 11%[=====> ] 553.12M 1.01MB/s in 8m 27s
2017-02-23 17:18:21 (1.09 MB/s) - Connection closed at byte 579993600. Retrying.
--2017-02-23 17:18:22-- (try: 2) http://data.illustris-project.org/app/cutout/subhalo/L75n1820FP/135/16937/2097151.127.0.0.16383.2097151/?token=bb8d5e2fd4a7c5d1d486 Connecting to data.illustris-project.org|140.247.151.71|:80... connected. HTTP request sent, awaiting response... 403 FORBIDDEN 2017-02-23 17:18:22 ERROR 403: FORBIDDEN.
Hi Hongyu,
This is unfortunately a limitation of the current system which is hard to pin down.
The practical solution now is to split the cutout into parts. As you're requested it, it is all fields of all particle types. Do you really need this all? If so, you can maybe make 4 separate requests, one for each type (gas, dm, stars, bhs). If not, you can make even smaller requests of just the fields you want.
Tasks 5 and 6 in the API documentation cover this.
Hi Dylan,
Thank you very much for your reply. I'll try to download them separately.