Hi all! I'm trying to make a simple column density map with data from a TNG-Cluster
halo using a TNG JupyterLab session, but I seem to be unable to load the necessary
data given my 10 GB memory limit. Here's a snippet of my code:
basePath = '../sims.TNG/TNG-Cluster/output/'
haloID = 51 ## Original ID (from the left panels of Figure 2 of Nelson et al. 2024)
fields = ['Masses','Coordinates'] gas = il.snapshot.loadOriginalZoom(basePath,
99, haloID, 'gas', fields=fields)
I can load just the 'Masses' field quickly and without any problem. As expected, it
only takes 0.7 GB of memory for 'only' 193 million cells. Therefore, I would expect
'Coordinates' to require 4.2 GB, which should still be well under the 10 GB limit.
Can you help me understand what might be happening here or what I am doing
wrong?
Thanks so much in advance for this and thanks for the amazing data!
Joe
Dylan Nelson
9 Jul
The numbers are right, unfortunately loadOriginalZoom was never optimized and is extremely simple.
It simply concatenates the two relevant numpy arrays together, and this (temporarily) request 2x the memory.
You could easily fix this, by improving the function, or by just loading the two specific hdf5 files directly.
Of course this won't help for a much bigger halo, where the ideas in this thread are useful.
Hi all! I'm trying to make a simple column density map with data from a TNG-Cluster
halo using a TNG JupyterLab session, but I seem to be unable to load the necessary
data given my 10 GB memory limit. Here's a snippet of my code:
I can load just the 'Masses' field quickly and without any problem. As expected, it
only takes 0.7 GB of memory for 'only' 193 million cells. Therefore, I would expect
'Coordinates' to require 4.2 GB, which should still be well under the 10 GB limit.
Can you help me understand what might be happening here or what I am doing
wrong?
Thanks so much in advance for this and thanks for the amazing data!
Joe
The numbers are right, unfortunately
loadOriginalZoomwas never optimized and is extremely simple.It simply concatenates the two relevant numpy arrays together, and this (temporarily) request 2x the memory.
You could easily fix this, by improving the function, or by just loading the two specific hdf5 files directly.
Of course this won't help for a much bigger halo, where the ideas in this thread are useful.