Hi Allpix Squared team,
I am a new user and I am working on simulating an x-ray tube. The framework is easy to understand and use, so, many thanks for this!!!
At the moment I am facing some troubles with analyzing the data with PyROOT (python 3.11.5 and ROOT 6.28/06). I am using a code very similar to this example (tools/root_analysis_macros/display_mc_hits.py · master · Allpix Squared / Allpix Squared · GitLab) and since I have a rather large root file to go over (some million of events) I could track down a memory leak to these lines where the AP2 vectors are assigned to branches (tools/root_analysis_macros/display_mc_hits.py · master · Allpix Squared / Allpix Squared · GitLab). Have you noticed something similar? If yes, is there another way to access the TNonSplitBrowsable objects under the main TBranchElements of the TTrees?
Hi @ggiakous ,
thank you very much for the kind words! We’re happy to hear that new users find an easy entry point to Allpix Squared!
To be very honest, these are not the most maintained parts of the repository and serve as an example. I have used and adapted it several times, but not with the large amounts of data you seem to be dealing with.
It would be interesting to understand whether this memory leak happens on the “ROOT” side or the “user” side. Did you try to delete the vectors before overwriting their pointer per loop? There’s a bit of help at least for C based analyses here (section 14.14.3.1 GetEntry), did you try something similar?
Best
Paul
Hi all,
I faced very similar issues when trying to read with pyroot a ROOT file from an allpix simulation having only MCTrack and 1mln of simulated events. I agree with your deduction @ggiakous i think it has to do with the way ROOT handles pointers, other Pyroot users online seem to have a similar issue.
@pschutze Regarding where it happens, it seems to be a combination of both sides:
buffers for std::vector<*> branches seems to be allocated per event and not fully exposed for reset via PyROOT and storing C++ objects or addresses in Python prolongs the lifetime of the C++ objects, preventing ROOT from freeing memory (altough my main knowledge on this comes from the ROOT forum posts and chatgpt so don’t quote me on this). It might also have to do in the interplay between ROOT and python on who has “ownership” of the created objects or pointers that does not allow for their reset.
POSSIBLE SOLUTIONS:
So far options are to either use uproot, but i could not make it work yet with allpix vectors, i will try again perhaps in the next days, or to batch the reading of the ROOT file, or to use gc.collect() at the end of each loop in which you read an event or something similar to manually free the accumulation in the memory of pointers or whatever is the problem.
from my tests gc.collect() does not seem to do much (with or without it psutil gives the same RAM usage every N cycles). I do not know if it would be possible to pinpoint the problem to some variables exactly.
When checking the system monitor i can see python memory usage go up and it seems to be the only process sucking RAM in. That being said i had no luck in managing to free the memory accumulated while python runs.
However if someone else more expert on the topic can think of a better solution please let me know 
Bests
Luca
Solved: in the end for me it worked to update ROOT to the latest version and same for python, then set up an allpix-venv to use to run my python scripts and now the RAM consumption drastically reduced.
Hi @luca_tero ,
this is great news! Can you maybe post the following information:
- python version
- ROOT version
- Did you do any changes to the scripts (
gc or alike)?
This would be great for anyone trying to reproduce this.
Thank you and cheers
Paul
Hi
Python version 3.12.3
ROOT version 6.36.08
I did not changed any scripts in allpix or root or anything else.
If people still have this problem even after updating to latest ROOT or python i could also upload the code i use to read from root into python, and perhaps also the bash (made by Rickard in my group) that i used to reinstall allpix, ROOT and all dependencies and set up the virtual environment.
Bests
Luca