Simulation filling up memory until it gets killed

Issues
1

Hello,
I am simulating neutron measurements using a converter and then a standard 300um and 500 um timepix silicon detectors. The simulation was working fine until I switched from the [GenericPropagation] module to the [TransientPropagation] module. I left some commented lines in there, which are settings I played around with to see if there is a difference (there is not). What happens is that the memory fills up and when it reaches the maximum capacity the program is killed without any error message (only that it is killed). I played around with “workers” and “buffer_per_worker”, putting it higher or lower. It only changes the time it takes to fill up the memory. It is difficult to find a minimal working code to reproduce the error as it takes several hours before the memory is filled up. It is necessary to have several thousand events for the memory to fill up. My suspicion is that somewhere some allocated memory is not freed up until the program is finished or killed. I watched the memory fill up and getting freed up. The only work around I have right now is to simulate a small amount of events per file.

I use the latest stable Allpix code (3.1.1), Geant4 v11.2.2 and root v6.32.02 on Ubuntu 22.04.5 LTS.

I hope, someone who understand the code better than me can fix it? I am not sure what other information I can give. Let me know if there is anything.

Kind regards
Stefan

I cannot upload files (it says new users are not allowed to upload files), so I just dump the config and related file content here:
//main config start//
[Allpix]
number_of_events = 100000
detectors_file = “Moon_Detector_PE.conf”
#workers = 100

[GeometryBuilderGeant4]

[DepositionGeant4]
physics_list = QGSP_INCLXX_HP_EMZ
source_type = “macro”
file_name = “GPS.mac”
record_all_tracks = 1

number_of_particles = 1

#cutoff_time = 882s

[ElectricFieldReader]
model = “linear”
bias_voltage = 150V
depletion_voltage = 40V

[WeightingPotentialReader]
model = “pad”

[TransientPropagation]
temperature = 323K
charge_per_step = 200
timestep = 0.5ns
integration_time = 30ns
distance = 4

[PulseTransfer]
timestep = 0.5ns

[ROOTObjectWriter]
file_name = “PEconv.root”
exclude = PropagatedCharge,DepositedCharge
//main config end//

//detector file start//
[detector]
type = “timepix”
sensor_thickness = 500um
position = 0 0 0mm
orientation = 0 0 0

[DeadLayer]
type = “box”
size = 14.4mm 14.4mm 800nm
position = 0mm 0mm -250.4um
orientation = 0 0 0deg
material = “silicon”
role = “passive”

[Electrode]
type = “box”
size = 14.4mm 14.4mm 600nm
position = 0mm 0mm -251.1um
orientation = 0 0 0deg
material = “aluminum”
role = “passive”

[Converter]
type = “box”
size = 14.4mm 14.4mm 2mm
position = 0mm 0mm -1251.4um
orientation = 0 0 0deg
material = “G4_POLYETHYLENE”
role = “passive”
//detector file end//

//GPS.mac//
/gps/particle neutron

/gps/ene/type Lin
/gps/ene/intercept 1
/gps/ene/min 1e8 eV
/gps/ene/max 1e9 eV

/gps/pos/type Plane
/gps/pos/shape Rectangle
/gps/pos/halfx 7.2 mm
/gps/pos/halfy 7.2 mm
/gps/pos/centre 0 0 -12 mm
/gps/direction 0 0 1
//GPS.mac//

2

Hi @StefanGohl ,

This sounds interesting! My first thought was to ask whether you noticed that it gets killed when you hit a specific event number if you fix the random seed. This would point to a large deposit in one event, leading to very many charge carriers being propagated and eating up all the RAM.
However, you mention that the memory seems to fill up as time goes on, if I understand correctly.

Does this persist even with a small buffer or if you run single-threaded? I take it that it does from your statement, just double-checking.

The first thing I would do is reduce the distance in [TransientPropagation]. A distance of 4 means that you collect information on the pixel the charge is moving in along with the surrounding pixels four layers out (so in total 81 pixels for each charge carrier movement), which comes to quite a lot of resources in the end providing little information. A distance of 1 makes the induced charge be calculated on 3x3 pixels, which is normally plenty.

You could also try running without [ROOTObjectWriter]. I don’t think there is a memory leak, but rather the open ROOT file becoming excessively large as events go on, but this could be a way to show it. You could also try including fewer objects in the output file.

Kind regards,
Håkan

3

Hello Hakan,
I was playing around with the buffer size and number of threads (I didn’t run only a single threat), but this only changed the time it took for the memory to fill up. The number of events that were processed before the crash was random, though I didn’t run it with a fixed seed. However, the memory fills up slowly over time as events are processed. I did reduce the distance to 3 before without a noticeable change.

I will try your suggestions for more tests and come back with the results.

Kind regards,
Stefan