Issue with simulation locking up without termination

Aatuser · September 21, 2021, 2:17pm

Ran simulation with below configuration. The issue began around 15,000 events where I noticed there was no change with no termination from the simulation.

[AllPix]
log_level = "WARNING"
log_format = "DEFAULT"
number_of_events = 90000
detectors_file = "sic_sensor.conf"
model_paths = ./

[GeometryBuilderGeant4]
world_material = "vacuum"

[DepositionGeant4]
# --- next two lines for macro source
source_type = "macro"
file_name = "source_close.mac"
#
#--- next 6 lines for mono energetic alpha beam source
#
#particle_type = "alpha"
#source_energy = 5.486MeV
#source_type = "beam"
#beam_size = 0.001mm
#source_position = 4.5mm -0.0mm -0.5mm
#beam_direction = -6 0 1
#source_position = 2.7mm -0.0mm -0.5mm
#beam_direction = 0 0 1
physics_list = QBBC
enable_pai = true
#charge_creation_energy=7.8eV
max_step_length = 1.0um
number_of_particles = 1
range_cut = 1um
output_plots_scale = 4000ke
output_plots = true

[ElectricFieldReader]
model="linear"
bias_voltage=20.V
depletion_depth=220um
output_plots = true

#[ElectricFieldReader]
#model = "mesh"
#file_name = "efield.init"
#output_plots = 1

[DopingProfileReader]
model = "regions"
doping_concentration = [[20.um, 2.0e11], [21.um, 1.0e15], [22.um, 1.0e15]]

[SicPropagation]
temperature = 293K
charge_per_step = 250
propagate_holes = true
integration_time = 50ns
output_plots = true
timestep_min = 0.5ps
timestep_max = 0.5ns
output_animations = false
#output_plots_use_equal_scaling = false
#output_plots_align_pixels = true
#output_plots_use_pixel_units = true
output_linegraphs = false
recombination_model = "srh_auger"

[PulseTransfer]
max_depth_distance = .1um
#collect_from_implant = true
output_plots = true
output_plots_bins = 2048
output_plots_scale = 3000ke
#output_pulsegraphs = true

[DefaultDigitizer]
qdc_resolution = 14
qdc_slope = 10000
qdc_smearing = 50000
output_plots_scale = 4000ke
output_plots = true

#[CSADigitizer]
#model = "simple"
#rise_time_constant = 2.e-9s
#feedback_time_constant = 10e-9s
#integration_time = 0.2e-6s
output_plots = true
#output_pulsegraphs = true

#[TextWriter]
#include = "DepositedCharge"


# replace by [VisualizationGeant4] to run the visualization
[Ignore]
#[VisualizationGeant4]
mode = "gui"
accumulate = 1

#[ROOTObjectWriter]
#file_name = "sic_v1.root"

simonspa · September 22, 2021, 8:58am

Dear @Aatuser

I’m not sure I can help you with the information provided, because:

you are using a module in your simulation chain that is not part of the official Allpix Squared release, so I don’t have its source code or binary
The G4 source macro file would be necessary
the geometry and detector models used are missing

However, I did a wild guess and

switched to the directly defined source in your configuration
used a geometry from the examples with a single detector
switched [SicPropagation] for [GenericPropagation] as the parameters seem to be the same

and ran the simulation, but cannot reproduce the issue you are having. Could you make sure this is not a problem with the PC you are running on (lack of disk space or so) and provide me with additional information that might help me in tracking down this issue?

All the best,
Simon

Aatuser · September 23, 2021, 12:43am

Simon,

I checked the disk space and appear to have ~200 GB. The source macro is

gps/source/intensity .85
/gps/particle alpha

/gps/particle alpha
/gps/pos/type Plane
/gps/pos/shape Circle
/gps/ang/type iso
/gps/pos/centre 0. 0. -110.500 um
#/gps/pos/centre 0. 0. -0.15025 mm
#/gps/pos/centre 0. 0. -0.1703 mm
/gps/pos/radius 1.5 mm
/gps/ene/mono 5.486 MeV

/gps/source/add .12
/gps/particle alpha

/gps/particle alpha
/gps/pos/type Plane
/gps/pos/shape Circle
/gps/ang/type iso
/gps/pos/centre 0. 0. -110.500 um
#/gps/pos/centre 0. 0. -0.15025 mm
#/gps/pos/centre 0. 0. -0.1703 mm
/gps/pos/radius 1.5 mm
/gps/ene/mono 5.443 MeV

/gps/source/add .03
/gps/particle alpha

I’m not sure about the geometry but I think the one you used is correct.

Aatuser · September 24, 2021, 3:27am

@simonspa what version/Linux do you run?

simonspa · September 24, 2021, 5:52am

Dear Abigail,

also with your macro source definition I cannot reproduce your issue - for me the simulation works just fine, generating all 90k events requested and then terminating normally.

I personally run (and develop) Allpix Squared on Ubuntu, however, please have a look at the supported operating systems listed in the user manual.

Is this issue something that always occurs or only sometimes? Please provide me with a little more information:

What version of Allpix Squared are you using exactly? (printed out at start or with allpix --version)
How did you install this? Docker? CVMFS? Compiled yourself?
If this only appears sporadically, could you check if it always happens when starting with the same random_seed? The seed used is printed at startup - if you place this in the config you can exactly repeat a simulation again

All the best,
Simon

Aatuser · September 28, 2021, 12:38am

@simonspa
Is this issue something that always occurs or only sometimes?
This happens very frequently, and on different machines. It has occurred with different models and counts as low as 4,400 (hangs on 4,399).
What version of Allpix Squared are you using exactly?

We are running: Ubuntu 2-0.04.2LTS
Also - we are using Geant4 version 10.7 and ROOT version 6.25
How did you install this? Docker? CVMFS? Compiled yourself?
We compiled it ourselves, following the build instructions that use CMAKE. We use the same procedure with the previous version 1.6.X, with no issues.

Adding the same random seed did not help. Could it be an issue with the PulseTransfer model being used?

simonspa · September 29, 2021, 9:27am

Hi Abigail,

okay, this is pretty much my own setup where I never experienced these issues. Maybe your pointer is right and it’s the PulseTransfer module. If you switch this for SimpleTransfer (thus ditching the transient simulation of course…) do you still have that problem?

If not, then it might be related to charge carriers arriving very late and therefore creating huge pulses that require lots of memory. In principle the system libs should throw a std::bad_alloc exception then but maybe not all systems do this?

Some more guesswork: the issue would then be (apart from code we could optimize on our side…) in the private module SicPropagation which you are using. If this is about simulating charge carrier propagation in SiC then I would recommend to stick with the original code in GenericPropagation (or TransientPropagation if that was the basis) and to just flip out the mobility and recombination models. They are located in src/physics.

If you would like me to have a look, please feel free to contact me directly if the code cannot be shared publicly.

All the best,
Simon

Aatuser · October 19, 2021, 6:02pm

Simon,
Apologies for the delay in response. I was able to get the code to work by taking an example that works and rebuilding the code using simple transfer. Since it works, I do believe the error was due to using the pulse transfer module.

simonspa · October 20, 2021, 6:46am

Dear @Aatuser

that’s good to hear, I’m glad you were able to solve it!
We will have some improvements on the whole pulse storage at some point to avoid this type of issue all together…

Cheers,
Simon

Aatuser · December 14, 2021, 2:35pm

Simon, we noticed we began to have the same issue pop up again. Upon further investigationg it seems the issue might be when enabling pai in depositionGeant4

simonspa · December 15, 2021, 3:02pm

Hi Abigail,

I can 100% reproduce this on my local machine, so this seems to be a Geant4 bug. I can recreate this problem both with Geant4 10.7 and the newly released Geant4 11.0. When setting

enable_pai = true

I get infinite loops somewhere in Geant4 as gdb tells me.

I will try to find some time to dig into Geant4 to find out what the problem might be, in the meantime you could try to get some feedback on this in the Geant4 user forums maybe:

Best regards,
Simon