Submit allpix2 jobs to CERN HTCondor

Simulation / Configuration Questions
1

Dear developers and users,

Do anyone konws how to run allpix2 with CREN lxplus batch service? When I run allpix2 with large event numbers, the output file in ROOTObjectWriter will be very huge. So I plan analysing this ROOT file in HTCondor and save results back to my local lxplus.

Regards,
Shen

2

Dear Shen,

we have been experimenting with a job submission tool a little bit in the past, but as you can see in our issue tracker, we don’t have a good solution available yet: https://gitlab.cern.ch/allpix-squared/allpix-squared/-/issues/100

May I ask what exactly you mean with analysing the ROOT file in HTCondor?

It is possible to run Allpix Squared jobs on HTCondor, but it doesn’t fully seem like what you would like to do, because then you would still have the rather large ROOT file.

In order to run Allpix Squared on HTCondor, what I do is the following:
Prepare a script, e.g. runAllpixSquared.sh. In this script the actual call to Allpix Squared is placed, e.g.

#!/bin/bash

echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node

source /cvmfs/clicdp.cern.ch/software/allpix-squared/latest/x86_64-centos7-gcc8-opt/setup.sh

echo "Which ap2: `which allpix`"
echo "Directory: `pwd`"

echo "These are the run options: $@"

allpix -c /path/to/allpix/squared/examples/example.conf $@
echo "Done."

In addition, a file like condorParameters.cfg is recommended:

Executable = runAllpixSquared.sh
Requirements = (OpSysAndVer == "CentOS7")
transfer_output_files = output/histo.root
output = condor_job_$(cluster).out
log = jobsub.log
queue

Not all of the parameters are required. You can extend this file with more parameters (see https://batchdocs.web.cern.ch/local/submit.html)

Finally, submit a job with a command like this:

condor_submit -batch-name "allpix_job" arguments="-o root_file=histo.root" condorParameters.cfg

I hope I could help you with these explanations and hints - let me know if you have further questions.

Cheers
Paul

EDIT: My apologies, lxplus works a bit differently than our DESY based grid, so there seems to be an issue. I’ll come back to you with a solution.

EDIT2: I updated the instructions above. Changes are: Providing the absolute filename to the config file in the run script and explcitly stating the name of the file to be transferred back in condorParameters.cfg.

3

Hi,

Just to add - this is the submission file I used many times for HTCondor in CERN LXPLUS:

# redirect logging and error output:
output                  = apsq.$(ClusterId).$(ProcId).out
error                   = apsq.$(ClusterId).$(ProcId).err
log                     = apsq.$(ClusterId).log

# set up file transfer back to this folder
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT_OR_EVICT

# ship environment variables
getenv                  = True

# queue the job
+JobFlavour = "nextweek"
queue

the condor command is then:

condor_submit executable=allpix arguments="-c config.conf" condorfile.txt

I hope that helps!
Simon

4

Many thanks to @pschutze and @simonspa , I’m now using Paul’s configurations and it works very well. I will try Simon’s code later.

Cheers,
Shen

5

Hi Shen,

better late than never: If you find a good solution, please feel free to share it with us, as it might help us working towards a more generic submission tool.

Cheers
Paul

6

Hi @pschutze,

Since my cern account will be expired soon, I created a new ID in this forum.
Here is my batch jobs code (almost same with yours):

For runAllpixSquared.sh
eg.

#!/bin/bash

echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node

source /cvmfs/clicdp.cern.ch/software/allpix-squared/1.5.1/x86_64-centos7-gcc8-opt/setup.sh

echo "Which ap2: `which allpix`"
echo "Directory: `pwd`"

echo "These are the run options: $@"

allpix -c /path/to/your/ap2/config/file/myconfig.conf $@

echo "AP2 simulation Done."

For condorParameters.cfg
eg.

Executable = runAllpixSquared.sh
Requirements = (OpSysAndVer == "CentOS7")
transfer_output_files = output/target_file.root
output = condor_job_$(cluster).out
log = jobsub_run3.log
+JobFlavour = "testmatch"
queue

And the sumbit command is:

condor_submit -batch-name "allpix_job" arguments="-o ROOTObjectWriter.file_name=target_file.root" condorParameters.cfg

The codes above meet my needs well, hope it can help more users.

Cheers,
Shen