Filtering ESDs on the Grid

Using the command one can filter ESDs given a list of events. Here is a perscription for how to do that using the Grid. These commands have been tested and validated using release on a lxplus node.

  1. Create a file called eventlist.txt containing the events you want to select. The format should have two columns the first is run number the second is event number. This file must be in the local directory where you will issue the prun command.
  2. Set up your release and grid environment. I have used
  3. Issue the following command in a directory containing the file eventlist.txt:
    %>prun –exec “ filter-files -o filter.ESD.pool.root -s eventlist.txt \`echo %IN | sed ‘s/,/ /g’\`” –outputs filter.ESD.pool.root –athenaTag=,AtlasProduction –nFilesPerJob 12 –dbRelease LATEST –inDS ESDDATASET –outDS OUTPUTDATASETNAME
    Here you would like to change ESDDATASET to the dataset you wish to run on, and OUTPUTDATSETNAME to the output dataset name. Everything else should be the same.
  4. When the prun jobs have completed dq2-get the results using:
  5. cd to the directory of OUTPUTDATSETNAME
  6. Merge the output ESD files with the following command: –omitvalidation=ALL inputESDFile=`ls user*.root* | tr -d ‘\n’ |sed ‘s/.rootu/.root,u/g’` –ignoreerrors=ALL autoConfiguration=everything outputESDFile=Filter.merge.ESD.pool.root

This final command will produce a single output file from all the subjob ESDs, it ignores ESDs with zero entries, and the time it takes to run is dependent on the number of ESDs. For run 156682 with 66 subjobs it took ~10min to run interactively on an lxplus node. In total this took ~40min from start to finish.

Leave a Reply

You must be logged in to post a comment.