Command Line Tools

If you’ve installed KM3Pipe via pip, you have access to some useful command line utilities out of the box.

KM3Pipe

Most of the commands have to be prefixed with km3pipe to avoid possible nameclashes and also for an improved overview. You can for example simply run km3pipe -h in your shell to see all available commands:

$ km3pipe --help
KM3Pipe command line utility.

Usage:
    km3pipe test
    km3pipe update [GIT_BRANCH]
    km3pipe createconf [--overwrite] [--dump]
    km3pipe detx DET_ID [-m] [-t T0_SET] [-c CALIBR_ID] [-o OUTFILE]
    km3pipe detectors [-s REGEX] [--temporary]
    km3pipe rundetsn [--temporary] RUN DETECTOR
    km3pipe retrieve DET_ID RUN [-o OUTFILE]
    km3pipe git
    km3pipe git-short
    km3pipe (-h | --help)
    km3pipe --version

Options:
    -h --help           Show this screen.
    -m                  Get the MC detector file (flips the sign of DET_ID).
    -c CALIBR_ID        Geometrical calibration ID (eg. A01466417)
    -o OUTFILE          Output filename.
    -t T0_SET           Time calibration ID (eg. A01466431)
    -s REGEX            Regular expression to filter the runsetup name/id.
    --temporary         Use a temporary session. [default: False]
    --overwrite         Overwrite existing config [default: False]
    DET_ID              Detector ID (eg. D_ARCA001).
    DETECTOR            Detector (eg. ARCA).
    GIT_BRANCH          Git branch to pull (eg. develop).
    RUN                 Run number.

update

The command km3pipe update [GIT_BRANCH] should be used to (once installed) get latest version of KM3Pipe. If no git branch is specified, it will pull the master branch, which always holds the stable releases.

If you want to try the newest features, pull the develop branch via km3pipe update develop. This is 99.9% stable, since we always do our experiments in feature/x branches. However, we might break it sometimes. Have a look at our git repository to see what we’re working on if you’re interested.

runtable

To get a list of runs taken with one of the KM3NeT detectors, you can use the runtable command.

The following command pulls the last 10 runs which matches the regular expression PHYS. In other words, you’ll get a list of physics runs:

km3pipe runtable -n 10 -s PHYS 14

An example output is:

RUN UNIXSTARTTIME   STARTTIME_DEFINED       RUNSETUPID      RUNSETUPNAME    T0_CALIBSETID   DATETIME
848 3611    1465506000553   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-09 21:00:00.553000+00:00
849 3612    1465506060554   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-09 21:01:00.554000+00:00
850 3613    1465509600606   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-09 22:00:00.606000+00:00
851 3614    1465509660607   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-09 22:01:00.607000+00:00
852 3615    1465520400799   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 01:00:00.799000+00:00
853 3616    1465520460800   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 01:01:00.800000+00:00
854 3617    1465531200966   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 04:00:00.966000+00:00
855 3618    1465531260967   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 04:01:00.967000+00:00
856 3619    1465542000119   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 07:00:00.119000+00:00
857 3620    1465542060119   Y       A01466427       PHYS.1606v1-TMP.HV-SFP.Power-XTRA.700ns         2016-06-10 07:01:00.119000+00:00

triggersetup

Get the trigger setup (description and optical/acoustic DataFilter settings) for a given runsetup ID:

$ triggersetup -h
Prints the trigger information of a given run setup.

Usage:
    triggersetup RUNSETUP_OID
    triggersetup (-h | --help)
    triggersetup --version

Options:
    RUNSETUP_OID   The run setup identifier (e.g. A02004580)
    -h --help      Show this screen.

triggermap

Shows a histogram (similar to the one on the online monitoring pages) of the trigger contribution for events:

$ triggermap -h
This script creates histogram which shows the trigger contribution for events.

Usage:
    triggermap [-d DET_ID -p PLOT_FILENAME -u DU] FILENAME
    triggermap --version

Option:
    FILENAME          Name of the input file.
    -u DU             Only plot for the given DU.
    -d DET_ID         Detector ID [default: 29].
    -p PLOT_FILENAME  The filename of the plot [default: trigger_map.png].
    -h --help         Show this screen.

DataBase

streamds

The utility streamds can be used to interact with the database directly from the shell:

$ streamds --help
Access the KM3NeT StreamDS DataBase service.

Usage:
    streamds
    streamds list
    streamds upload [-q] CSV_FILE
    streamds info STREAM
    streamds get STREAM [PARAMETERS...]
    streamds (-h | --help)
    streamds --version

Options:
    STREAM      Name of the stream.
    CSV_FILE    Tab separated data for the runsummary tables.
    PARAMETERS  List of parameters separated by space (e.g. detid=29).
    -q          Dryrun! This will upload the parameters with a TEST_ prefix.
    -h --help   Show this screen.

PipeInspector

PipeInspector is a tool to inspect different kinds of data formats used within the KM3NeT collaboration. It utilises the KM3Pipe framework to deal with data I/O and allows easy access to the stored information.

PipeInspector

It is currently in an early alpha status, but already able to handle the DAQ binary data, ROOT and Aanet-ROOT format.

If you installed KM3Pipe via pip, you’ll be able to launch pipeinspector directly from the terminal:

pipeinspector /path/to/data/file.ext

HDF5 CLI Utils

tohdf

Convert an aanet/root/evt/jpp file to hdf5.

Example:

tohdf5 --aa-fmt=jevt_jgandalf some_jgandalf_file.aa.root

Help output:

  $ tohdf5 --help
  Convert ROOT and EVT files to HDF5.

  Usage:
      tohdf5 [options] FILE...
      tohdf5 (-h | --help)
      tohdf5 --version

  Options:
      -h --help                       Show this screen.
      -n EVENTS                       Number of events/runs.
      -o OUTFILE                      Output file.
      -j --jppy                       (Jpp): Use jppy (not aanet) for Jpp readout
      -l --with-timeslice-hits        (Jpp) Include timeslice-hits [default: False]
      -s --with-summaryslices         (Jpp) Include summary slices [default: False]
      --aa-format=<fmt>               (Aanet): Which aanet subformat ('minidst',
                                      'orca_recolns', 'gandalf', 'gandalf_new',
                                      'generic_track') [default: None]
      --aa-lib=<lib.so>               (Aanet): path to aanet binary (for old
                                      versions which must be loaded via
                                      `ROOT.gSystem.Load()` instead of `import aa`)
      --aa-old-mc-id                  (aanet): read mc id as `evt.mc_id`, instead
                                      of the newer `mc_id = evt.frame_index - 1`
--aa-run-id-from-header         (Aanet) read run id from header, not event.
      --correct-zed                   (Aanet) Correct offset in mc tracks (aanet)
                                      [default: False]
      --do-not-correct-mc-times       (Aanet) Don't correct MC times.
      --skip-header                   (Aanet) don't read the full header.
                                      Entries like `genvol` and `neventgen` will
                                      still be retrived. This switch enables
                                      skipping the `get_aanet_header` function only.
                                      [default: False]
      --ignore-hits                   Don't read the hits, please [default: False].
      -e --expected-rows NROWS        Approximate number of events.  Providing a
                                      rough estimate for this (100, 10000000, ...)
                                      will greatly improve reading/writing speed and
                                      memory usage. Strongly recommended if the
                                      table/array size is >= 100 MB. [default: 10000]

calibrate

Apply calibration and time calibration to an HDF5 file.

Example:

calibrate km3net_jul13_90m_r1494.detx km3net_jul13_90m_muatm10T23.h5

$ calibrate -h
Apply calibration and time calibration from a DETX to an HDF5 file.

Usage:
    calibrate DETXFILE HDF5FILE
    calibrate (-h | --help)
    calibrate --version

Options:
    -h --help       Show this screen.

hdf2root

Convert a HDF5 file to a plain ROOT file (requires rootpy + root_numpy).

Example:

hdf52root FOO.h5 BAR.h5

$ hdf2root --help
Convert HDF5 to vanilla ROOT.

Usage:
     hdf2root FILES...
     hdf2root (-h | --help)

Options:
    -h --help           Show this screen.

h5info

Show some H5 metadata (KM3 H5 version, km3pipe version, etc).

Example:

$ h5info km3net_jul13_90m_muatm50T655.km3_v5r1.JTE_r2356.root.0-499.h5
format_version: b'4.1'
km3pipe: b'7.1.2.dev'
pytables: b'3.4.0'


$ h5info --help

Show the km3pipe etc. version used to write a H5 file.

Usage:
  h5info FILE [-r]
  h5info (-h | --help)
  h5info --version

Options:
  FILE        Input file.
  -r --raw    Dump raw metadata.
  -h --help   Show this screen.

h5tree

Print header info (TODO)

h5tree

Print the structure of a H5 file + minimal metadata.

For a less pretty, more verbose output, use the ptdump util instead.

Example:

$ h5tree elec.h5
KM3HDF5 v4.2
Number of Events: 169163
├── hits
│  ├── _indices
│  ├── channel_id
│  ├── dom_id
│  ├── event_id
│  ├── time
│  ├── tot
│  └── triggered
├── mc_hits
│  ├── _indices
│  ├── a
│  ├── event_id
│  ├── origin
│  ├── pmt_id
│  └── time
├── reco
│  └── gandalf
├── talala

ptdump (from PyTables)

Inspect the contents of a HDF5 file, walking through all the subgroups.

Read the PyTables docs for more details.

Example output:

┌─[moritz@averroes ~/km3net/data ]
└─╼ ptdump nueCC.h5
/ (RootGroup) ''
/event_info (Table(121226,), shuffle, zlib(5)) ''
/hits (Table(0,), shuffle, zlib(5)) ''
/mc_hits (Table(0,), shuffle, zlib(5)) ''
/mc_tracks (Table(242452,), shuffle, zlib(5)) ''
/reco (Group) ''
/reco/aa_shower_fit (Table(121226,), shuffle, zlib(5)) ''
/reco/dusj (Table(121226,), shuffle, zlib(5)) ''
/reco/j_gandalf (Table(121226,), shuffle, zlib(5)) ''
/reco/q_strategy (Table(121226,), shuffle, zlib(5)) ''
/reco/reco_lns (Table(121226,), shuffle, zlib(5)) ''
/reco/thomas_features (Table(121226,), shuffle, zlib(5)) ''

pttree (from PyTables)

Show the memory consumption of a HDF5 file. As you can see below, the overwhelming majority of space is used by the hits, as expected.

Example output:

┌─[moritz@ceres ~/pkg/km3pipe/examples/data ]
└─╼ pttree km3net_jul13_90m_muatm50T655.km3_v5r1.JTE_r2356.root.0-499.h5

------------------------------------------------------------

/ (RootGroup)
+--hits (Group)
|     ... 7 leaves, mem=35.0MiB, disk=8.1MiB [66.3%]
+--mc_hits (Group)
|     ... 6 leaves, mem=15.2MiB, disk=3.8MiB [31.6%]
+--mc_tracks (Table)
|     mem=858.4KiB, disk=251.6KiB [ 2.0%]
`--event_info (Table)
      mem=56.6KiB, disk=6.3KiB [ 0.1%]

------------------------------------------------------------
Total branch leaves:    15
Total branch size:      51.2MiB in memory, 12.2MiB on disk
Mean compression ratio: 0.24
HDF5 file size:         12.5MiB
------------------------------------------------------------

km3h5concat

This tool can be used to merge HDF5 files:

$ km3h5concat -h
Concatenate KM3HDF5 files via pipeline.

Usage:
    km3h5concat [options] OUTFILE FILE...
    km3h5concat (-h | --help)
    km3h5concat --version

Options:
    -h --help                       Show this screen.
    --verbose                       Print more output.
    --debug                         Print everything.
    -n=NEVENTS                      Number of events; if not given, use all.