ToHDF5 Pipeline example

# Authors: Moritz Lotze <mlotze@km3net.de>
# License: BSD-3
# Date: 2018-07-12
# Status: Under construction...

from __future__ import absolute_import, print_function, division

import tables as tb

from km3pipe import Pipeline
from km3pipe.io import EvtPump, HDF5Sink

from km3modules.common import StatusBar

Preparation

Let’s define the inputs / outputs first – those would be coming from a CLI parser in practice.

N_EVENTS = 200000
IN_FNAME = 'data/numu_cc.evt'
OUT_FNAME = 'data/numu_cc.h5'

Also, in this case we don’t really want to dump the data onto disk, so we create an in-memory-ony file, and pass it as the h5file arg to the hdf5sink. to actually write out a file, just specify an outfile name (commented out here).

OUTFILE = tb.open_file(
    # create the file in memory only
    OUT_FNAME,
    'w',
    driver="H5FD_CORE",
    driver_core_backing_store=0,
)

Setting up the pipeline

pipe = Pipeline(timeit=True)
pipe.attach(EvtPump, filename=IN_FNAME)
pipe.attach(StatusBar, every=25)
pipe.attach(
    HDF5Sink,
    # filename=OUT_FNAME,
    h5file=OUTFILE,
)

Out:

km3pipe.io.evt.EvtPump.EvtPump: Automatic tag parsing enabled.
km3pipe.io.evt.EvtPump.EvtPump: Opening data/numu_cc.evt

Draining the pipeline

pipe.drain(N_EVENTS)

Out:

Pipeline and module initialisation took 0.006s (CPU 0.001s).
--------------------------[ Blob      25 ]---------------------------
--------------------------[ Blob      50 ]---------------------------
--------------------------[ Blob      75 ]---------------------------
--------------------------[ Blob     100 ]---------------------------
================================[ . ]================================
km3pipe.io.hdf5.HDF5Sink.HDF5Sink: HDF5 file written to: dump.h5
============================================================
109 cycles drained in 7.306375s (CPU 2.801937s). Memory peak: 321.33 MB
  wall  mean: 0.031177s  medi: 0.026820s  min: 0.022870s  max: 0.148772s  std: 0.015022s
  CPU   mean: 0.025471s  medi: 0.025212s  min: 0.021987s  max: 0.033946s  std: 0.001715s
EvtPump - process: 3.255s (CPU 2.637s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.029858s  medi: 0.025472s  min: 0.021843s  max: 0.147596s  std: 0.014994s
  CPU   mean: 0.024196s  medi: 0.023974s  min: 0.019653s  max: 0.029505s  std: 0.001620s
StatusBar - process: 0.001s (CPU 0.001s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000157s  medi: 0.000156s  min: 0.000147s  max: 0.000170s  std: 0.000010s
  CPU   mean: 0.000158s  medi: 0.000157s  min: 0.000148s  max: 0.000171s  std: 0.000011s
HDF5Sink - process: 0.136s (CPU 0.132s) - finish: 3.893s (CPU 0.019s)
  wall  mean: 0.001248s  medi: 0.001149s  min: 0.000811s  max: 0.007418s  std: 0.000690s
  CPU   mean: 0.001213s  medi: 0.001152s  min: 0.000813s  max: 0.004396s  std: 0.000403s

Total running time of the script: ( 0 minutes 7.319 seconds)