.. note::
    :class: sphx-glr-download-link-note

    Click :ref:`here <sphx_glr_download_auto_examples_plot_pipeline_toh5.py>` to download the full example code
.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_pipeline_toh5.py:


=======================
ToHDF5 Pipeline example
=======================




.. code-block:: python


    # Authors: Moritz Lotze <mlotze@km3net.de>
    # License: BSD-3
    # Date: 2018-07-12
    # Status: Under construction...

    from __future__ import absolute_import, print_function, division

    import tables as tb

    from km3pipe import Pipeline
    from km3pipe.io import EvtPump, HDF5Sink

    from km3modules.common import StatusBar







Preparation
-----------
Let's define the inputs / outputs first -- those would be coming from a CLI
parser in practice.



.. code-block:: python


    N_EVENTS = 200000
    IN_FNAME = 'data/numu_cc.evt'
    OUT_FNAME = 'data/numu_cc.h5'







Also, in this case we don't really want to dump the data onto disk, so we
create an in-memory-ony file, and pass it as the ``h5file`` arg to the
hdf5sink. to actually write out a file, just specify an outfile name
(commented out here).



.. code-block:: python


    OUTFILE = tb.open_file(
        # create the file in memory only
        OUT_FNAME,
        'w',
        driver="H5FD_CORE",
        driver_core_backing_store=0,
    )







Setting up the pipeline
-----------------------



.. code-block:: python


    pipe = Pipeline(timeit=True)
    pipe.attach(EvtPump, filename=IN_FNAME)
    pipe.attach(StatusBar, every=25)
    pipe.attach(
        HDF5Sink,
        # filename=OUT_FNAME,
        h5file=OUTFILE,
    )





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    km3pipe.io.evt.EvtPump.EvtPump: Automatic tag parsing enabled.
    km3pipe.io.evt.EvtPump.EvtPump: Opening data/numu_cc.evt


Draining the pipeline
---------------------



.. code-block:: python


    pipe.drain(N_EVENTS)




.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Pipeline and module initialisation took 0.006s (CPU 0.001s).
    --------------------------[ Blob      25 ]---------------------------
    --------------------------[ Blob      50 ]---------------------------
    --------------------------[ Blob      75 ]---------------------------
    --------------------------[ Blob     100 ]---------------------------
    ================================[ . ]================================
    km3pipe.io.hdf5.HDF5Sink.HDF5Sink: HDF5 file written to: dump.h5
    ============================================================
    109 cycles drained in 7.306375s (CPU 2.801937s). Memory peak: 321.33 MB
      wall  mean: 0.031177s  medi: 0.026820s  min: 0.022870s  max: 0.148772s  std: 0.015022s
      CPU   mean: 0.025471s  medi: 0.025212s  min: 0.021987s  max: 0.033946s  std: 0.001715s
    EvtPump - process: 3.255s (CPU 2.637s) - finish: 0.000s (CPU 0.000s)
      wall  mean: 0.029858s  medi: 0.025472s  min: 0.021843s  max: 0.147596s  std: 0.014994s
      CPU   mean: 0.024196s  medi: 0.023974s  min: 0.019653s  max: 0.029505s  std: 0.001620s
    StatusBar - process: 0.001s (CPU 0.001s) - finish: 0.000s (CPU 0.000s)
      wall  mean: 0.000157s  medi: 0.000156s  min: 0.000147s  max: 0.000170s  std: 0.000010s
      CPU   mean: 0.000158s  medi: 0.000157s  min: 0.000148s  max: 0.000171s  std: 0.000011s
    HDF5Sink - process: 0.136s (CPU 0.132s) - finish: 3.893s (CPU 0.019s)
      wall  mean: 0.001248s  medi: 0.001149s  min: 0.000811s  max: 0.007418s  std: 0.000690s
      CPU   mean: 0.001213s  medi: 0.001152s  min: 0.000813s  max: 0.004396s  std: 0.000403s


**Total running time of the script:** ( 0 minutes  7.319 seconds)


.. _sphx_glr_download_auto_examples_plot_pipeline_toh5.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download

     :download:`Download Python source code: plot_pipeline_toh5.py <plot_pipeline_toh5.py>`



  .. container:: sphx-glr-download

     :download:`Download Jupyter notebook: plot_pipeline_toh5.ipynb <plot_pipeline_toh5.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.readthedocs.io>`_