km3pipe.io

A collection of io for different kinds of data formats.

Package Contents

Classes

EvtPump() Provides a pump for EVT-files.
DAQPump() A pump for binary DAQ files.
CLBPump() A pump for binary CLB files.
AanetPump() A pump for binary Aanet files.
EventPump() A pump for DAQEvents in JPP files.
CHPump() A pump for ControlHost data.
HDF5Pump() Read KM3NeT-formatted HDF5 files, event-by-event.
HDF5Sink() Write KM3NeT-formatted HDF5 files, event-by-event.
HDF5MetaData() Metadata to attach to the HDF5 file.
PicklePump() A pump for Python pickles.

Functions

get_logger(name) Helper function to get a logger
GenericPump(filenames, use_jppy=False, name=’GenericPump’, **kwargs) A generic pump which utilises the appropriate pump.
read_calibration(detx=None, det_id=None, from_file=False, det_id_table=None) Retrive calibration from file, the DB.
class km3pipe.io.EvtPump[source]

Bases:km3pipe.core.Pump

Provides a pump for EVT-files.

Parameters:
filename: str

The file to read the events from.

parsers: list of str or callables

The parsers to apply for each blob (e.g. parsers=[‘km3sim’, a_parser]) You can also pass your own function, which takes a single argument blob and mutates it. str values will be looked up in the kp.io.evt.EVT_PARSERS dictionary and ignored if not found. If parsers=’auto’, the EvtPump will try to find the appropriate parsers, which is the default behaviour. [default: ‘auto’]

cache_enabled: bool

If enabled, a cache of the event indices is created when loading the file. Enable it if you want to jump around and inspect the events non-consecutively. [default: False]

basename: str

The common part of the filenames if you want to process multiple files e.g. file1.evt, file2.evt and file3.evt. During processing, the files will be concatenated behind the scenes. You need to specify the index_stop and index_start (1 and 3 for the example).

suffix: str

A string to append to each filename (before “.evt”), when basename is given. [default: ‘’]

index_start: int

The starting index if you process multiple files at once. [default: 1]

index_stop: int

The last index if you process multiple files at once. [default: 1]

n_digits: int or None

The number of digits for indexing multiple files. [default: None] None means no leading zeros.

exclude_tags: list of strings

The tags in the EVT file, which should be ignored (e.g. if they cause parse errors)
configure(self)
prepare_blobs(self)

Populate the blobs

extract_header(self)

Create a dictionary with the EVT header information

get_blob(self, index)

Return a blob with the event at the given index

process(self, blob=None)

Pump the next blob to the modules

finish(self)

Clean everything up

class km3pipe.io.DAQPump[source]

Bases:km3pipe.core.Pump

A pump for binary DAQ files.

configure(self)
next_blob(self)

Get the next frame from file

seek_to_frame(self, index)

Move file pointer to the frame with given index.

get_blob(self, index)

Return blob at given index.

determine_frame_positions(self)

Record the file pointer position of each frame

process(self, blob)

Pump the next blob to the modules

finish(self)

Clean everything up

class km3pipe.io.CLBPump[source]

Bases:km3pipe.core.Pump

A pump for binary CLB files.

configure(self)
determine_packet_positions(self)

Record the file pointer position of each frame

seek_to_packet(self, index)

Move file pointer to the packet with given index.

next_blob(self)

Generate next blob in file

get_blob(self, index)

Return blob at given index.

process(self, blob)
finish(self)

Clean everything up

class km3pipe.io.AanetPump[source]

Bases:km3pipe.core.Pump

A pump for binary Aanet files.

Parameters:
filename: str, optional

Name of the file to open. If this parameter is not given, filenames needs to be specified instead.

ignore_hits: bool, optional [default=False]

If true, don’t read our the hits/mchits.

bare: bool, optional [default=False]

Do not create KM3Pipe specific data, just wrap the bare aanet API. This will only give you blob['evt'].
configure(self)
get_blob(self, index)
blob_generator(self)

Create a blob generator.

process(self, blob=None)
class km3pipe.io.EventPump[source]

Bases:km3pipe.core.Pump

A pump for DAQEvents in JPP files.

Parameters:
filename: str

Name of the file to open.
configure(self)
blob_generator(self)
extract_event(self)
process(self, blob)
class km3pipe.io.CHPump[source]

Bases:km3pipe.core.Pump

A pump for ControlHost data.

configure(self)
process(self, blob)

Wait for the next packet and put it in the blob

show_performance_statistics(self)
finish(self)

Clean up the JLigier controlhost connection

next(self)
class km3pipe.io.HDF5Pump[source]

Bases:km3pipe.core.Pump

Read KM3NeT-formatted HDF5 files, event-by-event.

Parameters:
filename: str

From where to read events. Either this OR filenames needs to be defined.

filenames: list_like(str)

Multiple filenames. Either this OR filename needs to be defined.

skip_version_check: bool [default: False]

Don’t check the H5 version. Might lead to unintended consequences.

ignore_hits: bool [default: False]

If True, do not read any hit information.

cut_mask: str

H5 Node path to a boolean cut mask. If specified, use the boolean array found at this node as a mask. False means “skip this event”. Example: cut_mask="/pid/survives_precut"

shuffle: bool, optional [default: False]

Shuffle the group_ids, so that the blobs are mixed up.

shuffle_function: function, optional [default: np.random.shuffle

The function to be used to shuffle the group IDs.

reset_index: bool, optional [default: True]

When shuffle is set to true, reset the group ID - start to count the group_id by 0.
configure(self)
process(self, blob)
get_blob(self, index)
finish(self)
class km3pipe.io.HDF5Sink[source]

Bases:km3pipe.core.Module

Write KM3NeT-formatted HDF5 files, event-by-event.

The data can be a kp.Table, a numpy structured array, a pandas DataFrame, or a simple scalar.

The name of the corresponding H5 table is the decamelised blob-key, so values which are stored in the blob under FooBar will be written to /foo_bar in the HDF5 file.

Parameters:
filename: str, optional [default: ‘dump.h5’]

Where to store the events.

h5file: pytables.File instance, optional [default: None]

Opened file to write to. This is mutually exclusive with filename.

complib : str [default: zlib]

Compression library that should be used. ‘zlib’, ‘lzf’, ‘blosc’ and all other PyTables filters are available.

complevel : int [default: 5]

Compression level.

chunksize : int [optional]

Chunksize that should be used for saving along the first axis of the input array.

flush_frequency: int, optional [default: 500]

The number of iterations to cache tables and arrays before dumping to disk.

pytab_file_args: dict [optional]

pass more arguments to the pytables File init

n_rows_expected = int, optional [default: 10000]
append: bool, optional [default: False]
configure(self)
process(self, blob)
flush(self)

Flush tables and arrays to disk

finish(self)
class km3pipe.io.HDF5MetaData[source]

Bases:km3pipe.core.Module

Metadata to attach to the HDF5 file.

Parameters:
data: dict
configure(self)
class km3pipe.io.PicklePump[source]

Bases:km3pipe.core.Pump

A pump for Python pickles.

configure(self)
get_blob(self, index)
km3pipe.io.get_logger(name)[source]

Helper function to get a logger

km3pipe.io.log[source]
km3pipe.io.GenericPump(filenames, use_jppy=False, name='GenericPump', **kwargs)[source]

A generic pump which utilises the appropriate pump.

km3pipe.io.read_calibration(detx=None, det_id=None, from_file=False, det_id_table=None)[source]

Retrive calibration from file, the DB.