Warning

This is A PREVIEW for NEST 3.0 and NOT an OFFICIAL RELEASE! Some functionality may not be available and information may be incomplete!

Recording from simulations

Recording devices (or recorders, in short) are used to sample or collect observable quantities like potentials, conductances, or spikes from neurons and synapses.

To determine what happens to recorded data, each recording device can specify a recording backend in its record_to property. The default backend is memory, which just stores the recorded data in memory for later retrieval. Other backends write the data to file or to the screen. The different backends and their usage is explained in detail in the section about Recording Backends.

Recording devices can fundamentally be subdivided into two groups:

  • Collectors collect events sent to them. Neurons are connected to collectors and the collector collects the events emitted by the neurons connected to it.

  • Samplers actively interrogate their targets at given time intervals. This means that the sampler must be connected to the neuron (not the neuron to the sampler), and that the neuron must support the particular type of sampling.

Recording devices can only reliably record data that was generated during the previous simulation time step interval. See the guide about running simulations for details about the temporal aspects of the simulation loop.

Note

Due to the need for internal buffering and the unpredictable order of thread execution, events are not necessarily recorded in chronological order.

The time span in which a recorder is actually recording, can be specified using the additional properties start and stop. These define activation and inactivation times of the device in ms. An additional property origin is available to shift the recording window by a certain amount, which can be useful in simulation protocols with repeated simulations. The following rules apply:

  • Collectors collect all events with timestamps T that fulfill

    start < T <= stop.
    

    Events with timestamp T == start are not recorded.

  • Sampling devices sample at times t = nh (h being the simulation resolution) with

    start < t <= stop
    (t-start) mod interval == 0
    

Common recorder properties

All recorders have a set of common properties that can be set using SetDefaults on the model class or SetStatus on a device instance:

label

A string (default: “”) specifying an arbitrary textual label for the device. Recording backends might use the label to generate device specific identifiers like filenames and such.

n_events

The number of events that were collected by the recorder can be read out of the n_events entry. The number of events can be reset to 0. Other values cannot be set.

origin

A positive floating point number (default : 0.0) used as the reference time for start and stop.

record_to

A string (default: “memory”) containing the name of the recording backend where to write data to. An empty string turns all recording of individual events off.

start

A positive floating point number (default: 0.0) specifying the activation time in ms, relative to origin.

stop

A floating point number (default: infinity) specifying the deactication time in ms, relative to origin. The value of stop must be greater than or equal to start

Recorders for every-day situations

multimeter – Sampling continuous quantities from neurons

Most sampling use cases are covered by the multimeter, which allows to record analog values from neurons. Models which have such values expose a recordables property that lists all recordable quantities. This property can be inspected using GetDefaults on the model class or GetStatus on a model instance. It cannot be changed by the user.

>>> nest.GetDefaults('iaf_cond_alpha')['recordables']
['g_ex', 'g_in', 't_ref_remaining', 'V_m']

The record_from property of a multimeter (a list, empty by default) can be set to contain the name(s) of one or more of these recordables to have them sampled during simulation.

mm = nest.Create('multimeter', 1, {'record_from': ['V_m', 'g_ex']})

The sampling interval for recordings (given in ms) can be controlled using the multimeter parameter interval. The default value of 1.0 ms can be changed by supplying a new value either in the call to Create or by using SetStatus on the model instance.

nest.SetStatus(mm, 'interval': 0.1})

The recording interval must be greater than or equal to the simulation resolution, which defaults to 0.1 ms.

Warning

The set of variables to record from and the recording interval must be set before the multimeter is connected to any neuron. These properties cannot be changed afterwards.

After configuration, a multimeter can be connected to the neurons it should record from by using the standard Connect routine.

neurons = nest.Create('iaf_psc_alpha', 100)
nest.Connect(mm, neurons)

To learn more about possible connection patterns and additional options when using Connect, see the guide on connection management.

The above call to Connect would fail if the neurons would not support the sampling of the values V_m and g_ex. It would also fail if carried out in the wrong direction, i.e., trying to connect the neurons to mm.

Note

A pre-configured multimeter is available under the name voltmeter. Its record_from property is already set to record the variable V_m from the neurons it is connected to.

spike_recorder – Collecting spikes from neurons

The most universal collector device is the spike_recorder, which collects and records all spikes it receives from neurons that are connected to it. Each spike received by the spike recorder is immediately handed over to the selected recording backend for further processing.

Any node from which spikes are to be recorded, must be connected to the spike recorder using the standard Connect command. The connection weights and delays are ignored by the spike recorder, which means that the spike recorder records the time of spike creation rather than that of their arrival.

>>> neurons = nest.Create('iaf_psc_alpha', 5)
>>> sr = nest.Create('spike_recorder')
>>> nest.Connect(neurons, sr)

The call to Connect will fail if the connection direction is reversed (i.e., connecting sr to neurons).

weight_recorder – Recording weights from synapses

The change in synaptic weights over time is a key observable property in studies of plasticity in neuronal network models. To access this information, the weight_recorder can be used. In contrast to other recording devices, which are connected to a specific set of neurons, the weight recorder is instead set as a parameter in the synapse model.

After assigning an instance of a weight recorder to the synapse model by setting its weight_recorder property, the weight recorder collects the global IDs of source and target neurons together with the weight for each spike event that travels through the observed synapses.

To only record from a subset of connected synapses, the weight recorder accepts NodeCollections in the parameters senders and targets. If set, they restrict the recording of data to only synapses that fulfill the given criteria.

>>> wr = nest.Create('weight_recorder')
>>> nest.CopyModel("stdp_synapse", "stdp_synapse_rec", {"weight_recorder": wr})

>>> pre = nest.Create("iaf_psc_alpha", 10)
>>> post = nest.Create("iaf_psc_alpha", 10)

>>> nest.Connect(pre, post, syn_spec="stdp_synapse_rec")

Where does data end up?

After a recording device has sampled or collected data, the recording backends are responsible for how the data are processed.

Theoretically, recording backends are not restricted in what they do with the data. The ones included in NEST can collect data in memory, display it on the terminal, or write it to files.

To specify the recording backend for a given recording device, the property record_to of the latter has to be set to the name of the recording backend to be used. This can either happen already in the call to Create or by using SetStatus on the model instance.

sr = nest.Create('spike_recorder', params={'record_to': 'ascii'})

Storing data in memory using the memory backend is the default for all recording devices as this does not require any additional setup of data paths or filesystem permissions and allows a convenient readout of data by the user after simulation.

Each recording backend may provide a specific set of parameters (explained in the backend documentation below) that will be included in the model status dictionary once the backend is set. This means that these parameters can only be reviewed and changed after the backend has been selected. In particular, recording-device specific per-device parameters cannot be set using SetDefaults, but must rather be supplied either in the call to Create or set on an instance using SetStatus.

Note

Even though parameters of different recording backends may have the same name, they are separate entities internally. This means that a value that was set for a parameter of a recording device when a specific backend was selected has to be set again on the new backend, if the backend is changed later on.

The full list of available recording backends and their respective properties can be obtained from the kernel’s status dictionary.

nest.GetKernelStatus("recording_backends")
{u'ascii': {},
 u'memory': {},
 u'screen': {},
 u'sionlib': {u'buffer_size': 1024,
  u'filename': u'',
  u'sion_chunksize': 262144,
  u'sion_collective': False,
  u'sion_n_files': 1}}

The example shows that only the sionlib backend has backend-specific global properties, which can be modified by supplying a nested dictionary to SetKernelStatus.

nest.SetKernelStatus({"recording_backends": {'sionlib': {'buffer_size': 512}}})

Store data in main memory

When a recording device sends data to the memory backend, it is stored internally in efficient vectors. These vectors are made available to the user level in the device’s status dictionary under the key events.

The events dictionary always contains the global IDs of the source nodes of the recorded data in the field sender. It also always contains the time of the recording. Depending on the setting of the property time_in_steps, this time can be stored in two different formats:

  • If time_in_steps is false (which is the default), the time is stored as a single floating point number in the field times, interpreted as the simulation time in ms

  • If time_in_steps is true, the time is stored as a pair consisting of the integer number of simulation time steps in units of the simulation resolution in times and the negative offset from the next such grid point as a floating point number in ms in offset.

All additional data collected or sampled by the recording device is contained in the events dictionary in arrays. These data are named based on the recordable they came from and with the appropriate data type (either integer or floating point).

The number of events that were collected by the memory backend can be read out of the n_events entry in the status dictionary of the recording device. To delete data from memory, n_events can be set to 0. Other values cannot be set.

Parameter summary

events

A dictionary containing the recorded data in the form of one numeric array for each quantity measured. It always has the sender global IDs of recorded events under the key senders and the time of the recording, the format of which depends on the setting of time_in_steps.

n_events

The number of events collected or sampled since the last reset of n_events. By setting n_events to 0, all events recorded so far will be discarded from memory.

time_in_steps

A Boolean (default: false) specifying whether to store time in steps, i.e., in integer multiples of the simulation resolution (under the key times of the events dictionary) plus a floating point number for the negative offset from the next grid point in ms (under key offset), or just the simulation time in ms under key times. This property cannot be set after Simulate has been called.

Write data to plain text files

The ascii recording backend writes collected data persistently to a plain text ASCII file. It can be used for small to medium sized simulations, where the ease of a simple data format outweighs the benefits of high-performance output operations.

This backend will open one file per recording device per thread on each MPI process. This can cause a high load on the file system in large simulations. This backend can become prohibitively inefficient, particularly on machines with distributed filesystems. In case you experience such scaling problems, the recording backend for SIONlib may be a possible alternative.

Filenames of data files are determined according to the following pattern:

data_path/data_prefix(label|model_name)-node_id-vp.file_extension

The properties data_path and data_prefix are global kernel properties. They can, for example, be set during repetitive simulation protocols to separate the data originating from individual runs. The label replaces the model name component if it is set to a non-empty string. node_id and vp denote the zero-padded global ID and virtual process of the recorder writing the file. The filename ends in a dot and the file_extension.

The life of a file starts with the call to Prepare and ends with the call to Cleanup. Data that is produced during successive calls to Run in between a pair of Prepare and Cleanup calls will be written to the same file, while the call to Run will flush all data to the file, so it is available for immediate inspection.

When creating a new recording, if the file name already exists, the Prepare call will fail with a corresponding error message. To instead overwrite the old file, the kernel property overwrite_files can be set to true using SetKernelStatus. An alternative way for avoiding name clashes is to re-set the kernel properties data_path or data_prefix, so that another filename is chosen.

Data format

Any file written by the ascii recording backend starts with an informational header. The first header line contains the NEST version, with which the file was created, followed by the version of the recording backend in the second. The third line describes the data by means of the field names for the different columns. All lines of the header start with a # character.

The first field of each record written is the node ID of the neuron the event originated from, i.e., the source of the event. This is followed by the time of the measurement, the recorded floating point values and the recorded integer values.

The format of the time field depends on the value of the property time_in_steps. If set to false (which is the default), time is written as a single floating point number representing the simulation time in ms. If time_in_steps is true, the time of the event is written as a pair of values consisting of the integer simulation time step in units of the simulation resolution and the negative floating point offset in ms from the next integer grid point.

Note

The number of decimal places for all decimal numbers written can be controlled using the recorder property precision.

Parameter summary

file_extension

A string (default: “dat”) that specifies the file name extension, without leading dot. The generic default was chosen, because the exact type of data cannot be known a priori.

filenames

A list of the filenames where data is recorded to. This list has one entry per local thread and is a read-only property.

label

A string (default: “”) that replaces the model name component in the filename if it is set.

precision

An integer (default: 3) that controls the number of decimal places used to write decimal numbers to the output file.

time_in_steps

A Boolean (default: false) specifying whether to write time in steps, i.e., in integer multiples of the simulation resolution plus a floating point number for the negative offset from the next grid point in ms, or just the simulation time in ms. This property cannot be set after Simulate has been called.

Write data to the terminal

When initially conceiving and debugging simulations, it can be useful to check recordings in a more ad hoc fashion. The recording backend screen can be used to dump all recorded data onto the console for quick inspection.

The first field of each record written is the node ID of the neuron the event originated from, i.e., the source of the event. This is followed by the time of the measurement, the recorded floating point values, and the recorded integer values.

The format of the time field depends on the value of the property time_in_steps. If set to false (which is the default), time is written as one floating point number representing the simulation time in ms. If time_in_steps is true, the time of the event is written as a value pair consisting of the integer simulation time step and the floating point offset in ms from the next grid point.

Note

Using this backend for production runs is not recommended, as it may produce huge amounts of console output and considerably slow down the simulation.

Parameter summary

precision

controls the number of decimal places used to write decimal numbers to the terminal.

time_in_step

A boolean (default: false) specifying whether to print time in steps, i.e., in integer multiples of the resolution and an offset, rather than just in ms.

Store data to an efficient binary format

Availability

This recording backend is only available if NEST was compiled with support for MPI and SIONlib.

The sionlib recording backend writes collected data persistently to a binary container file (or to a rather small set of such files). This is especially useful for large-scale simulations running in a distributed way on many MPI processes/OpenMP threads. In such usage scenarios, writing to plain text files (see recording backend for ASCII files) would cause a large overhead because of the huge number of generated files and thus be very inefficient.

The implementation of the sionlib backend is based on the SIONlib library. Depending on the I/O architecture of the compute cluster or supercomputer and the global settings of the sionlib recording backend, either a single container file or a set of these files is created. In case of a single file, it is named according to the following pattern:

data_path/data_prefixfilename

In case of multiple files, this name is extended for each file by a dot followed by a consecutive number. The properties data_path and data_prefix are global kernel properties. They can for example be set during repetitive simulation protocols to separate the data originating from individual runs.

The life of a set of associated container files starts with the call to Prepare and ends with the call to Cleanup. Data that is produced during successive calls to Run in between a pair of Prepare and Cleanup calls will be written to the same file set. When creating a new recording, if the filename already exists, the Prepare call will fail with a corresponding error message. To instead overwrite the old file set, the kernel property overwrite_files can be set to true using SetKernelStatus. An alternative way for avoiding name clashes is to re-set the kernel properties data_path or data_prefix, so that another full filename is composed.

Data format

In contrast to other recording backends, the sionlib backend writes the data from all recorders using it to a single container file(s). The file(s) contain the data in a custom binary format, which is composed of a series of blocks in the following order:

  • The body block contains the actual data records; the layout of an individual record depends on the type of the device and is described by a corresponding entry in the device info block

  • The file info block keeps the file’s metadata, like version information and such

  • The device info block stores the properties and a data layout description for each device that uses the sionlib backend

  • The tail block contains pointers to the file info block

The data layout of the NEST SIONlib file format v2 is shown in the following figure.

NEST SIONlib binary file format

Figure 18 NEST SIONlib binary file format.

Reading the data

As the binary format of the files produced by the sionlib does not conform to any standard, parsing them manually might be a bit cumbersome. To ease this task, we provide a reader module for Python that makes the files available in a convenient way. The source code and further documentation for this module can be found in its own repository.

Recorder-specific parameters

label

A recorder-specific string (default: “”) that serves as alias name for the recording device, and which is stored in the metadata section of the container files.

Global parameters (to be set via SetKernelStatus)

filename

The filename (default: “output.sion”) part of the pattern according to which the full filename (incl. path) is generated (see above).

sion_n_files

The number of container files (default: 1) used for storing the results of a single call to Simulate (or of a single Prepare-Run-Cleanup cycle). The default is one file. Using multiple files may have a performance advantage on large computing clusters, depending on how the (parallel) file system is accessed from the compute nodes.

sion_chunksize

In SIONlib nomenclature, a single OpenMP thread running on a single MPI process is called a task. For each task, a specific number of bytes is allocated in the container file(s) from the beginning. This number is set by the parameter sion_chunksize (default: 262144). If the number of bytes written by each task during the simulation is known in advance, it is advantageous to set the chunk size to this value. In this way, the size of the container files has not to be adjusted by SIONlib during the simulation. This yields a slight performance advantage. Choosing a value for sion_chunksize which is too large does not hurt that much because SIONlib container files are sparse files (if supported by the underlying file system) which only use up the disk space which is actually required by the stored data.

buffer_size

The size of task-specific buffers (default: 1024) within the sionlib recording backend in bytes. These buffers are used to temporarily store data generated by the recording devices on each task. As soon as a buffer is full, its contents are written to the respective container file. To achieve optimum performance, the size of these buffers should at least amount to the size of the file system blocks.

sion_collective

Flag (default: false) to enable the collective mode of SIONlib. In collective mode, recorded data is buffered completely during Run and only written at the very end of Run to the container files, all tasks acting synchronously. Furthermore, within SIONlib so-called collectors aggregate data from a specific number of tasks, and actually only these collectors directly access the container files, in this way minimizing load on the file system. The number of tasks per collector is determined automatically by SIONlib. However, collector size can also be set explicitly by the user via the environment variable SION_COLLSIZE before the start of NEST. On large simulations which also generate a large amount of data, collective mode can offer a performance advantage.