SPyRiT: a single-pixel image reconstruction toolkit based on PyTorch

SPyRiT is a PyTorch-based deep image reconstruction package primarily designed for single-pixel imaging. SPyRiT is modular and may be useful for other linear inverse problems.

Single-pixel imaging

Single-pixel imaging allows high-quality images to be provided by a device that is only equipped with a single point detector. It is based on the acquisition of the inner products of a scene with some light patterns

\[y_k = p_k^{\top}x,\]

where \(x \in \mathbb{R}^{N}\) represents the image of the scene and \(p_k \in \mathbb{R}^N\) represents the \(k\)-th light pattern displayed on the spatial light modulator. Next, the image needs to be reconstructed from the sequence of measurements obtained with \(K\) different light patterns. The case \(K < N\) is often considered to reduce the total acquisition time.

SPyRiT’s full network

_images/full.png

SPyRiT’s full network

SPyRiT allows to simulate measurements and perform image reconstruction using a full network. A full network is built using a measurement operator \(\mathcal{P}\), a noise operator \(\mathcal{N}\), a preprocessing operator \(\mathcal{B}\), a reconstruction operator \(\mathcal{R}\), and a learnable neural network \(\mathcal{G}_{\theta}\). All operators inherit from PyTorch’s torch.nn.Module class (see here), which allows them to be easily combined into a full network.

Simulation of the measurements

We implement the acquisition as

\(y = (\mathcal{N} \circ \mathcal{P})(x),\)

where \(\mathcal{P}\) is a linear operator that models the light patterns, \(\mathcal{N}\) is a noise operator, and \(\circ\) denotes the composition.

Image reconstruction

Learning-based reconstruction approaches estimate the unknown image as \(x^* = \mathcal{I}_\theta(y)\), where \(\theta\) represents the learnable parameters of the inversion model \(\mathcal{I}_\theta\).

A typical inversion operator \(\mathcal{I}_\theta\) can be written as

\(\mathcal{I}_\theta = \mathcal{G}_\theta \circ \mathcal{R} \circ \mathcal{B},\)

where \(\mathcal{B}\) is a preprocessing operator, \(\mathcal{R}\) is a linear reconstruction operator, and \(\mathcal{G}_\theta\) is a trainable neural network or any available image-domain denoiser.

Learning phase

In the case of supervised learning, the training phase solves

\(\min_{\theta}{\sum_i \mathcal{L}\left(x^{(i)},\mathcal{I}_\theta(y^{(i)})\right)},\)

where \(\mathcal{L}\) is the training loss, and \(\{x^{(i)},y^{(i)}\}_i\) is a set of training pairs.

By introducing the full network \(F_{\theta}(x) = (\mathcal{G}_\theta \circ \mathcal{R} \circ \mathcal{B} \circ \mathcal{N} \circ \mathcal{P})(x)\), the training phase relies on a database containing images only

\(\min_{\theta}{\sum_i \mathcal{L}\left(x^{(i)},\mathcal{F}_\theta(x^{(i)})\right)}.\)

The full network allows noisy data to be simulated on the fly, providing data augmentation while avoiding storing the measurements.

Getting started

Installation

The SPyRiT package is available for Linux, MacOs and Windows:

pip install spyrit

Advanced installation guidelines are available on GitHub. Check out our available tutorials to get started with SPyRiT.

Package structure

The main functionalities of SPyRiT are implemented in the subpackage spyrit.core , which contains 8 submodules:

  1. Measurement operators (spyrit.core.meas) compute linear measurements \(\bar{y} = \mathcal{P}x\).

  2. Noise operators (spyrit.core.noise) corrupt measurements \(y=\mathcal{N}(\bar{y})\) with noise.

  3. Preprocessing operators (spyrit.core.prep) are used to process noisy measurements, \(m=\mathcal{B}(y)\) , before reconstruction. They typically compensate for the image normalization previously performed.

  4. Reconstruction operators (spyrit.core.recon) comprise both standard linear reconstruction operators \(\mathcal{R}\) and full network definitions \(\mathcal{F}_\theta\).

  5. Neural networks (spyrit.core.nnet) include well-known neural networks \(\mathcal{G_{\theta}}\), generally used as denoiser layers.

  6. Training (spyrit.core.train) provide the functionalities for training reconstruction networks.

  7. Warping (spyrit.core.warp) contains the warping operators that are used to simulate moving objects.

  8. Torch utilities (spyrit.core.torch) contains utility functions for PyTorch that are used throughout the package.

In addition, the subpackage spyrit.misc contains various utility functions for Numpy / PyTorch that can be used independently of the core functionalities.

spyrit.core

Core module for Spyrit package, containing the main classes and functions.

spyrit.misc

Contains miscellaneous Numpy / Pytorch functions useful for spyrit.core.

Cite us

When using SPyRiT in scientific publications, please cite the following paper:

    1. Beneti-Martin, L Mahieu-Williame, T Baudier, N Ducros, “OpenSpyrit: an Ecosystem for Reproducible Single-Pixel Hyperspectral Imaging,” Optics Express, Vol. 31, No. 10, (2023). DOI.

When using SPyRiT specifically for the denoised completion network, please cite the following paper:

  • A Lorente Mur, P Leclerc, F Peyrin, and N Ducros, “Single-pixel image reconstruction from experimental data using neural networks,” Opt. Express 29, 17097-17110 (2021). DOI.

Join the project

Feel free to contact us by e-mail for any question. Active developers are currently Nicolas Ducros, Thomas Baudier, Juan Abascal and Romain Phan. Direct contributions via pull requests (PRs) are welcome.

The full list of contributors can be found here.