SPyRiT’s documentation
SPyRiT is a PyTorch-based deep image reconstruction package primarily designed for single-pixel imaging.
SPyRiT allows to simulate measurements and perform image reconstruction using a full network structure. It takes a normalized image as input and performs data simulation and image reconstruction in a single forward pass or in separate steps. A full network generally consists of a measurement operator, a noise operator, a preprocessing operator, a reconstruction operator, and a learnable neural network. All operators inherit from PyTorch’s nn.Module class, which allows them to be easily combined into a full network.
The complete network contains two main parts: a physics simulation part that simulates measurements from images, and a reconstruction part that estimates the unknown image from measurements.
The Physics Simulation part consists of a Measurement operator (\(\mathcal{N}\)) and a Noise operator (\(\mathcal{P}\)).
The reconstruction part consists of a preprocessing (\(\mathcal{B}\)) that produces the pre-processed measurements from the noisy measurements, a reconstruction operator (\(\mathcal{R}\)) that estimates the unknown image from the pre-processed measurements, and an optional neural network (\(\mathcal{G_{\theta}}\)) that can be trained to improve the reconstruction quality.
Installation
The spyrit package is available for Linux, MacOs and Windows:
pip install spyrit
Advanced installation guidelines are available on GitHub.
Single-pixel imaging
Modelling of the measurements
Single-pixel imaging aims to recover an image \(x\in\Re^N\) from a few noisy scalar products \(y\in\Re^M\), where \(M\ll N\). We model the acquisition as
\(y = (\mathcal{N} \circ \mathcal{P})(x),\)
where \(\mathcal{P}\) is a linear operator, \(\mathcal{N}\) is a noise operator, and \(\circ\) denotes the composition of operators.
Image reconstruction
Learning-based reconstruction approaches estimate the unknown image as \(x^* = \mathcal{I}_\theta(y)\), where \(\mathcal{I}_\theta\) represents the parameters that are learned during a training phase. In the case of supervised learning, the training phase solves
\(\min_{\theta}{\sum_i \mathcal{L}\left(x_i,\mathcal{I}_\theta(y_i)\right)},\)
where \(\mathcal{L}\) is the training loss between the true image \(x\) and its estimate, and \(\{x_i,y_i\}_i\) is a set of training pairs.
Consider the typical reconstruction operator \(\mathcal{I}_\theta\) which can be written as:
\(\mathcal{I}_\theta = \mathcal{G}_\theta \circ \mathcal{R} \circ \mathcal{B},\)
where \(\mathcal{B}\) is a preprocessing operator, \(\mathcal{R}\) is a (standard) linear reconstruction operator, and \(\mathcal{G}_\theta\) is a neural network that can be trained during the training phase. Alternatively, \(\mathcal{R}\) can be simply “plugged”. In this case, it is trained beforehand.
To introduce the full network, a forward pass can be written as follows:
\(F_{\theta}(x) = (\mathcal{G}_\theta \circ \mathcal{R} \circ \mathcal{B} \circ \mathcal{N} \circ \mathcal{P})(x).\)
The full network can be trained using a database containing only images:
\(\min_{\theta}{\sum_i \mathcal{L}\left(x_i,\mathcal{F}_\theta(x_i)\right)}.\)
This pipeline allows noisy data to be simulated on the fly, providing data augmentation while avoiding storing the measurements.
Package structure
The main functionalities of SPyRiT are implemented in the subpackage
spyrit.core , which contains six submodules:
Measurement operators (meas) compute linear measurements \(\mathcal{P}x\) from images \(x\), where \(\mathcal{P}\) is a linear operator (matrix) and \(x\) is a vectorized image (see
spyrit.core.meas).Noise operators (noise) corrupt measurements \(y=(\mathcal{N}\circ\mathcal{P})(x)\) with noise (see
spyrit.core.noise).Preprocessing operators (prep) are used to process noisy measurements, \(m=\mathcal{B}(y)\) , prior to reconstruction. They typically compensate for the image normalization previously performed (see
spyrit.core.prep).Reconstruction operators (recon) comprise both standard linear reconstruction operators \(\mathcal{R}\) and full network definitions \(\mathcal{F}_\theta\), which include both forward and reconstruction layers (see
spyrit.core.recon).Neural networks (nnet) include well-known neural networks \(\mathcal{G_{\theta}}\), generally used as denoiser layers (see
spyrit.core.nnet).Training (train) provide the functionalities for training reconstruction networks (see
spyrit.core.train).
Subpackages
Tutorials
Cite us
When using SPyRiT in scientific publications, please cite the following paper:
Beneti-Martin, L Mahieu-Williame, T Baudier, N Ducros, “OpenSpyrit: an Ecosystem for Reproducible Single-Pixel Hyperspectral Imaging,” Optics Express, Vol. 31, No. 10, (2023). DOI.
When using SPyRiT specifically for the denoised completion network, please cite the following paper:
A Lorente Mur, P Leclerc, F Peyrin, and N Ducros, “Single-pixel image reconstruction from experimental data using neural networks,” Opt. Express 29, 17097-17110 (2021). DOI.
Join the project
Feel free to contact us by e-mail <mailto:nicolas.ducros@creatis.insa-lyon.fr> for any question. Active developers are currently Nicolas Ducros, Thomas Baudier, Juan Abascal and Romain Phan. Direct contributions via pull requests (PRs) are welcome.
The full list of contributors can be found here.