Light field microscopy (LFM) is a scanning-free 3-dimensional (3D) microscopic imaging method based on the theory of light field. This technique allows sub-second (~10 Hz) large volumetric imaging ([~0.1 to 1 mm]³) with ~1 μm spatial resolution in the condition of weak scattering and semi-transparence, which has never been achieved by other methods. Just as in traditional light field rendering, there are two steps for LFM imaging: light field capture and processing. In most setups, a microlens array is used to capture the light field. As for processing, it can be based on two kinds of representations of light propagation: the ray optics picture^[1] and the wave optics picture.^[2] The Stanford University Computer Graphics Laboratory published their first prototype LFM in 2006^[1] and has been working on the cutting edge since then.

Light field generation

Ray parameterization of the light field microscopy. (A) Light field parameterization without any relay lens. The object plane conjugates with the microlens array plane via the objective, and the objective plane conjugates with the sensor plane via microlenses: the intermediate image of two points is on the microlens array plane, one microlens corresponding to one point; each subimage behind the corresponding microlens incorporates an image of the objective. (B) Light field parameterization with a relay system. The conjugation between points on the focal plane and mirolenses still remains; however, the subimage behind each microlens only incorporates a part of the objective. In both systems, a ray is parameterized as a combination of a 2D coordinate of the microlens through which the ray passes and a 2D coordinate of the subimage pixel on which it falls.

A light field is a collection of all the rays flowing through some free space, where each ray can be parameterized with four variables.^[3] In many cases, two 2D coordinates–denoted as ${\displaystyle (s,t)}$ & ${\displaystyle (u,v)}$–on two parallel planes with which the rays intersect are applied for parameterization. Accordingly, the intensity of the 4D light field can be described as a scalar function: ${\textstyle L_{f}(s,t,u,v)}$, where ${\displaystyle f}$ is the distance between two planes.

LFM can be built upon the traditional setup of a wide-field fluorescence microscope and a standard CCD camera or sCMOS.^[1] A light field is generated by placing a microlens array at the intermediate image plane of the objective (or the rear focal plane of an optional relay lens) and is further captured by placing the camera sensor at the rear focal plane of the microlenses. As a result, the coordinates of the microlenses ${\displaystyle (s,t)}$ conjugate with those on the object plane (if additional relay lenses are added, then on the front focal plane of the objective) ${\displaystyle (s',t')}$; the coordinates of the pixels behind each microlens ${\displaystyle (u,v)}$ conjugate with those on the objective plane ${\displaystyle (u',v')}$. For uniformity and convenience, we shall call the plane ${\displaystyle (s',t')}$ the original focus plane in this article. Correspondingly, ${\displaystyle f}$ is the focal length of the microlenses (i.e., the distance between microlens array plane and the sensor plane).

In addition, the apertures and the focal-length of each lens and the dimensions of the sensor and microlens array should all be properly chosen to ensure that there is neither overlap nor empty areas between adjacent subimages behind the corresponding microlenses.

Realization from the ray optics picture

This section mainly introduces the work of Levoy et al., 2006.^[1]

Tomographic views based on synthetic refocusing

Step 1: Digital refocusing

Digital refocus of light field. Assume the original image has been focused on the plane that conjugates with the microlens array plane, thus the image should be synthesized by summing pixels behind each microlens to proform a digital focusing on this plane. Now, we want to refocus onto another plane whose conjugated plane is αf away from the sensor plane by rendering the rays defined between the microlens array plane and the sensor plane. To get the intensity of each point on the refocus plane, we sum the rays whose reverse extension lines end up at this point. This figure is the demonstration of a 1-dimension synthetic refocus, and the other dimension can be independently refocused in the same mathematical manner. This figure is a modification of Fig. 1 in Ren Ng 2005.^[4]

Synthetic focusing uses the captured light field to compute the photograph focusing at any arbitrary section. By simply summing all the pixels in each subimage behind the microlens (equivalent to collecting all radiation coming from different angles that falls on the same position), the image is focused exactly on the plane that conjugates with the microlens array plane:

${\displaystyle E_{f}(s,t)={1 \over f^{2}}\iint L_{f}(s,t,u,v)\cos ^{4}\phi ~dudv}$,

where ${\displaystyle \phi }$ is the angle between the ray and the normal of the sensor plane, and ${\textstyle \phi =\arctan({\sqrt {u^{2}+v^{2}}}/f)}$ if the origin of the coordinate system of each subimage is located on the principal optic axis of the corresponding microlens. Now, a new function can defined to absorb the effective projection factor ${\displaystyle \cos ^{4}\phi }$ into the light field intensity ${\displaystyle L_{f}}$ and obtain the actual radiance collection of each pixel: ${\displaystyle {\bar {L}}_{f}=L_{f}\cos ^{4}\phi }$.

In order to focus on some other plane besides the front focal plane of the objective, say, the plane whose conjugated plane is ${\displaystyle f'=\alpha f}$ away from the sensor plane, the conjugated plane can be moved from ${\displaystyle f}$ to ${\displaystyle \alpha f}$ and reparameterize its light field back to the original one at ${\displaystyle f}$:

${\displaystyle {\bar {L}}_{\alpha f}(s,t,u,v)={\bar {L}}_{f}(u+(s-u)/\alpha ,v+(t-v)/\alpha ,u,v)}$.

Thereby, the refocused photograph can be computed with the following formula:

${\displaystyle E_{\alpha f}(s,t)={1 \over \alpha ^{2}f^{2}}\iint {\bar {L}}_{f}(u(1-1/\alpha )+s/\alpha ,v(1-1/\alpha )+t/\alpha ,u,v)~dudv}$.

Consequently, a focal stack is generated to recapitulate the instant 3D imaging of the object space. Furthermore, tilted or even curved focal planes are also synthetically possible.^[5] In addition, any reconstructed 2D image focused at an arbitrary depth corresponds to a 2D slice of a 4D light field in the Fourier domain, where the algorithm complexity can be reduced from ${\displaystyle O(n^{4})}$ to ${\displaystyle O(n^{2}\log n)}$.^[4]

Step 2: Point spread function measurement

Due to diffraction and defocus, however, the focal stack ${\displaystyle FS}$ differs from the actual intensity distribution of voxels ${\displaystyle V}$, which is really desired. Instead, ${\displaystyle FS}$ is a convolution of ${\displaystyle V}$ and a point spread function (PSF):

${\displaystyle V*PSF=FS.}$

Thus, the 3D shape of the PSF has to be measured in order to subtract its effect and to obtain voxels' net intensity. This measurement can be easily done by placing a fluorescent bead at the center of the original focus plane and recording its light field, based on which the PSF's 3D shape is ascertained by synthetically focusing on varied depth. Given that the PSF is acquired with the same LFM setup and digital refocusing procedure as the focal stack, this measurement correctly reflects the angular range of rays captured by the objective (including any falloff in intensity); therefore, this synthetic PSF is actually free of noise and aberrations. The shape of the PSF can be considered identical everywhere within our desired field of view (FOV); hence, multiple measurements can be avoided.

Step 3: 3D deconvolution

In the Fourier domain, the actual intensity of voxels has a very simple relation with the focal stack and the PSF:

${\displaystyle {\mathcal {F}}(V)={\frac {{\mathcal {F}}(FS)}{{\mathcal {F}}(PSF)}}}$,

where ${\displaystyle {\mathcal {F}}}$ is the operator of the Fourier transform. However, it may not be possible to directly solve the equation above, given the fact that the aperture is of limited size, resulting in the PSF being bandlimited (i.e., its Fourier transform has zeros). Instead, an iterative algorithm called constrained iterative deconvolution in the spatial domain is much more practical here:^[6]

1. ${\displaystyle \bigtriangleup ^{(k+1)}~\leftarrow ~FS-V^{(k)}*PSF}$;
2. ${\displaystyle V^{(k+1)}~\leftarrow ~\max(V^{(k)}+\bigtriangleup ^{(k+1)},0)}$.

This idea is based on constrained gradient descent: the estimation of ${\displaystyle V}$ is improved iteratively by calculating the difference between the actual focal stack ${\displaystyle FS}$ and the estimated focal stack ${\textstyle V*PSF}$ and correcting ${\displaystyle V}$ with the current difference (${\displaystyle V}$ is constrained to be non-negative).

Fourier Slice Photography

The formula of ${\displaystyle E_{\alpha f}(s,t)}$ can be rewritten by adopting the concept of the Fourier Projection-Slice Theorem.^[7] Because the photography operator ${\displaystyle {\mathcal {P}}_{\alpha }[{\bar {L_{f}}}](s,t)=E_{\alpha f}(s,t)}$ can be viewed as a shear followed by projection, the result should be proportional to a dilated 2D slice of the 4D Fourier transform of a light field. Precisely, a refocused image can be generated from the 4D Fourier spectrum of a light field by extracting an 2D slice, applying an inverse 2D transform, and scaling. Before the proof, we first introduce some operators:

1. Integral Projection Operator: ${\displaystyle {\mathcal {I}}_{M}^{N}[f](x_{1},...,x_{M})=\int {f(x_{1},...,x_{N})dx_{M+1}...dx_{N}}}$
2. Slicing operator: ${\displaystyle {\mathcal {S}}_{M}^{N}[f](x_{1},...,x_{M})=f(x_{1},...,x_{M},0,...,0)}$
3. Photography Change of Basis: Let ${\displaystyle {\mathcal {B}}_{\alpha }}$ denote an operator for a change of basis of an 4-dimensional function so that ${\displaystyle {\mathcal {B}}[f](\mathrm {x} )=f({\mathcal {B}}^{-1}\mathrm {x} )}$, with ${\displaystyle {\mathcal {B}}_{\alpha }={\begin{bmatrix}\alpha &0&1-\alpha &0\\0&\alpha &0&1-\alpha \\0&0&1&0\\0&0&0&1\end{bmatrix}}}$.
4. Fourier Transform Operator: Let ${\displaystyle {\mathcal {F}}^{N}}$ denote the N-dimensional Fourier transform operator.

By these definitions, we can rewrite ${\displaystyle {\mathcal {P}}_{\alpha }[{\bar {L_{f}}}](s,t)={1 \over \alpha ^{2}f^{2}}\iint {\bar {L}}_{f}(u(1-1/\alpha )+s/\alpha ,v(1-1/\alpha )+t/\alpha ,u,v)~dudv\equiv {\frac {1}{\alpha ^{2}f^{2}}}{\mathcal {I}}_{2}^{4}\circ {\mathcal {B}}_{\alpha }[L_{f}]}$.

According to the generalized Fourier-slice theorem,^[7] we have

${\displaystyle {\mathcal {F}}^{M}\circ {\mathcal {I}}_{M}^{N}\circ {\mathcal {B}}\equiv {\mathcal {S}}_{M}^{N}\circ {\frac {{\mathcal {B}}^{-T}}{|{\mathcal {B}}^{-T}|}}\circ {\mathcal {F}}^{N}}$,

and hence the photography operator has the form

${\displaystyle {\mathcal {P}}_{\alpha }[{\bar {L_{f}}}](s,t)\equiv {\frac {1}{f^{2}}}{\mathcal {F}}^{-2}\circ {\mathcal {S}}_{2}^{4}\circ {\mathcal {B}}_{\alpha }^{-T}\circ {\mathcal {F}}^{4}}$.

According to the formula, we know a photograph is the inverse 2D Fourier transform of a dilated 2D slice in the 4D Fourier transform of the light field.

Discrete Fourier Slice Photography

If all we have available are samples of the light field, instead of use Fourier slice theorem for continuous signal mentioned above, we adopt discrete Fourier slice theorem, which is a generalization of the discrete Radon transform, to compute refocused image.^[8]

Assume that a lightfield ${\displaystyle {\bar {L}}_{f}}$ is periodic with periods ${\displaystyle T_{s},T_{t},T_{u},T_{v}}$ and is defined on the hypercube ${\displaystyle H=[-T_{s}/2,T_{s}/2]\times [-T_{t}/2,T_{t}/2]\times [-T_{u}/2,T_{u}/2]\times [-T_{v}/2,T_{v}/2]}$. Also, assume there are ${\displaystyle N_{s}\times N_{t}\times N_{u}\times N_{v}}$ known samples of the light field ${\displaystyle ({\hat {s}}\Delta s,{\hat {t}}\Delta t,{\hat {u}}\Delta u,{\hat {v}}\Delta v)}$, where ${\displaystyle {\hat {s}},{\hat {t}},{\hat {u}},{\hat {v}}\in \mathbb {Z} }$ and ${\displaystyle \Delta s,\Delta t,\Delta u,\Delta v={\frac {T_{s}}{N_{s}}},{\frac {T_{t}}{N_{t}}},{\frac {T_{u}}{N_{u}}},{\frac {T_{v}}{N_{v}}}}$, respectively. Then, we can define ${\displaystyle {\bar {L}}_{f}^{d}}$ using trigonometric interpolation with these sample points:

${\displaystyle {\bar {L}}_{f}^{d}({\hat {s}},{\hat {t}},{\hat {u}},{\hat {v}})=\sum _{\omega _{\hat {s}}}\sum _{\omega _{\hat {t}}}\sum _{\omega _{\hat {u}}}\sum _{\omega _{\hat {v}}}{\mathcal {\bar {L}}}_{f}^{d}(\omega _{\hat {s}},\omega _{\hat {t}},\omega _{\hat {u}},\omega _{\hat {v}})e^{2\pi i({\hat {s}}\omega _{\hat {s}}+{\hat {t}}\omega _{\hat {t}}+{\hat {u}}\omega _{\hat {u}}+{\hat {v}}\omega _{\hat {v}})}}$,

where

${\displaystyle {\mathcal {\bar {L}}}_{f}^{d}(\omega _{\hat {s}},\omega _{\hat {t}},\omega _{\hat {u}},\omega _{\hat {v}})=\sum _{\hat {s}}\sum _{\hat {t}}\sum _{\hat {u}}\sum _{\hat {v}}{\bar {L}}_{f}^{d}({\hat {s}},{\hat {t}},{\hat {u}},{\hat {v}})e^{2\pi i({\hat {s}}\omega _{\hat {s}}+{\hat {t}}\omega _{\hat {t}}+{\hat {u}}\omega _{\hat {u}}+{\hat {v}}\omega _{\hat {v}})}}$.

Note that the constant factors are dropped for simplicity.

To compute its refocused photograph, we replace infinite integral in the formula of ${\displaystyle {\mathcal {P}}_{\alpha }}$ with summation whose bounds are ${\displaystyle [-T_{u}/2,T_{u}/2]}$ and ${\displaystyle [-T_{v}/2,T_{v}/2]}$. That is,

${\displaystyle {\mathcal {P}}_{\alpha }^{d}[{\bar {L_{f}^{d}}}]({\hat {s}},{\hat {t}})={1 \over \alpha ^{2}f^{2}}\sum _{-N_{u}/2}^{N_{u}/2}\sum _{-N_{v}/2}^{N_{v}/2}{\bar {L}}_{f}^{d}({\hat {u}}\Delta u(1-1/\alpha )+{\hat {s}}\Delta s/\alpha ,{\hat {v}}\Delta v(1-1/\alpha )+{\hat {t}}\Delta t/\alpha ,{\hat {u}}\Delta u,{\hat {v}}\Delta v)}$.

Then, by discrete Fourier slice theorem indicates, we can represent the photograph using Fourier slice:

${\displaystyle {\mathcal {P}}_{\alpha }^{d}[{\bar {L_{f}^{d}}}]({\hat {s}},{\hat {t}})=\sum _{-N_{u}/2}^{N_{u}/2}\sum _{-N_{v}/2}^{N_{v}/2}{e^{2\pi i({\hat {s}}\omega _{\hat {s}}+{\hat {t}}\omega _{\hat {t}})}{\mathcal {\bar {L}}}_{f}^{d}}{(\alpha \omega _{\hat {s}},\alpha \omega _{\hat {t}},(1-\alpha )\omega _{\hat {s}},(1-\alpha )\omega _{\hat {t}})}}$

Realization from the wave optics picture

Although ray-optics based plenoptic camera has demonstrated favorable performance in the macroscopic world, diffraction places a limit on the LFM reconstruction when staying with ray-optics parlance. Hence, it may be much more convenient to switch to wave optics. (This section mainly introduce the work of Broxton et al., 2013.^[2])

${\displaystyle \mathbf {f} =\mathrm {H} ~\mathbf {g} .}$

Applications

See also

• Light field
• Microlens
• Tomography
• Aperture synthesis
• Voxel

References