Fourier Feature Network for 3D Vessel Reconstruction from Biplane Angiograms

Sean Wu1, Naoki Kaneko2, David S. Liebeskind2, Fabien Scalzo1,
1Pepperdine University, 2UCLA
Paper Code
Free-viewpoint portraits

Fourier Features Network as follows: (1) We first take a maximum intensity projection to obtain the simulated biplane data. (2) We then extract rows from both images, and by leveraging deep learning, we reconstruct the corresponding 3D slices.

Abstract

3D reconstruction of biplane cerebral angiograms remains a challenging, unsolved research problem due to the loss of depth information and the unknown pixelwise correlation between input images. The occlusions arising from only two views complicate the reconstruction of fine vessel details and the simultaneous addressing of inherent missing information. In this paper, we take an incremental step toward solving this problem by reconstructing the corresponding 2D slice of the cerebral angiogram using biplane 1D image data. We developed a coordinate-based neural network that encodes the 1D image data along with a deterministic Fourier feature mapping from a given input point, resulting in a slice reconstruction that is more spatially accurate. Using only one 1D row of biplane image data, our Fourier feature network reconstructed the corresponding volume slices with a peak signal-to-noise ratio (PSNR) of 26.32±0.36, a structural similarity index measure (SSIM) of 61.38±1.79, a mean squared error (MSE) of 0.0023±0.0002, and a mean absolute error (MAE) of 0.0364±0.0029. Our research has implications for future work aimed at improving backprojection-based reconstruction by first examining individual slices from 1D information as a prerequisite.


Justification of Gaussian Fourier Features

Demonstration: Here is a simple demonstration of why Gaussian Fourier Features with a scaling parameter B of 10.0 is needed. Note: The demonstration below shows a simple network overfitting on one image to show that gaussian fourier features allow learning of higher frequency functions. This gives us justification to leverage the positional encoding in our research.

Explanation: Overfitting a simple NN on one image. See how gaussian fourier features can model higher frequency details better than basic and no positional encoding.

Method Overview

Descriptive Text About the Image

Network Architecture: Detailed visualization of our Fourier feature network, where we show the input channels, output channels, kernel size, padding, and stride for each convolutional layer.

Qualitative Results

Descriptive Text About the Image

Visualization: Comparison of Fourier feature 2D reconstruction with a vanilla 2D decoder and the backprojection method from our previous paper. The green regions represent vessels that were properly reconstructed. The red regions are fine vessels that were not reconstructed by the Fourier feature network.

Related Links

Here are some excellent related researches.

Backprojection-based 3D Reconstruction introduces the concept of reconstructing 3D representation from various multi-view scenarios.

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains Original paper that gives in-depth theory behind why Fourier Features are superior.

2D to 3D reconstruction of biplane angiograms with conditional GAN.

BibTeX

@article{wu2024fourier,
  author={Wu, Sean and Kaneko, Naoki and Liebeskind, David and Scalzo, Fabien},
  title={Fourier Feature Network for 3D Vessel Reconstruction from Biplane Angiograms},
  journal={(In Review) Journal of Machine Vision and Applications}
  year={2024}
}