Project

Plenoptima

In medical image analysis (MI), achieving high-resolution detail is crucial for accurate diagnoses, but remains restricted by extended acquisition times and the need for high frame rates. Existing deep learning-based superresolution (SR) techniques, which often employ simplified degradation models (e.g., Additive White Gaussian Noise (AWGN)), fail to capture the complex, modality-dependent noise characteristics encountered in clinical settings. To address these limitations, we introduce a Neural Parametric Steered Mixture of Experts (N-SMoE) framework, trained within a Generative Adversarial Network (GAN) paradigm, and incorporating an advanced Stochastic Degradation Model (SDM). The SDM applies diverse levels of blurring and noise to downsampled images, thus simulating realistic clinical conditions and mitigating training instability. In contrast to conventional bilinear or bicubic interpolation approaches, the N-SMoE encoder leverages a lightweight, multilayer Laplacian resizer with bandpass filtering capabilities, while a multi-head attention module isolates high-frequency structural patterns to estimate Gaussian primitives for the decoder. The SMoE decoder, unlike conventional deep learning (DL) black-box models, utilizes gating functions and two-dimensional kernels to produce an edge-aware representation capable of modeling both continuous and smooth transitions, which makes it an excellent tool for SR-related restoration tasks. The resulting proposed neural parametric autoregressive framework enhances interpretability and surpasses state-of-the-art (SotA) methods in terms of fidelity, perceptual quality, and subjective evaluations across multiple medical imaging modalities.

Project Details

Project term

April 4, 2024–December 31, 2024

Affiliations

Technical University of Berlin

Institute

Communication Systems Group

Principal Investigator

Prof. Dr. Thomas Sikora

Methods

We proposed Neural parametric Steered Mixture of Experts regression framework which uses Generative Adversarial Network training paradigm for Advanced imaging modalities, including magnetic resonance imaging (MRI), computed tomography (CT), optical coherence tomography (OCT), and ultrasound.

Results

Our proposed approach achieved state of the art performance, and we submitted it to Elsevier Signal Processing Journal.

Discussion

The proposed method yields promising and convincing results; however, it requires a significant amount of computational resources. Our current focus is on developing an environmentally friendly approach.

Additional Project Information

DFG classification: 408 Electrical Engineering and Information Technology
Software: PyTorch, CUDA
Cluster: CLAIX

Publications

Thesis
Aytaç Özkan, neural Parametric Steered Mixture of Experts for Inverse Prblems, PhD dissertation, will submit March 2025.