VQPET: Leveraging Vector-quantized Codebook Prior for PET Image Synthesis

Positron emission tomography (PET) image synthesis is a highly ill-posed problem that requires auxiliary priors to 1) alleviate the loss of high-quality (HQ) information in low-quality (LQ) inputs, and 2) impose additional constraints to reduce mapping uncertainty. However, existing auxiliary priors in PET image synthesis often provide inadequate guidance due to inaccurate prior information or limited prior expressiveness. To overcome the aforementioned limitations, the vector-quantized (VQ) codebook prior is employed as a promising solution. By learning discrete latent feature representations of HQ images through deep models, the VQ codebook prior encompasses accurate HQ information and possesses great expressiveness. Building upon this, we propose a novel two-stage framework, VQPET, that introduces the VQ codebook prior for PET image synthesis. In the first stage, it pretrains a VQGAN on an additional large-scale HQ PET dataset, encoding intrinsic HQ features as code items in the VQ codebook. The VQ codebook prior is thus derived from the high-level features obtained from the pretrained VQGAN and serves as an additional constraint for downstream synthesis. In the second stage, it develops a codebook-prior-guided network (CPGNet) that effectively exploits the VQ codebook prior to produce realistic outputs. Specifically, CPGNet progressively incorporates the VQ codebook prior at multiple decoding levels, providing reliable guidance for HQ synthesis. Compared to previous works, VQPET innovatively leverages additional large-scale HQ datasets to transfer pretrained prior knowledge for enhanced synthesis and functions as a general framework applicable to any encoder-decoder network. Extensive experiments demonstrate the substantial effect and robust generalizability of VQPET.