HSI-Diffusers

This repository contains the source code for the paper Semantic Hyperspectral Image Synthesis for Cross-Modality Knowledge Transfer in Surgical Data Science, presented at the International Conference on Information Processing in Computer-Assisted Interventions (IPCAI) 2025.

Based off the following repositories:

• Latent Diffusion Models
• The diffusers package
• The htc package

---

Installation and setup

Using pyproject.toml

To set up the virtual environment using pyproject.toml, run the following command will create the environment with the necessary dependencies:

pip install -r pyproject.toml

Example usage

Examples of using our repository are found in examples/. These include:

Sample training script

As our training data is not publicly available, you can run a sample training script (with the same training config and hyperparameters as for data training) as follows:

python examples/sample_training_script.py

Sample synthesis script

This script shows an example of loading a latent diffusion model from a checkpoint and generating samples. You can run it as follows:

python examples/sample_synthesis.py

Run the tests

To run the tests, you can use the following command:

pytest tests/

Configs

The configurations of the model architectures and training data augmentations we used for our paper are found in configurations/. These include:

• configurations/hsi/ae/autoencoder_config.json: The autoencoder architecture configuration.
• configurations/hsi/ae/autoencoder_loss_config.json: The autoencoder loss configuration.
• configurations/hsi/data/data_config.json: The training and validation data transformation set configuration, including training data augmentations.
• configurations/hsi/diffusion_model/diffusion_model_config.json: The latent diffusion model architecture configuration.

Full Project Overview

├── configurations                 <- Contains the model architecture and training augmentation configurations used in the paper.
│   └── hsi                        <- Data from third party sources.
│       ├── ae                     <- Contains the autoencoder architecture configuration.
│       ├── data                   <- Contains the training augmentation configuration.
│       └── diffusion_model        <- Contains the latent diffusion model architecture configuration.
│
├── examples              
│   ├── sample_synthesis.py        <- Example script of loading a conditional latent diffusion model from a checkpoint and generating samples.
│   └── sample_training_script.py  <- Example script of training an autoencoder and the latent diffusion model from dummy data.
│
├── hsi_diffusers                  <- Contains the source code.
│   ├── data
│   │   └── hsi  
│   │       └── transforms.py      <- Defines the (training) data augmentations.  
│   ├── training
│   │   ├── model_wrappers.py      <- Defines the autoencoder and (conditional) latent diffusion model wrappers so that they are compatible with diffusers and pytorch-lightning.
│   │   └── model_wrapper
│   │       ├── lr_schedulers.py   <- Defines the learning rate schedulers used during training.
│   │       ├── autoencoder       
│   │       │   └── losses.py      <- Defines the classes which calculate the autoencoder loss during training.
│   │       └── base_models     
│   │           └── encoders.py    <- Defines the spatial rescaler used to rescale the semantic conditioning of the latent diffusion model.
│   └── utils                      <- Contains util functions such as creating an object from configuration files and checkpoint loading.
│
└── tests                          <- Contains the unit tests of the source code.

Citation

If you use this repository, we’d appreciate it if you cited our paper.

@article{2025hsidiffusers,
  title={Semantic hyperspectral image synthesis for cross-modality knowledge transfer in surgical data science},
  author={Tran Ba, Viet and H{\"u}bner, Marco and Bin Qasim, Ahmad and Rees, Maike and Sellner, Jan and Seidlitz, Silvia and Christodoulou, Evangelia and {\"O}zdemir, Berkin and Studier-Fischer, Alexander and Nickel, Felix and others},
  journal={International Journal of Computer Assisted Radiology and Surgery},
  pages={1--9},
  year={2025},
  publisher={Springer}
}

Funding

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. [101002198]).