Reproduce thunder main leaderboard results

[afilt]

Hi @PierreMarza !

I would like to test a custom model: I first pre-compute the embeddings then run the benchmark.
Is that correct ? How are computed the leaderboard's results between online_loading, image_pre_loading and embedding_pre_loading ? Also, I don't see reference to mixed precision, what's the policy to run this benchmark - full precision ?

I've tried online loading but it takes 1 hour for a ViT-B on BACH compared to 2 minutes feature extraction + 2 minutes linear probing.

Lastly, is there somewhere I can find the per-dataset results for models not benchmarked in the paper (for instance H0-mini) ?

Thank you.
Alexandre

[PierreMarza]

Hi @afilt !

Glad to see that you are using thunder to benchmark some model :)

Data loading

The online_loading, image_pre_loading and embedding_pre_loading settings are only related to the linear_probing and segmentation tasks (only tasks involving some training -- either linear probe or segmentation decoder), and should not have an impact on the final performance. We simply provide this --loading-mode flag to give more flexibility to the user in terms of the data loading:

• online_loading: batch images are loaded online during training and you perform encoder forward passes.
• image_pre_loading: all images are first loaded into RAM and you also perform encoder forward passes.
• embedding_pre_loading: pre-computed embeddings are loaded and no forward pass with the encoder is performed.

online_loading and image_pre_loading might be needed in special cases, e.g. if you want to do some LoRA adaptation of the encoder. However, if your goal is to do standard linear probing evaluation, I would indeed suggest to first pre-compute embeddings (e.g. thunder benchmark h0mini patch_camelyon pre_computing_embeddings), before to run linear_probing with --loading-mode=embedding_pre_loading (e.g. thunder benchmark h0mini patch_camelyon linear_probing --loading-mode=embedding_pre_loading) to speed up your experiments.

Precision

Yes, we currently extract embeddings with full precision for all models.

Detailed linear probing results

We just added a table reporting all linear probing results (F1-scores) for all models and datasets on thunder: https://mics-lab.github.io/thunder/leaderboards/#per-dataset-linear-probing-performance. Here, you can filter models based on their training domain (histo vs natural) and modalities (vision-only vs vision-language), and you can also search for a specific model (e.g. "H0-MINI").

Let us know if you have any other question!
Best,
Pierre

[afilt]

Thank you for this clear and detailed answer.
I've indeed started precomputing the embeddings. You may want to (re?) explain what you just wrote in the documentation, because I've started doing the default (which is online_loading): basically that the most efficient way of just performing linear probing (which I guess is the use-case people will try first) is to first pre-compute embeddings.

Regarding precision, I think it can be prohibitive for some users to run full precision on ViT-giant models. This would mean on your end to add additional results on your side with the mixed-precision models and may be time-consuming. Probably good to put it in your backlog ?

Thank you for the reporting table, really clear !

[afilt]

I have an additional comment about SPIDER: downloading it through thunder download triggers the creation of 3M+ individual pngs. I think it will be impossible for some users to benchmark on this dataset. Could you have tasks just on a subset ? Does that really make sense to have so many images for linear probing experiments (which converges quite fast in terms of number of images) ?

Also, my slurm script ended at the end of the 4th hour and I could not complete the creation of the spider-skin images (like 2.5M out of 2.6M). I don't see how to resume the creation of pngs so I need to start from scratch (unless I modify the code for instance not to extract the tar archives. Could those archives be at least deleted on the fly when all images within have been extracted ? Because for now I don't which tar hasn't be extracted yet.

Thank you !

[PierreMarza]

Tutorial on data loading on documentation

Thanks for the suggestion! I've turned my previous explanation into a small tutorial: https://mics-lab.github.io/thunder/data_loading/

Mixed precision

That's a good point. Even if, to the best of our knowledge, our different tasks should run fine on a single V100 GPU for most considered models and datasets, it is true that adding support for mixed precision could be relevant. Indeed, we should then compute additional results with such a mixed precision. We'll see what we can do here!

SPIDER

Thanks for reporting this concern about SPIDER. We will also see what we can do to make this whole process lighter, and to allow resuming un-finished downloading.

If you have any code change suggestion regarding the last two points, feel free to let us know, or to even open a PR to propose a specific adjustment. Thanks!

[afilt]

Super nice, thanks !
When you say "settings are only related to the linear_probing and segmentation tasks", isn't it also the case for adversarial_attack, simple_shot, etc. ?

Regarding mixed precision I think it could be included in the pretrained model configurations (for custom models, the user should be responsible for implementing mixed precision himself). On my side I simply did:

import torch
from torch import autocast
import torchvision.transforms as transforms
import timm
from thunder.models import PretrainedModel

class MyModel(PretrainedModel):

    def __init__(self):
        super().__init__()
        #...
        self.device_type = "cuda" if torch.cuda.is_available() else "cpu"
        self.amp_dtype = torch.float16 if self.device_type == "cuda" else torch.float32


    def forward(self, x):
        with autocast(device_type=self.device_type, dtype=self.amp_dtype):
            feats = self.model.forward_features(x)
        return feats

So mixed precision is enabled by default (hardcoded). I get you could ask a self.mixed_precision parameter inside the model configurations that defaults to false then adjust the implementation by doing something like:

    def forward(self, x):
        with autocast(device_type=self.device_type, dtype=torch.float16 if self.mixed_precision else torch.float32):
            feats = self.model.forward_features(x)
        return feats

[afilt]

For spider, probably the way to go is to create a more compact format, for instance a series of chunks of size 1000 and then use a custom loader or something similar to Webdataset ?
For resuming, I believe removing the archives once they were completely extracted could be nice: the script should only decompress the remaining archives (it is not perfect resuming though if the script crashes while extracting one archive but better then nothing).

[afilt]

If you find it useful to integrate in the documentation, here is a script to precompute embeddings based on a custom model:

export THUNDER_BASE_DATA_FOLDER="/your/output/dir/"

MODEL_PATH="custom:/path/to/custom/model.py"
TASK="pre_computing_embeddings"

DATASETS=(
  bracs
  bach
  break_his
  ccrcc
  crc
  esca
  mhist
  ocelot
  pannuke
  patch_camelyon
  segpath_epithelial
  segpath_lymphocytes
  spider_breast
  spider_colorectal
  spider_skin
  spider_thorax
  tcga_crc_msi
  tcga_tils
  tcga_uniform
  wilds
  spider_skin
  bracs
  spider_thorax
)

echo "Starting Thunder benchmarks..."
echo "Model: $MODEL_PATH"
echo "Task: $TASK"
echo "--------------------------------------"

for DATASET in "${DATASETS[@]}"; do
  echo "Running dataset: $DATASET"
  WANDB_MODE=offline thunder benchmark "$MODEL_PATH" "$DATASET" "$TASK"
  echo "Finished dataset: $DATASET"
  echo "--------------------------------------"
done

echo "All benchmarks completed!"

[afilt]

And another one to run the full benchmark suit needed for the leaderboard (once embeddings were precomputed):

export THUNDER_BASE_DATA_FOLDER="/your/output/dir/"
MODEL_PATH="custom:/path/to/custom/model.py"

################ Classification ################ 
TASKS=(
  linear_probing
  knn
  adversarial_attack
  simple_shot
)

DATASETS=(
  bach
  bracs
  break_his
  ccrcc
  crc
  esca
  mhist
  patch_camelyon
  spider_breast
  spider_colorectal
  spider_skin
  spider_thorax
  tcga_crc_msi
  tcga_tils
  tcga_uniform
  wilds
  spider_skin
  bracs
  spider_thorax
)


echo "Starting Thunder Classification benchmarks..."
echo "Model: $MODEL_PATH"
echo "--------------------------------------"

for DATASET in "${DATASETS[@]}"; do
  echo "Running dataset: $DATASET"
  for TASK in "${TASKS[@]}"; do
    WANDB_MODE=offline thunder benchmark "$MODEL_PATH" "$DATASET" "$TASK" --loading-mode=embedding_pre_loading
    echo "Finished task: $TASK"
    echo "--------------------------------------"
  done
  echo "Finished dataset: $DATASET"
  echo "--------------------------------------"
done

echo "All classification benchmarks completed!"


################ Segmentation ################ 
TASKS=(
  segmentation
)

DATASETS=(
  ocelot
  pannuke
  segpath_epithelial
  segpath_lymphocytes
)
echo "Starting Thunder Segmentation benchmarks..."
echo "Model: $MODEL_PATH"
echo "--------------------------------------"

for DATASET in "${DATASETS[@]}"; do
  echo "Running dataset: $DATASET"
  for TASK in "${TASKS[@]}"; do
    WANDB_MODE=offline thunder benchmark "$MODEL_PATH" "$DATASET" "$TASK" --loading-mode=embedding_pre_loading
    echo "Finished task: $TASK"
    echo "--------------------------------------"
  done
  echo "Finished dataset: $DATASET"
  echo "--------------------------------------"
done

echo "All segmentation benchmarks completed!"