About tokenizer rFID evaluation

[Pan-Zihao]

Why is the rFID around 2.2 when I use the tokenizer weights to perform inference on the 50K samples of the imagenet256 validation set? This is quite different from the result of around 1.5 in the paper. Could you please provide the details of the tokenizer rFID evaluation? Thank you

[sebgao]

Hi, thank you for your interest in our work. Please provide commands or tools you used to evaluate rFID so I can inspect further.

[Pan-Zihao]

I used clean-fid, input the 50k image folder path in imagenet-val and the 50k image folder path reconstructed using your ddp script. The reconstructed images are saved in png, 256 size.

[sebgao]

Let me first assume clean-fid provide the correct implemetation (but i think there's subtle differences against the standard practice). Are you using the imagenet-val original samples in the imagenet val folder?

[Pan-Zihao]

Let me first assume clean-fid provide the correct implemetation (but i think there's subtle differences against the standard practice). Are you using the imagenet-val original samples in the imagenet val folder?

Yes, 50K .jpeg images

[sebgao]

Oh that's where the discrepancy happens. You need to evaluate reconstructed samples against the image with the same center cropping region used as the input into the model, which is the common practice. Actually, I provide the folder to store these cropped images (see

D-AR/tokenizer/tokenizer_image/reconstruction_vq_ddp.py

Line 167 in cd921b8

Image.fromarray((rgb_gt*255.).astype(np.uint8)).save(f"{args.sample_dir}/gt/{index:06d}.png")

). The difference is caused by: the reconstrcued images are not the same region with the original images in imagenet val folder (which needs also center cropping).

[Pan-Zihao]

Oh that's where the discrepancy happens. You need to evaluate reconstructed samples against the image with the same center cropping region used as the input into the model, which is the common practice. Actually, I provide the folder to store these cropped images (see

D-AR/tokenizer/tokenizer_image/reconstruction_vq_ddp.py

Line 167 in cd921b8

Image.fromarray((rgb_gt*255.).astype(np.uint8)).save(f"{args.sample_dir}/gt/{index:06d}.png")
). The difference is caused by: the reconstrcued images are not the same region with the original images in imagenet val folder (which needs also center cropping).

Thanks for your reply, I'll try again

[sebgao]

Hope this helps! Actually I have already provided two approaches of out-of-box rFID evaluations in https://github.com/showlab/D-AR/blob/main/eval_recon_rfid.sh, which I tested and they successfully reproduce the exact results as in the paper.