Evaluation results of Stage 1 are lower than reported in the paper

[Eku127]

Thanks for sharing the training dataset and code. I followed your guide to generate the training data and used the provided training code to successfully train a model using RxR + R2R + envdrop data. However, after evaluating on R2R_v1, I found that the results are significantly lower than those reported in Table 1 of your paper: (NE = 5.43, OS = 62.5, SR = 52.8, SPL = 47.2).

I have also tested the results on R2R_v1-3, the results are almost the same.

I did not modify the data generation code, and I kept all training parameters unchanged, except for adjusting the per_device_train_batch_size=1 and gradient_accumulation_steps=4.
To assist with troubleshooting, I’ve attached my training and evaluation scripts for reference. Also the trainning log is attached.

Could you please advise if you followed any additional steps not mentioned in the guide, or if there are any known issues that might lead to this performance drop ?I would also like to know the results you obtained when you re-ran the experiments internally.

I would appreciate any suggestions to ensure the reproduction of the same results.

Attachments:

1. Training Scripts
   streamvln_train.zip

2. Evaluaiton Scripts
   streamvln_eval.zip

3. Final Results (Updated)
   results.zip

4. Training Log
   streamvln-12670.zip

5. Trained ckpt (Updated)
   OneDrive

Update (250811)

1. The results after the bugfix in dataset code. Auther‘s Reply Result: Evaluation results of Stage 1 are lower than reported in the paper #27 (comment)

Source	Version / Note	NE	OS	SR	SPL
Paper	Table 3 Row 1	6.05	53.8	45.5	41.6
Paper	Table 1 Row 20	5.43	62.5	52.8	47.2
Author's Reply	v1	5.88	49.8	45.3	42.3
Eku127's Results	v1 bs 64	6.36	48.2	42.1	38.3
Eku127's Results	v1 bs 128	6.29	50.0	43.6	39.9
Author's Reply	v1-3	5.90	51.4	46.7	43.3
Eku127's Results	v1-3 bs 64	5.93	51.4	46.0	42.3
Eku127's Results	v1-3 bs 128	6.34	50.4	43.9	40.0

[kellyiss]

The training loss appears to be normal. The issue might be related to the evaluation. You may first check whether the config stored with your checkpoint differs from the one we released. We only rerun the evaluation when we release the code. We'll check if there's something wrong with the dataset.

[missTL]

May I ask if you have encountered the following errors during the training process?

11%|█ | 67/624 [33:09<4:10:13, 26.95s/it]

11%|█ | 68/624 [33:36<4:09:53, 26.97s/it]

11%|█ | 69/624 [34:06<4:18:18, 27.92s/it]

11%|█ | 70/624 [34:32<4:12:58, 27.40s/it]

11%|█ | 70/624 [34:32<4:12:58, 27.40s/it]

Traceback (most recent call last):
File "/checkpoint/binary/train_package/streamvln/streamvln_train.py", line 1785, in
train()
File "/checkpoint/binary/train_package/streamvln/streamvln_train.py", line 1758, in train
trainer.train()
File "/root/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1903, in train
return inner_training_loop(
File "/root/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2200, in _inner_training_loop
for step, inputs in enumerate(epoch_iterator):
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/accelerate/data_loader.py", line 462, in iter
next_batch = next(dataloader_iter)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 701, in next
data = self._next_data()
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 757, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/checkpoint/binary/train_package/streamvln/dataset/vln_action_dataset.py", line 754, in getitem
sample_frames = [os.path.join(video_path, 'rgb', video_frames[i]) for i in sample_step_ids]
File "/checkpoint/binary/train_package/streamvln/dataset/vln_action_dataset.py", line 754, in
sample_frames = [os.path.join(video_path, 'rgb', video_frames[i]) for i in sample_step_ids]
IndexError: list index out of range

[Eku127]

The training loss appears to be normal. The issue might be related to the evaluation. You may first check whether the config stored with your checkpoint differs from the one we released. We only rerun the evaluation when we release the code. We'll check if there's something wrong with the dataset.

Thanks for the prompt reply and yes, there are quite a few differences. I'm sharing the config file I got after training for your reference. Also the full ckpt is uploaded.

Further, is there any code or script to check the correctness of the dataset? Any effort that I can make to check the dataset, especially the generated EnvDrop?

Attachment
config.json

[Eku127]

May I ask if you have encountered the following errors during the training process?

No, the training process worked smoothly.

[LeonGoretzkatju]

Thanks for sharing the training dataset and code. I followed your guide to generate the training data and used the provided training code to successfully train a model using RxR + R2R + envdrop data. However, after evaluating on R2R_v1, I found that the results are significantly lower than those reported in Table 1 of your paper: (NE = 5.43, OS = 62.5, SR = 52.8, SPL = 47.2).

I have also tested the results on R2R_v1-3, the results are almost the same.

I did not modify the data generation code, and I kept all training parameters unchanged, except for adjusting the per_device_train_batch_size=1 and gradient_accumulation_steps=4. To assist with troubleshooting, I’ve attached my training and evaluation scripts for reference. Also the trainning log is attached.

Could you please advise if you followed any additional steps not mentioned in the guide, or if there are any known issues that might lead to this performance drop ?I would also like to know the results you obtained when you re-ran the experiments internally.

I would appreciate any suggestions to ensure the reproduction of the same results.

Attachments:

1. Training Scripts
   [streamvln_train.zip](https://github.com/user-attachments/files/21616541/streamvln_train.zip)

2. Evaluaiton Scripts
   [streamvln_eval.zip](https://github.com/user-attachments/files/21616540/streamvln_eval.zip)

3. Final Results (Updated)
   [results.zip](https://github.com/user-attachments/files/21648751/results.zip)

4. Training Log
   [streamvln-12670.zip](https://github.com/user-attachments/files/21616547/streamvln-12670.zip)

5. Trained ckpt (Updated)
   [OneDrive](https://hkustgz-my.sharepoint.com/:f:/g/personal/jjiang127_connect_hkust-gz_edu_cn/Eg2KPK52OTNHqhd6UP2BbagBOm3xgLmo3SW_YCznar0ZlA?e=Rlz8aP)

I also meet the same problem. I trained this model on R2R+RxR+envdrop+scalevln, without any modification of any other parameters in the provided script, after 1 epoch training, the result is shown as follows,

{'sucs_all': 0.1457313895225525, 'spls_all': 0.12397201441137161, 'oss_all': 0.2718869149684906, 'ones_all': 7.7583088874816895, 'length': 1839}, which is also significantly lower than the metrics reported in the paper.

I would greatly appreciate it if you could provide some suggestions related to this problem.

Thanks a lot!

[kellyiss]

We only reorganized the dataset code at that time and quickly checked the outputs before uploading, so it was a bit rushed 😭. No worries — we're already working on fixing it.

[kellyiss]

Hi all,

The training bug in vln_action_dataset.py has been fixed.

We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:

R2R_v1
{
"sucs_all": 0.45350736379623413,
"spls_all": 0.42276284636696476,
"oss_all": 0.49864059686660767,
"ones_all": 5.881659030914307,
"length": 1839
}
R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.

R2R_v1-3 (first-stage results are slightly higher)
{
"sucs_all": 0.46655791997909546,
"spls_all": 0.43291992790826866,
"oss_all": 0.5149537920951843,
"ones_all": 5.900496959686279,
"length": 1839
}

[Eku127]

Hi all,

The training bug in vln_action_dataset.py has been fixed.

We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:

R2R_v1 { "sucs_all": 0.45350736379623413, "spls_all": 0.42276284636696476, "oss_all": 0.49864059686660767, "ones_all": 5.881659030914307, "length": 1839 } R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.

R2R_v1-3 (first-stage results are slightly higher) { "sucs_all": 0.46655791997909546, "spls_all": 0.43291992790826866, "oss_all": 0.5149537920951843, "ones_all": 5.900496959686279, "length": 1839 }

Thanks for the prompt fix! Should we re-train the model or just re-run the evaluation?

[kellyiss]

Hi all,
The training bug in vln_action_dataset.py has been fixed.
We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:
R2R_v1 { "sucs_all": 0.45350736379623413, "spls_all": 0.42276284636696476, "oss_all": 0.49864059686660767, "ones_all": 5.881659030914307, "length": 1839 } R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.
R2R_v1-3 (first-stage results are slightly higher) { "sucs_all": 0.46655791997909546, "spls_all": 0.43291992790826866, "oss_all": 0.5149537920951843, "ones_all": 5.900496959686279, "length": 1839 }

Thanks for the prompt fix! Should we re-train the model or just re-run the evaluation?

Since it’s a dataset bug, retraining is required.

[missTL]

Hi all,

The training bug in vln_action_dataset.py has been fixed.

We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:

R2R_v1 { "sucs_all": 0.45350736379623413, "spls_all": 0.42276284636696476, "oss_all": 0.49864059686660767, "ones_all": 5.881659030914307, "length": 1839 } R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.

R2R_v1-3 (first-stage results are slightly higher) { "sucs_all": 0.46655791997909546, "spls_all": 0.43291992790826866, "oss_all": 0.5149537920951843, "ones_all": 5.900496959686279, "length": 1839 }

May I ask if this is the result obtained from training solely on the R2R-CE and RxR-CE datasets? No EnvDrop data?

[kellyiss]

Hi all,
The training bug in vln_action_dataset.py has been fixed.
We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:
R2R_v1 { "sucs_all": 0.45350736379623413, "spls_all": 0.42276284636696476, "oss_all": 0.49864059686660767, "ones_all": 5.881659030914307, "length": 1839 } R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.
R2R_v1-3 (first-stage results are slightly higher) { "sucs_all": 0.46655791997909546, "spls_all": 0.43291992790826866, "oss_all": 0.5149537920951843, "ones_all": 5.900496959686279, "length": 1839 }

May I ask if this is the result obtained from training solely on the R2R-CE and RxR-CE datasets? No EnvDrop data?

R2R-CE + RxR-CE + EnvDrop

[missTL]

Hi all,
The training bug in vln_action_dataset.py has been fixed.
We have trained and tested on R2R_v1 and R2R_v1-3 respectively, and the results are:
R2R_v1 { "sucs_all": 0.45350736379623413, "spls_all": 0.42276284636696476, "oss_all": 0.49864059686660767, "ones_all": 5.881659030914307, "length": 1839 } R2R_v1 first-stage results correspond to the first row in Table 3 of the paper.
R2R_v1-3 (first-stage results are slightly higher) { "sucs_all": 0.46655791997909546, "spls_all": 0.43291992790826866, "oss_all": 0.5149537920951843, "ones_all": 5.900496959686279, "length": 1839 }

May I ask if this is the result obtained from training solely on the R2R-CE and RxR-CE datasets? No EnvDrop data?

R2R-CE + RxR-CE + EnvDrop

Could you please explain how to reproduce the result in this row of Table 1? I noticed in another issue it was stated as the result of R2R-CE + RxR-CE + EnvDrop, which does not match the result here.

So, in fact, this is the result that includes R2R-CE + RxR-CE + EnvDrop + VideoQA + MMC4 + DAgger data?

[Eku127]

After retraining and re-evaluation, I got the following results:

Source	Version / Note	NE	OS	SR	SPL
Paper	Table 3 Row 1	6.05	53.8	45.5	41.6
Paper	Table 1 w/o dagger	5.43	62.5	52.8	47.2
Author's Reply	v1	5.88	49.8	45.3	42.3
Eku127's Results	v1	6.36	48.2	42.1	38.3
Author's Reply	v1-3	5.90	51.4	46.7	43.3
Eku127's Results	v1-3	5.93	51.4	46.0	42.3

The results are significantly better than the first run — thanks for the bugfix @kellyiss!
That said, I still have a couple of questions regarding these numbers:

1. On R2R-v1-3, the gap between my run and the author's run is quite small, but on R2R-v1, there’s still a noticeable difference. What could be the possible reasons for this gap? @LeonGoretzkatju, how do your re-run results look — do they show the same pattern?

2. As mentioned by @kellyiss , the current results are close to those in Table 3 Row 1. However, if the Table 1 w/o dagger results were indeed obtained by training only with R2R-CE + RxR-CE + EnvDrop, shouldn’t they be similar to Table 3 Row 1? In contrast, Table 1 w/o dagger shows significantly better performance than both our trained and evaluated results — what could explain this discrepancy? Could this discrepancy be related to what @missTL observed — that the Table 1 w/o dagger results might also include training data from VideoQA + MMC4 + DAgger?

Any insights on these points would be really helpful. Thanks in advance for looking into this!

Attachment:

result-v1-3.json

result-v1.json

[kellyiss]

We would like to clarify that we have never stated that the results in Table 1 correspond to stage 1 without DAgger.
The complete training of StreamVLN consists of both two stages. It is clear that the results in Table 1 correspond to the third row in Table 3. “Extra data” refers to whether ScaleVLN is used or not.

Have you use the batch size 128? The total training step of stage 1 is 3740.

[Eku127]

We would like to clarify that we have never stated that the results in Table 1 correspond to stage 1 without DAgger.

In my previous reply, the term "dagger" refers to the results in Table 1 that does not have the dagger superscript, but not the DAgger data. I apologize for the misunderstanding and clarified the saying to Table 1 Row 20

Have you use the batch size 128? The total training step of stage 1 is 3740.

Nope, I used batch size 64. Will this affect the final evaluation results?

I will re-run the trainning and evaluation with default training configs and batch size 128.