Training fails with more than 1 GPU and/or more than 1 epoch

[hpckurt]

I get ValueError: ctypes objects containing pointers cannot be pickled when trying to train with more than a single GPU:

(env) kstine@n02:/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch$ python train.py --threads 4 --num-workers 4 --gpus 2 --max_epochs 100 ../janggi-d8-1b.bin ../janggi-d8-1b.bin 
Feature set: HalfKAv2^
Num real features: 6336
Num virtual features: 768
Num features: 7104
Training with ../janggi-d8-1b.bin validating with ../janggi-d8-1b.bin
[rank: 0] Global seed set to 42
Seed 42
Using batch size 16384
Smart fen skipping: True
Random fen skipping: 3
limiting torch to 4 threads.
Using log dir logs/
ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:478: LightningDeprecationWarning: Setting `Trainer(gpus=2)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=2)` instead.
  rank_zero_deprecation(
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python train.py --threads 4 --num-workers 4 --gpus 2 --max_ ...
  rank_zero_warn(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Using c++ data loader
Traceback (most recent call last):
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 96, in <module>
    main()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 93, in main
    trainer.fit(nnue, train, val)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 280, in start_processes
    idx, process, tf_name = start_process(i)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 275, in start_process
    process.start()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled
[W605 15:04:04.904764584 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

Using only a single GPU works fine.

[hpckurt]

In addition, when training with more than one epoch, the process crashes, regardless of how many threads, workers, or gpus are used:

Epoch 1:   0%|▏                                                                                                                                                                       | 1/1283 [00:00<00:08, 155.67it/s, loss=0.0168, v_num=7]
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt                                                                     
    return trainer_fn(*args, **kwargs)                                                                                                                                                                                                        
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl                                                                                  
    self._run(model, ckpt_path=self.ckpt_path)                                                                                                                                                                                                
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run                                                                                      
    results = self._run_stage()                                                                                                                                                                                                               
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage                                                                                
    self._run_train()                                                                                                                                                                                                                         
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train                                                                                
    self.fit_loop.run()                                                                                                                                                                                                                       
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run                                                                                             
    self.advance(*args, **kwargs)                                                                                                                                                                                                             
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance                                                                                     
    self._outputs = self.epoch_loop.run(self._data_fetcher)                                                                                                                                                                                   
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run                                                                                             
    self.advance(*args, **kwargs)                                                                                                                                                                                                             
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 229, in advance                                                                    
    self.trainer._call_callback_hooks("on_train_batch_end", batch_end_outputs, batch, batch_idx)                                                                                                                                              
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1394, in _call_callback_hooks                                                                      
    fn(self, self.lightning_module, *args, **kwargs)                                                                                                                                                                                          
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/tqdm_progress.py", line 264, in on_train_batch_end                                                        
    self.main_progress_bar.set_postfix(self.get_metrics(trainer, pl_module))                                                                                                                                                                  
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/base.py", line 245, in get_metrics                                                                        
    standard_metrics = get_standard_metrics(trainer, pl_module)                                                                                                                                                                               
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/base.py", line 274, in get_standard_metrics                                                               
    avg_training_loss = running_train_loss.cpu().item()                                                                                                                                                                                       
RuntimeError: CUDA error: an illegal memory access was encountered                                                                                                                                                                            
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.                                                                                                                       
For debugging consider passing CUDA_LAUNCH_BLOCKING=1                                                                                                                                                                                         
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 96, in <module>
    main()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 93, in main
    trainer.fit(nnue, train, val)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 63, in _call_and_handle_interrupt
    trainer._teardown()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1175, in _teardown
    self.strategy.teardown()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 492, in teardown
    _optimizers_to_device(self.optimizers, torch.device("cpu"))
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/optimizer.py", line 28, in _optimizers_to_device
    _optimizer_to_device(opt, device)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/optimizer.py", line 34, in _optimizer_to_device
    optimizer.state[p] = apply_to_collection(v, Tensor, move_data_to_device, device)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 74, in apply_to_collection
    return _apply_to_collection_slow(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 106, in _apply_to_collection_slow
    v = _apply_to_collection_slow(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 98, in _apply_to_collection_slow
    return function(data, *args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/apply_func.py", line 101, in move_data_to_device
    return apply_to_collection(batch, dtype=_TransferableDataType, function=batch_to)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 66, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/apply_func.py", line 95, in batch_to
    data_output = data.to(device, **kwargs)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

[gbtami]

Can you try to apply official-stockfish/nnue-pytorch@1f7974f and test it please?

[hpckurt]

@gbtami Same issue as before

[gbtami]

@hpckurt Can you provide the full traceback starting from the command you use please?
Btw have you installed requirements.txt or requirements-CUDA128.txt ?

[hpckurt]

@gbtami

Output when using 8 GPUs:

(env) kstine@n20:/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch$ python train.py --threads 8 --num-workers 112 --gpus 8 --max_epochs 2 ../janggi-d8-1b.bin ../janggi-d8-1b.bin
Feature set: HalfKAv2^
Num real features: 6336
Num virtual features: 768
Num features: 7104
Training with ../janggi-d8-1b.bin validating with ../janggi-d8-1b.bin
[rank: 0] Global seed set to 42
Seed 42
Using batch size 16384
Smart fen skipping: True
Random fen skipping: 3
limiting torch to 8 threads.
Using log dir logs/
ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:478: LightningDeprecationWarning: Setting `Trainer(gpus=8)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=8)` instead.
  rank_zero_deprecation(
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python train.py --threads 8 --num-workers 112 --gpus 8 --ma ...
  rank_zero_warn(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Using c++ data loader
Traceback (most recent call last):
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 98, in <module>
    main()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 95, in main
    trainer.fit(nnue, train, val)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 280, in start_processes
    idx, process, tf_name = start_process(i)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 275, in start_process
    process.start()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled
[W608 14:38:30.195886787 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

Output when using 1 GPU, but multiple Epochs:

(env) kstine@n20:/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch$ python train.py --threads 8 --num-workers 112 --gpus 1 --max_epochs 2 ../janggi-d8-1b.bin ../janggi-d8-1b.bin
Feature set: HalfKAv2^
Num real features: 6336
Num virtual features: 768
Num features: 7104
Training with ../janggi-d8-1b.bin validating with ../janggi-d8-1b.bin
[rank: 0] Global seed set to 42
Seed 42
Using batch size 16384
Smart fen skipping: True
Random fen skipping: 3
limiting torch to 8 threads.
Using log dir logs/
ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:478: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
  rank_zero_deprecation(
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python train.py --threads 8 --num-workers 112 --gpus 1 --ma ...
  rank_zero_warn(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Using c++ data loader
You are using a CUDA device ('NVIDIA H100 80GB HBM3') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
Ranger optimizer loaded. 
Gradient Centralization usage = False

  | Name         | Type                          | Params
---------------------------------------------------------------
0 | input        | DoubleFeatureTransformerSlice | 3.7 M 
1 | layer_stacks | LayerStacks                   | 152 K 
---------------------------------------------------------------
3.8 M     Trainable params
0         Non-trainable params
3.8 M     Total params
15.387    Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 112 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 112 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Epoch 1:   0%|                                                                                                                                                 | 1/1283 [00:00<00:45, 28.12it/s, loss=0.0142, v_num=8]Traceback (most recent call last):                                                                                                                                                                                    
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
    self.fit_loop.run()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 229, in advance
    self.trainer._call_callback_hooks("on_train_batch_end", batch_end_outputs, batch, batch_idx)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1394, in _call_callback_hooks
    fn(self, self.lightning_module, *args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/tqdm_progress.py", line 264, in on_train_batch_end
    self.main_progress_bar.set_postfix(self.get_metrics(trainer, pl_module))
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/base.py", line 245, in get_metrics
    standard_metrics = get_standard_metrics(trainer, pl_module)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/callbacks/progress/base.py", line 274, in get_standard_metrics
    avg_training_loss = running_train_loss.cpu().item()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 98, in <module>
    main()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/train.py", line 95, in main
    trainer.fit(nnue, train, val)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 63, in _call_and_handle_interrupt
    trainer._teardown()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1175, in _teardown
    self.strategy.teardown()
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 492, in teardown
    _optimizers_to_device(self.optimizers, torch.device("cpu"))
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/optimizer.py", line 28, in _optimizers_to_device
    _optimizer_to_device(opt, device)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/optimizer.py", line 34, in _optimizer_to_device
    optimizer.state[p] = apply_to_collection(v, Tensor, move_data_to_device, device)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 74, in apply_to_collection
    return _apply_to_collection_slow(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 106, in _apply_to_collection_slow
    v = _apply_to_collection_slow(
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 98, in _apply_to_collection_slow
    return function(data, *args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/apply_func.py", line 101, in move_data_to_device
    return apply_to_collection(batch, dtype=_TransferableDataType, function=batch_to)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_utilities/core/apply_func.py", line 66, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/scratch/m000001/kstine/nnuedata/variant-nnue-pytorch/env/lib/python3.10/site-packages/lightning_fabric/utilities/apply_func.py", line 95, in batch_to
    data_output = data.to(device, **kwargs)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

This is with requirements-CUDA128.txt since Hopper and Blackwell cards don't support CUDA 11.