During the process of using DeepMD kit, how can we improve the GPU utilization?

[PolyuWeldingSpock]

During the process of training my data with DeepMD kit, I noticed that the GPU utilization is only 60%. May I ask if there are any parameter settings that can enhance the GPU utilization during the DeepMD kit training process? My configuration is 6 CPUs + 64GB memory, paired with a 3080Ti graphics card.

[njzjz]

Increasing the batch size may help. Have you used the latest version?

[PolyuWeldingSpock]

@njzjz sorry about my answer. Yes, I mean the dev version

[njzjz]

If you built from source, I want to confirm if you have enabled GPU support by setting export DP_VARIANT=cuda during installation.

[PolyuWeldingSpock]

I installed with conda not from source.

[njzjz]

The GPU utilization is the percentage of the GPU time in the total time. You can enable enable_profiler for profiling.

[PolyuWeldingSpock]

OK, Thanks for your help.