Bad performance with NoisyLinear

[sparisi]

I am trying to implement Rainbow DQN and I am using NoisyLinear layers. The performance of the epsilon-greedy policy is indeed better, but when I turn eval mode on (basically disabling the noise in the layers) the performance is awful. I understand that sometimes switching to eval mode may make the performance worse, but this is REALLY bad, it's like the network can't do anything without the noise. Is this normal?

[vmoens]

Exciting! Would love to see that implementation!

Do you reset the noise prior to eval?

Let's think about how to best handle noise resetting perhaps - this could be a good occasion to refresh this class that we haven't touched for a very long time now!

[sparisi]

Thanks for your reply!
I just call network.eval() (network is a nn.Sequential model) and don't do anything else. How should I reset noise?

A few more details: while the network is being trained in train() mode, I periodically make a copy of it, call eval() on the copy and run the greedy policy on a few testing episodes. The testing one usually has better performance (sum of rewards) since epsilon = 0, except with NoisyLinear.

[vmoens]

Can you share how you copy the module?

I'm keen on refactoring a bit the whole thing, and have the noise component stored as part of the data (reset by a TensorDictPrimer transform in the env). That way, env.reset will cause the noise to be reset as well.
We could also use the exploration_mode to tell whether the noise should be 0 (no exploration) or non-zero (during inference). I think these will facilitate the handling of the noise + exploration.

[sparisi]

I think it's the standard way.

• Q-Network is a nn.Sequential.
• I initialized two Q-Network at the beginning of training: q_train and q_test.
• q_test.eval()
• Every fixed number of steps, I do q_test.load_state_dict(q_train).
• (I also do q_test.eval() after copy, but it should be unnecessary).
• Testing episodes start, where action = np.argmax(q_test(obs)).

I am not really sure if original Noisy Nets paper by Fortunato tested them like that. And I think Rainbow did not run any testing episode and just reported performance of q_train as training progresses.