Gnosis archive node rerunning

[stdevMac]

Description
Our Gnosis Archive node was suddenly restarted, and then, all of a sudden it started rerunning the blocks from 0, even tho the head was in the latest blocks.

Steps to Reproduce
In order to replicate the behavior, please provide a detailed list of steps:

1. Sync a node
2. Hard reboot of the machine (it may work or not, may depend of the moment the restart was set)

Actual behavior
Node is rerunning all the blocks on db

Expected behavior
Keep syncing as expected

Logs

01 Aug 06:51:02 | Loaded 0 static nodes from file: /nethermind/Data/static-nodes.json 
01 Aug 06:51:02 | Connected to 0 bootnodes, 10 trusted/persisted nodes 
01 Aug 06:51:02 | Found 38834425 blocks to load from DB starting from current head block 2537574 (0x150233...0bd7d6) 
01 Aug 06:51:02 | Rerunning block after reorg or pruning: 2537575 (0xe42bbf...02092f) 
01 Aug 06:51:03 | Current contract validators (4): [0x6dc0c0be4c8b2dfe750156dc7d59faabfb5b923d, 0x9233042b8e9e03d5dc6454bbbe5aee83818ff103, 0x9e41ba620feba8198369c26351063b26ec5b7c9e, 0xa13d45301207711b7c0328c6b2b64862abfe9b7a]. 
01 Aug 06:51:03 | Rerunning block after reorg or pruning: 2537576 (0xf86129...8dec90) 

....


01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549364 (0x76fd12...95d9e2) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549365 (0x1b0ebf...b1eebc) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549366 (0xaed086...5105b7) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549367 (0x765c0f...43c739) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549368 (0x407347...3e674c) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549369 (0x69f182...ae5d88) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549370 (0xe87884...9cbc2b) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549371 (0x8faab7...03787f) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549372 (0x5df6e1...983eeb) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549373 (0x552d33...c10e02) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549374 (0xad71ce...cf2d02) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549375 (0x7f0aff...722a75) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549376 (0x0fe154...e45448) 
01 Aug 06:54:20 | Rerunning block after reorg or pruning: 2549377 (0x87736f...ee0df7) 

Please include any relevant logs that may help identify the issue.

How to Solve it

1. Stop the node
2. Run the node with --Init.ProcessingEnabled=false flag
3. Make a call to debug_resetHead with the hash of the latest block saved on db:

curl localhost:8545   -X POST   -H "Content-Type: application/json"   --data '{
       "jsonrpc": "2.0",
       "id": 0,
       "method": "debug_resetHead",
       "params": ["0x....head....hash"]
     }'

4. Read the logs and make sure the request didn't fail, you may need to do the request multiple times.
5. Restart the node without --Init.ProcessingEnabled=false flag.