Provide logging and status checking on WorkerDispatcher._subprocess

[olliesilvester]

i04 was recently seeing a 504 gateway timeout when triggering BlueAPI from GDA. Following the stack of logs, it seems like BlueAPI was hanging after successfully receiving a POST request (for submitting a task).

The most likely cause of this hanging appears to be an issue with Blueapi's WorkerDispatcher._subprocess. However, there is currently no way to get information about its status. It would be useful to add a way to get the status of this subprocess (maybe a heartbeat?), as well as some logging

Acceptance Criteria

• Logging statements added for this subprocess, as well as a way to get its status

[callumforrester]

I presume the 504 happened because the main process tried to contact the subprocess and found it was in a bad state, so do we just need to improve the error message when that happens?

[olliesilvester]

It's hard to tell exactly what happened, but yes likely something like this. We only saw that BlueAPI got the request, but never completed it.

Ideally the pod would automatically restart if it finds itself in this bad state