Allow the client to retry (with backoff) when a connection to the server cannot be made #1159
Description
Periodic network glitches and instability should not stop a plan from being able to run. We should add a retry with backoff to the client, being mindful of preventing a retry storm when retry is multiply nested inside e.g. load balancing, UI components. Therefore for now this should just be in the python client for the CLI and python callers (e.g. MX UDC) and retrying is left to any future clients.
Acceptance Criteria
- The server being unavailable for a period <= 1s does not cause an exception in the python client or CLI
- The server being unavailable does not lead to excessive requests >x per second- x = 5? x = 10?
- The client does not keep trying indefinitely, giving up after 3-5 tries
- Added logging and tracing to identify the requests and number of attempts