scheduler-run trigger: support one-off containers

[josegonzalez]

We'll need to figure out if we can support both interactive and non-interactive containers, or just non-interactive.

https://www.kubernetes.io/docs/concepts/workloads/pods/init-containers

[josegonzalez]

Init containers don't seem to be what we want - they are more like "startup" processes.

What we should probably do is:

• Implement enter (scheduler-enter: add and implement #21)
• Create an arbitrary deployment with the command specified as the entrypoint. This should be scaled to 1.
  • Shouldn't have any probes and a restartPolicy of never
• Enter the app container in the aforementioned deployment
• Have a command to list all run containers
  • run:list might be useful regardless, and easily implementable upstream
• Have a command to kill run deployments - run:kill $APP $RUN_ID
• Have a cron job that kills run deployments older than 24 hours - run:cleanup [$APP]
  • Maybe this can kill anything without an active exec attached?

[josegonzalez]

Open question: How do we handle resources attached to a run container? Could be done via the resource command with some explicit defaults, and we'd add run to a list of Process types that are "special"? Alternatively, they could be flags for the run command, but that would be harder to share with other schedulers.

[josegonzalez]

Instead of a deployment, we should use a Job: https://codeblog.dotsandbrackets.com/one-off-kubernetes-jobs/