Add finalizer logic to agents, modelconfig and MCP server

[nimishamehta5]

Adding finalizer logic to components to address #479.
Please review if the cleanup logic is correct, it's mostly untouched, this PR just adds the logic of ensuring and removing finalizers in the reconciler.

The PR also adds a test file for the reconciler for now only covering the finalizer & cleanup logic.

Tested by running the controller locally -

2025-08-18T08:34:30-07:00	INFO	reconciler	Added finalizer to agent	{"namespace": "kagent", "name": "test-agent"}

~ kubectl get agent test-agent -o yaml | grep -A 3 finalizers
  finalizers:
  - kagent.dev/agent-cleanup
  generation: 1
  name: test-agent

~ kubectl get agent test-agent -o yaml | grep -A 10 status
status:
  conditions:
  - lastTransitionTime: "2025-08-18T15:35:41Z"
    message: ""
    reason: AgentReconciled
    status: "True"
    type: Accepted
  - lastTransitionTime: "2025-08-18T15:35:41Z"
    message: Deployment is not ready, 0/1 pods are ready
    reason: DeploymentNotReady
    status: "False"
    type: Ready
  configHash: zcUvKidj/++BrGS6ZturdvZ5RwYZz/pvIsbOcJXvaOw=
  observedGeneration: 1

2025-08-18T08:36:45-07:00	INFO	reconciler	Removed finalizer from agent after cleanup	{"namespace": "kagent", "name": "test-agent"}



2025-08-18T20:04:41-07:00	INFO	reconciler	Added finalizer to ModelConfig	{"namespace": "kagent", "name": "default-model"}

➜  ~ k delete modelconfig default-model
modelconfig.kagent.dev "default-model" deleted

2025-08-18T20:07:30-07:00	INFO	reconciler	Performing ModelConfig cleanup	{"namespace": "kagent", "name": "default-model"}
2025-08-18T20:07:30-07:00	INFO	reconciler	Removed finalizer from ModelConfig after cleanup	{"namespace": "kagent", "name": "default-model"}

[yuval-k]

IMHO while finalizers are a solution to the problem, they are more often than not more trouble than they are worth, as they can prevent resource and namespace deletion, which users find not intuitive.

The problem we want to solve here is to clean-up agents from our DB when agents get deleted from k8s, right?
What if instead of finalizers, we do a reconcile loop on start-up (and maybe on an interval if we think that's needed), that compares the agent in the db to the agents in the cluster and reconciles the db to match the cluster?

[EItanya]

What if instead of finalizers, we do a reconcile loop on start-up (and maybe on an interval if we think that's needed), that compares the agent in the db to the agents in the cluster and reconciles the db to match the cluster?

I agree that finalizers can get a little messy, but I'm also a little worried about the performance implications of running full system reconciles on a schedule. Is there a middle ground, non-ns blocking finalizers or some such?

[nimishamehta5]

I don't think schedule-based reconciliation is needed here. We could add a cleanup reconciler of sorts that watches agents / other resources with a deletionTimestamp != nil predicate, and does the necessary cleanup in the database but it feels a little heavy handed to do this. As for performance, I don't think it will have a huge performance hit if we limit the items queued up for reconciliation. However, my preference is still finalizers.

Any other middle-ground ideas @yuval-k @EItanya?