Conflict on weird fields

[revolunet]

Hi, i dont really understand some conflict errors, maybe someone can help

Heres an example; these fields appear in the diff :

• kapp.k14s.io/nonce : sounds legit
• image : legit as its a new version
• initialDelaySeconds and cpu : i guess its been "rewritten" by kube API

These changes looks legit but make kapp fails, any idea how to prevent this ?

    Updating resource deployment/app-strapi (apps/v1) namespace: env-1000jours-sre-kube-workflow-4y3w36:
      API server says:
        Operation cannot be fulfilled on deployments.apps "app-strapi": the object has been modified; please apply your changes to the latest version and try again (reason: Conflict):
          Recalculated diff:
 11, 10 -     kapp.k14s.io/nonce: "1660057353002414185"
 12, 10 +     kapp.k14s.io/nonce: "1660062422261409721"
223,222 -   progressDeadlineSeconds: 600
225,223 -   revisionHistoryLimit: 10
230,227 -   strategy:
231,227 -     rollingUpdate:
232,227 -       maxSurge: 25%
233,227 -       maxUnavailable: 25%
234,227 -     type: RollingUpdate
237,229 -       creationTimestamp: null
269,260 -         image: something/strapi:sha-3977fb22378f2debdcacf4eeb6dd6f26dab24377
270,260 -         imagePullPolicy: IfNotPresent
271,260 +         image: something/strapi:sha-4ed2921f2fac053671f80fa02b72d124a23fa8c0
276,266 -             scheme: HTTP
279,268 -           successThreshold: 1
285,273 -           protocol: TCP
291,278 -             scheme: HTTP
292,278 +           initialDelaySeconds: 0
297,284 -             cpu: "1"
298,284 +             cpu: 1
300,287 -             cpu: 500m
301,287 +             cpu: 0.5
307,294 -             scheme: HTTP
309,295 -           successThreshold: 1
310,295 -           timeoutSeconds: 1
311,295 -         terminationMessagePath: /dev/termination-log
312,295 -         terminationMessagePolicy: File
316,298 -       dnsPolicy: ClusterFirst
317,298 -       restartPolicy: Always
318,298 -       schedulerName: default-scheduler
319,298 -       securityContext: {}
320,298 -       terminationGracePeriodSeconds: 30

[100mik]

nonce is something only another kapp deploy would change.

Could you tell us more about how is kapp being used here?

Is it via kapp-controller?
Are there more than one users/clients interacting with it using kapp?

To clarify: I mentioned "clients" because kapp might be being used by a CI pipeline, etc.

[praveenrewar]

Hi @revolunet!

Is this error consistent or only happens sometimes?

initialDelaySeconds and cpu : i guess its been "rewritten" by kube API

Are you certain about this? Because the error

the object has been modified; please apply your changes to the latest version and try again (reason: Conflict):

would typically mean that after kapp calculated the diff and before it started applying those changes, something updated the resource in the background and hence we get a conflict. Comparing the recalculated diff and the original diff (can be seen using --diff-changes or -c) might help.

[revolunet]

in this case, kapp is used in a github action and apply some manifests produced with this :

https://github.com/SocialGouv/kube-workflow/blob/27ea1ad20b75fe0b4d5f472fa7d650db8b584436/packages/workflow/src/deploy/index.js#L172-L179

i dont think there is some kapp-controller and yes, many kapp could run in parallel but on different namespaces and this error happen quite often these days, maybe between 5 and 10%.

so i can add --diff-changes=true --diff-context=4 in the code above and get more diff ?

[praveenrewar]

so i can add --diff-changes=true --diff-context=4 in the code above and get more diff ?

Yeah, comparing the original diff with the recalculated diff would give us an idea of the fields that are getting updated in the background and we could then try to figure out a way to resolve it (maybe a rebase rule to not update those fields).

[revolunet]

So here's the full diff for the deployment that fails

2,  2   metadata:
  3,  3     annotations:
  4,  4       deployment.kubernetes.io/revision: "1"
  5     -     field.cattle.io/publicEndpoints: '[{"addresses":["51.103.10.142"],"port":443,"protocol":"HTTPS","serviceName":"xxx-develop-91uqrt:app-strapi","ingressName":"xxx-develop-91uqrt:app-strapi","hostname":"xxx","path":"/","allNodes":false}]'
  6,  5       kapp.k14s.io/change-group: kube-workflow/xxx-91uqrt
  7,  6       kapp.k14s.io/change-group.app-strapi: kube-workflow/app-strapi.xxx-91uqrt
  8,  7       kapp.k14s.io/change-rule.restore: upsert after upserting kube-workflow/restore.env-xxx
  9,  8       kapp.k14s.io/create-strategy: fallback-on-update
 10,  9       kapp.k14s.io/disable-original: ""
 11     -     kapp.k14s.io/identity: v1;xxx-develop-91uqrt/apps/Deployment/app-strapi;apps/v1
 12     -     kapp.k14s.io/nonce: "1660064438212705134"
     10 +     kapp.k14s.io/nonce: "1660122041210559682"
 13, 11       kapp.k14s.io/update-strategy: fallback-on-replace
 14, 12     creationTimestamp: "2022-08-09T17:03:48Z"
 15, 13     generation: 2
 16, 14     labels:
  ...
221,219     resourceVersion: "247149463"
222,220     uid: cf981ae2-2372-4ab8-961d-ce3155975a86
223,221   spec:
224     -   progressDeadlineSeconds: 600
225,222     replicas: 1
226     -   revisionHistoryLimit: 10
227,223     selector:
228,224       matchLabels:
229,225         component: app-strapi
230,226         kubeworkflow/kapp: xxx
231     -   strategy:
232     -     rollingUpdate:
233     -       maxSurge: 25%
234     -       maxUnavailable: 25%
235     -     type: RollingUpdate
236,227     template:
237,228       metadata:
238     -       creationTimestamp: null
239,229         labels:
240,230           application: xxx
241,231           component: app-strapi
242,232           kapp.k14s.io/association: v1.9b1e71da08ebc442e6cdc77552cb740a
267,257               name: strapi-configmap
268,258           - secretRef:
269,259               name: pg-user-develop
270     -         image: xxx/strapi:sha-3ab94da32cb3b479804c[796]
271     -         imagePullPolicy: IfNotPresent
    260 +         image: xxx/strapi:sha-6ea5a193875e11b54f4bf333409d1[808]
272,261           livenessProbe:
273,262             failureThreshold: 15
274,263             httpGet:
275,264               path: /_health
276,265               port: http
277     -             scheme: HTTP
278,266             initialDelaySeconds: 30
279,267             periodSeconds: 5
280     -           successThreshold: 1
281,268             timeoutSeconds: 5
282,269           name: app
283,270           ports:
284,271           - containerPort: 1337
285,272             name: http
286     -           protocol: TCP
287,273           readinessProbe:
288,274             failureThreshold: 15
289,275             httpGet:
290,276               path: /_health
291,277               port: http
292     -             scheme: HTTP
    278 +           initialDelaySeconds: 0
293,279             periodSeconds: 5
294,280             successThreshold: 1
295,281             timeoutSeconds: 1
296,282           resources:
297,283             limits:
298     -             cpu: "1"
    284 +             cpu: 1
299,285               memory: 1Gi
300,286             requests:
301     -             cpu: 500m
    287 +             cpu: 0.5
302,288               memory: 256Mi
303,289           startupProbe:
304,290             failureThreshold: 30
305,291             httpGet:
306,292               path: /_health
307,293               port: http
308     -             scheme: HTTP
309,294             periodSeconds: 5
310     -           successThreshold: 1
311     -           timeoutSeconds: 1
312     -         terminationMessagePath: /dev/termination-log
313     -         terminationMessagePolicy: File
314,295           volumeMounts:
315,296           - mountPath: /app/public/uploads
316,297             name: uploads
317     -       dnsPolicy: ClusterFirst
318     -       restartPolicy: Always
319     -       schedulerName: default-scheduler
320     -       securityContext: {}
321     -       terminationGracePeriodSeconds: 30
322,298         volumes:
323,299         - emptyDir: {}
324,300           name: uploads

[praveenrewar]

And the recalculated diff is the same as what you have shared in the first comment?

If so, I am seeing these 2 differences:

  5     -     field.cattle.io/publicEndpoints: '[{"addresses":["51.103.10.142"],"port":443,"protocol":"HTTPS","serviceName":"xxx-develop-91uqrt:app-strapi","ingressName":"xxx-develop-91uqrt:app-strapi","hostname":"xxx","path":"/","allNodes":false}]'

...snip...

 11     -     kapp.k14s.io/identity: v1;xxx-develop-91uqrt/apps/Deployment/app-strapi;apps/v1

When kapp initially calculates the diff, it tries to remove these fields, but before it could apply the change, the fields are getting removed by something else. Can you think of anything that might be removing these fields?
(I am not sure what could be causing kapp to remove the identity annotation in the first place)

[revolunet]

No its not the same logs, but i can see this on new fails too

 2   metadata:
  3,  3     annotations:
  4,  4       deployment.kubernetes.io/revision: "1"
  5     -     field.cattle.io/publicEndpoints: '[{"addresses":["51.103.10.142"],"port":443,"protocol":"HTTPS","serviceName":"env-xxx-1-5dc5hx:app-strapi","ingressName":"env-xxx-1-5dc5hx:app-strapi","hostname":"backoffice-env-xxx-1-5dc5hx.devr","path":"/","allNodes":false}]'
  6,  5       kapp.k14s.io/change-group: kube-workflow/env-xxx-1-5dc5hx
  7,  6       kapp.k14s.io/change-group.app-strapi: kube-workflow/app-strapi.env-xxx-1-5dc5hx
  8,  7       kapp.k14s.io/change-rule.restore: upsert after upserting kube-workflow/restore.env-xxx-1-5dc5hx
  9,  8       kapp.k14s.io/create-strategy: fallback-on-update
 10,  9       kapp.k14s.io/disable-original: ""
 11     -     kapp.k14s.io/identity: v1;env-xxx-1-5dc5hx/apps/Deployment/app-strapi;apps/v1
 12     -     kapp.k14s.io/nonce: "1660152849643438728"
     10 +     kapp.k14s.io/nonce: "1660164035630293852"
 13, 11       kapp.k14s.io/update-strategy: fallback-on-replace
 14, 12     creationTimestamp: "2022-08-10T17:36:46Z"

Hi, mmmm maybe the cattle.io annotation comes from our rancher when the ingress is provisionned.

can annotations be the cause of a conflict ?

[100mik]

Re: can annotations be the cause of a conflict ?

If an annotation is added after the initial diff, it might lead to this error.
We can configure kapp to use rebaseRules and ask kapp to copy that particular annotation from the resource on the cluster to the resource being applied before calculating the diff.

This would involve adding something like this to your manifests:

apiVersion: kapp.k14s.io/v1alpha1
kind: Config
rebaseRules:
- path: [metadata, annotations, field.cattle.io/publicEndpoints]
  type: copy
  sources: [existing]
  resourceMatchers:
  - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment}

This ensures that the diff remains the same when kapp recalculates the diff before applying the changes.

[100mik]

Re:
10, 9 kapp.k14s.io/disable-original: ""
11 - kapp.k14s.io/identity: v1;env-xxx-1-5dc5hx/apps/Deployment/app-strapi;apps/v1

Was the value of the label being used to identify the app (kubeworkflow/kapp) changed at some point?

I can reproduce something similar by doing something like:

• Create labelled app

kapp deploy -a label:kubeworkflow/kapp=app-name -f - --yes -c << EOF                                                                                                                                                                                                                                                   
apiVersion: v1
kind: ConfigMap
metadata:
  name: asdf
data:
  foo: bar
EOF

(succeeds!)

• Change the label being used to identify apps

kapp deploy -a label:kubeworkflow=app-name -f - --yes -c << EOF                                                                                                                                                                                                                                                   
apiVersion: v1
kind: ConfigMap
metadata:
  name: asdf
data:
  foo: bar
EOF
Target cluster 'https://192.168.64.11:8443' (nodes: minikube)

@@ update configmap/asdf (v1) namespace: default @@
  ...
  4,  4   metadata:
  5     -   annotations:
  6     -     kapp.k14s.io/identity: v1;default//ConfigMap/asdf;v1
  7,  5     creationTimestamp: "2022-08-10T22:27:10Z"
  8,  6     labels:
  9,  7       kapp.k14s.io/association: v1.e623db5b5c0d55f2a39d467ca3165a7f
 10     -     kubeworkflow/kapp: app-name
      8 +     kubeworkflow: app-name
 11,  9     managedFields:
 12, 10     - apiVersion: v1

Changes

Namespace  Name  Kind       Age  Op      Op st.  Wait to    Rs  Ri  
default    asdf  ConfigMap  31s  update  -       reconcile  ok  -  

Op:      0 create, 0 delete, 1 update, 0 noop, 0 exists
Wait to: 1 reconcile, 0 delete, 0 noop

3:57:41AM: ---- applying 1 changes [0/1 done] ----
3:57:41AM: update configmap/asdf (v1) namespace: default
3:57:41AM: ---- waiting on 1 changes [0/1 done] ----
3:57:41AM: ok: reconcile configmap/asdf (v1) namespace: default
3:57:41AM: ---- applying complete [1/1 done] ----
3:57:41AM: ---- waiting complete [1/1 done] ----

Succeeded

[revolunet]

Was the value of the label being used to identify the app (kubeworkflow/kapp) changed at some point?

its possible that it changed at some point in the past yes, but now its stable