CrashLoopBackOff due to EC2MetadataError: failed to make EC2Metadata request, status code: 401

[SarumObjects]

I've followed the guidelines here: https://github.com/zalando-incubator/kube-ingress-aws-controller/blob/master/deploy/kops.md but kube-ingress-aws-controller restarts every 2-10 minutes.
When I follow the log of the pod I get this error "EC2MetadataError: failed to make EC2Metadata request"
I have rebuilt the cluster and deleted it several times and cannot create the load balancer or target groups - although I have in the past. One of our clusters is still running so I have compared it in detail - and have got no differences except in the name.

We are blocked. This is our development environment. The instances have public & private IPs and the VPCs & SGs have been generated correctly.

Where should I look now please?
John

[AlexanderYastrebov]

Hello. What is the controller version you are using? Could you provide a more detailed error log message?

[szuecs]

@SarumObjects What kind of controller version and AWS integration do you use?
Kube2iam and all others had issues with jtblin/kube2iam#130.
The error message looks like aws/aws-sdk-go#870 and this is quite old and should be fixed by recent kubernetes AWS iam integrations.

[SarumObjects]

@szuecs v0.12 (I downloaded :latest) and created the cluster with Kops (1.22.22). I've built several similar clusters in the last 24 months (we're running one as prod) and I have burned and built a QA cluster (same script) some 4 times. The cluster validates successfully but when I install kube-ingress-aws-controller/skipper (same manifest as our prod cluster - different name) I get this error: "EC2MetadataError: failed to make EC2Metadata request"
@AlexanderYastrebov : This is the total log! I don't know how to debug this controller. I've searched the documentation for 'debug and 'verbose' - and I am stuck for over a week.

[szuecs]

@SarumObjects I think just paste the logs here until the crash would be great!

Latest version meaning v0.12.12?
We just merged updates to aws-sdk, maybe you want to try v0.12.14, when it's released in some minutes (automated process).

Also interesting would be if you would past the output of kubectl describe pods kube-ingress-aws-controller-....

We don't have really knowledge about Kops. Is the version you are referring to the same as the Kubernetes version?

[SarumObjects]

@szuecs the 'latest' still restarts.
here's the output from kubectl describe pods kube-ingress-aws-controller-..
kiac-describe.txt

kops is version 1.22.2
kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:42:41Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

[AlexanderYastrebov]

@SarumObjects Could you get ingress controller logs as well (kubectl logs kube-ingress-aws-controller-...)?

[SarumObjects]

@AlexanderYastrebov this the command and the complete log:
kubectl -n kube-system logs -f kube-ingress-aws-controller-5fbcd9fff8-vqrvg
time="2021-12-03T11:52:45Z" level=info msg="starting /kube-ingress-aws-controller v0.12.14"
time="2021-12-03T11:54:48Z" level=fatal msg="EC2MetadataError: failed to make EC2Metadata request\n\n\tstatus code: 401, request id:

[AlexanderYastrebov]

Could you try to run with --debug option (it would print more details in the logs)?
401 suggests some kind of problem with AWS credentials.

[SarumObjects]

there's no --debug at the command line.

[AlexanderYastrebov]

there's no --debug at the command line.

~$ docker run -it --rm registry.opensource.zalan.do/teapot/kube-ingress-aws-controller:latest --help
INFO[0000] starting /kube-ingress-aws-controller v0.12.14 
usage: kube-ingress-aws-controller [<flags>]

Flags:
  --help                         Show context-sensitive help (also try --help-long and --help-man).
  --version                      Print version and exit
  --debug                        Enables debug logging level
...

kube-ingress-aws-controller/controller.go

Line 191 in 5c66137

kingpin.Flag("debug", "Enables debug logging level").Default("false").BoolVar(&debugFlag)

[SarumObjects]

kubectl -n kube-system logs -f pod/kube-ingress-aws-controller-65775b947-dx9tl --ignore-errors=false
time="2021-12-03T14:50:47Z" level=debug msg=aws.NewAdapter
time="2021-12-03T14:50:47Z" level=debug msg=aws.ec2metadata.GetMetadata
2021/12/03 14:50:47 DEBUG: Request ec2metadata/GetToken Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.42.16 (go1.17.1; linux; amd64)
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip

---

time="2021-12-03T14:50:47Z" level=info msg="starting /kube-ingress-aws-controller v0.12.14"
2021/12/03 14:52:50 DEBUG: Send Request ec2metadata/GetToken failed, attempt 0/3, error RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": read tcp 100.96.4.21:34662->169.254.169.254:80: read: connection reset by peer
2021/12/03 14:52:50 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/instance-id HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.42.16 (go1.17.1; linux; amd64)
Accept-Encoding: gzip

---

2021/12/03 14:52:50 DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/plain
Date: Fri, 03 Dec 2021 14:52:50 GMT
Server: EC2ws
Content-Length: 0

---

2021/12/03 14:52:50 DEBUG: Validate Response ec2metadata/GetMetadata failed, attempt 0/3, error EC2MetadataError: failed to make EC2Metadata request

status code: 401, request id: 

time="2021-12-03T14:52:50Z" level=fatal msg="EC2MetadataError: failed to make EC2Metadata request\n\n\tstatus code: 401, request id: "

[szuecs]

This log here:

 caused by: Put "http://169.254.169.254/latest/api/token": read tcp 100.96.4.21:34662->169.254.169.254:80: read: connection reset by peer

169.254.169.254 is the metadata service by AWS. It sent a TCP RST packet, instead of sending us the data required to access AWS APIs.

What Kubernetes IAM integration do you use?
For me this looks like not to be an issue by the controller, but either AWS or the IAM integration that should support in getting the IAM done.
Maybe also your EC2 nodes have not the right permissions to access metadata service to access AWS APIs with sts::assumeRole, which is required for all Kubernetes AWS IAM integrations.

[SarumObjects]

Thats helpful. I'll look into the IAM permissions.

[szuecs]

@SarumObjects let us know what the error was to share with other folks that might find this issue. After that we can close it.