All Domino Versions hosted on AWS
When attempting to run any kubectl command from the central node you are confronted with the following message:
[ec2-user@ip-10-0-0-14 ~]$ kubectl get pods -A
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::841428822465:user/emeaplay5 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::841428822465:role/emeaplay214-EksStackNested-eksCreationRoleF889249E-REZ4GO9H1DRY
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 254
The user which kubectl makes use of in your AWS account lacks the necessary Role/Policy permissions to call sts:AssumeRole, or the trust relationship on the AWS account is incorrectly setup.
The resolution is as follows:
1. Creation of new IAM user
Create a new IAM user and ensure that you have an IAM user setup and copy the users ARN Details, Access Key and Secret Access Key:
AWS Console > IAM > Users > Add User > Select: Access Key Programmatic Access > Click Next 'No permissions needed at this time' > Select Next & Review > Create User
Note: when visible, select the Access Key and Secret Access Key - The Secret Access Key is only visible at the time of creating the user, AWS does not allow retrieval of a secret access key after its initial creation.
Ensure that you copy the users ARN (Amazon Resource Name)
AWS Console > IAM > Users > Select User
2. Creating a Custom Policy for access
Create a Custom Policy to give you the correct permissions for your role. (AWSDescribePol.txt).
AWS Console > IAM > Policies > Create Policy > Select the Json tab and copy/paste the json policy in the attached policy. For the purposes of this article, Resource access has been given as '*' but this can / should be restricted.
3. Trust Role Policy
Create a custom 'Trust Role' policy:
AWS Console > Roles > Create Policy > Select Customer trust role policy > Ensure the Json looks like the snippet below and substitute in the ARN of the IAM user:
Ensure the Trust Role has permissions to these policies :
AWSDescribePol (Newly Created Policy)
Click Next and Create Role
4. Configure AWS-CLI
On your Linux set your environment to ensure aws-cli can authenticate correctly with AWS, you can do this as follows (remember to substitute in your own Access Key ID, Secret and region)
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
Verify your credentials were set:
[ec2-user@ip-10-0-0-14 ~]$ aws sts get-caller-identity
5. Update aws-auth ConfigMap to include the new IAM User
Ask the cluster owner or admin to add your IAM user or role to aws-auth ConfigMap.
$ kubectl edit configmap aws-auth --namespace kube-system
Add the IAM user to mapUsers. For example:
- userarn: arn:aws:iam::841428822465:user/emeaplay5
6. Test Kubectl Access
Test out your access with kubectl:
[ec2-user@ip-10-0-0-14 ~]$ kubectl get pods -A | tail -n 2
kube-system kube-proxy-tcpx6 1/1 Running 0 9h
tigera-operator tigera-operator-6fbb48778f-bjps7 1/1 Running 1 (9h ago) 20h