Version :
All Domino Versions hosted on AWS
Issue:
When attempting to run any kubectl command from the central node you are confronted with the following message:
[ec2-user@ip-10-0-0-14 ~]$ kubectl get pods -A
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::841428822465:user/emeaplay5 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::841428822465:role/emeaplay214-EksStackNested-eksCreationRoleF889249E-REZ4GO9H1DRY
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 254
[ec2-user@ip-10-0-0-14 ~]$
Root Cause
The user which kubectl makes use of in your AWS account lacks the necessary Role/Policy permissions to call sts:AssumeRole, or the trust relationship on the AWS account is incorrectly setup.
Resolution
The resolution is as follows:
1. Creation of new IAM user
Create a new IAM user and ensure that you have an IAM user setup and copy the users ARN Details, Access Key and Secret Access Key:
AWS Console > IAM > Users > Add User > Click Next > Select existing policies now or attach later > Select Next & Review > Create User
Note: when visible, select the Access Key and Secret Access Key - The Secret Access Key is only visible at the time of creating the secret via the Security Credential Tab, AWS does not allow retrieval of a secret access key after its initial creation.
Ensure that you copy the users ARN (Amazon Resource Name)
AWS Console > IAM > Users > Select User
2. Creating a Custom Policy for access
Create a Custom Policy to give you the correct permissions for your role. (see attached AWSDescribePol.txt).
AWS Console > IAM > Policies > Create Policy > Select the Json tab and copy/paste the json policy in the attached policy. For the purposes of this article, Resource access has been given as '*' but this can / should be restricted.
3. Trust Role Policy
Create a custom 'Trust Role' policy:
AWS Console > Roles > Create Role> Select Customer trust role policy > Ensure the Json looks like the snippet below and substitute in the ARN of the IAM user:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::841428822465:user/emeaplay5"
},
"Action": "sts:AssumeRole"
}
]
}
Ensure the Trust Role has permissions to these policies :
AmazonEC2ContainerServiceAutoscaleRole
AWSDescribePol (your newly Created Policy)
Click Next and Create Role
4. Configure AWS-CLI
On your Linux set your environment to ensure aws-cli can authenticate correctly with AWS, you can do this as follows (remember to substitute in your own Access Key ID, Secret and region)
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
Verify your credentials were set:
[ec2-user@ip-10-0-0-14 ~]$ aws sts get-caller-identity
{
"UserId": "AKIAIOSFODNN7EXAMPLE",
"Account": "841428822465",
"Arn": "arn:aws:iam::841428822465:user/emeaplay5"
}
5. Update aws-auth ConfigMap to include the new IAM User
Ask the cluster owner or admin to add your IAM user or role to aws-auth ConfigMap.
$ kubectl edit configmap aws-auth --namespace kube-system
Add the IAM user to mapUsers. For example (indents matter):
mapUsers: |
- userarn: arn:aws:iam::841428822465:user/emeaplay5
username: emeaplay5
groups:
- system:masters
6. Test Kubectl Access
Test out your access with kubectl:
[ec2-user@ip-10-0-0-14 ~]$ kubectl get pods -A | tail -n 2
kube-system kube-proxy-tcpx6 1/1 Running 0 9h
tigera-operator tigera-operator-6fbb48778f-bjps7 1/1 Running 1 (9h ago) 20h
[ec2-user@ip-10-0-0-14 ~]$
Notes:
Configuring AWS-CLI
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html
Assuming Roles:
https://aws.amazon.com/premiumsupport/knowledge-center/iam-assume-role-cli/
Updating MapRoles
https://aws.amazon.com/premiumsupport/knowledge-center/eks-api-server-unauthorized-error/
Comments
0 comments
Please sign in to leave a comment.