terraform-aws-modules / terraform-aws-eks

AWS EKS Terraform module Terraform module which creates Amazon EKS (Kubernetes) resources Documentation • Frequently Asked Questions • Compute Resources • User Data • Network Connectivity • Upgrade Guides • Upgrade to v17.x • Upgrade to v18.x • Upgrade to v19.x • Upgrade to v20.x • Upgrade to v21.x External Documentation Please note that we strive to provide a comprehensive suite of documentation for __*configuring and utilizing the module(s)*__ defined here, and that documentation regarding EKS (including EKS managed node group, self managed node group, and Fargate profile) and/or Kubernetes features, usage, etc. are better left up to their respective sources: • AWS EKS Documentation • Kubernetes Documentation Usage EKS Auto Mode > [!CAUTION] > Due to the current EKS Auto Mode API, to disable EKS Auto Mode you will have to explicity set: > > > > If you try to disable by simply removing the block, this will fail to disable EKS Auto Mode. Only after applying with can you then remove the block from your configurations. EKS Auto Mode - Custom Node Pools Only EKS Provisioned Control Plane EKS Provisioned Control Plane allows you to provision a control plane with increased capacity for larger workloads. Valid tier values are , , , and . EKS Managed Node Group Cluster Access Entry When enabling , EKS will automatically create an access entry for the IAM role(s) used by managed node group(s) and Fargate profile(s). There are no additional actions required by users. For self-managed node groups and the Karpenter sub-module, this project automatically adds the access entry on behalf of users so there are no additional actions required by users. On clusters that were created prior to cluster access management (CAM) support, there will be an existing access entry for the cluster creator. This was previously not visible when using ConfigMap, but will become visible when access entry is enabled. EKS Hybrid Nodes Bootstrap Cluster Creator Admin Permissions Setting the is a one time operation when the cluster is created; it cannot be modified later through the EKS API. In this project we are hardcoding this to . If users wish to achieve the same functionality, we will do that through an access entry which can be enabled or disabled at any time of their choosing using the variable Enabling EFA Support When enabling EFA support via , there are two locations this can be specified - one at the cluster level, and one at the node group level. Enabling at the cluster level will add the EFA required ingress/egress rules to the shared security group created for the node group(s). Enabling at the node group level will do the following (per node group where enabled): • All EFA interfaces supported by the instance will be exposed on the launch template used by the node group • A placement group with per EFA requirements is created and passed to the launch template used by the node group • Data sources will reverse lookup the availability zones that support the instance type selected based on the subnets provided, ensuring that only the associated subnets are passed to the launch template and therefore used by the placement group. This avoids the placement group being created in an availability zone that does not support the instance type selected. > [!TIP] > Use the aws-efa-k8s-device-plugin Helm chart to expose the EFA interfaces on the nodes as an extended resource, and allow pods to request the interfaces be mounted to their containers. > > The EKS AL2 GPU AMI comes with the necessary EFA components pre-installed - you just need to expose the EFA devices on the nodes via their launch templates, ensure the required EFA security group rules are in place, and deploy the in order to start utilizing EFA within your cluster. Your application container will need to have the necessary libraries and runtime in order to utilize communication over the EFA interfaces (NCCL, aws-ofi-nccl, hwloc, libfabric, aws-neuornx-collectives, CUDA, etc.). If you disable the creation and use of the managed node group custom launch template ( and/or ), this will interfere with the EFA functionality provided. In addition, if you do not supply an for self-managed node group(s), or for the managed node group(s), this will also interfere with the functionality. In order to support the EFA functionality provided by , you must utilize the custom launch template created/provided by this module, and supply an / for the respective node group. The logic behind supporting EFA uses a data source to lookup the instance type to retrieve the number of interfaces that the instance supports in order to enumerate and expose those interfaces on the launch template created. For managed node groups where a list of instance types are supported, the first instance type in the list is used to calculate the number of EFA interfaces supported. Mixing instance types with varying number of interfaces is not recommended for EFA (or in some cases, mixing instance types is not supported - i.e. - p5.48xlarge and p4d.24xlarge). In addition to exposing the EFA interfaces and updating the security group rules, a placement group is created per the EFA requirements and only the availability zones that support the instance type selected are used in the subnets provided to the node group. In order to enable EFA support, you will have to specify on both the cluster and each node group that you wish to enable EFA support for: Examples • EKS Auto Mode: EKS Cluster with EKS Auto Mode • EKS Capabilities: EKS Cluster with EKS Capabilities • EKS Hybrid Nodes: EKS Cluster with EKS Hybrid nodes • EKS Managed Node Group: EKS Cluster with EKS managed node group(s) • Karpenter: EKS Cluster with Karpenter provisioned for intelligent data plane management • Self Managed Node Group: EKS Cluster with self-managed node group(s) Contributing We are grateful to the community for contributing bugfixes and improvements! Please see below to learn how you can take part. •…