Add support for OCI containers (#292)

* Add support for OCI containers

Update the configure-eda.sh script to install rootless docker for compute nodes.

Add playbook and script to configure users to run rootless docker.

Add support for pyxis and enroot from:

https://docs.aws.amazon.com/en_us/parallelcluster/latest/ug/tutorials_11_running-containerized-jobs-with-pyxis.html

Resolves #292

* Add support for pyxis containers

Build and install enroot and pyxis on external login nodes.

Configure /etc/subuid and /etc/subgid for AD users.

Resolves #259

Author: cartalla

docs/containers.md

@@ -0,0 +1,142 @@
+# Containers
+
+Slurm supports running jobs in unprivileged containers a couple of different ways.
+It natively supports running jobs in unprivileged [Open Container Initiative (OCI) containers](https://slurm.schedmd.com/containers.html).
+Starting with ParallelCluster 3.11.1, it also supports running docker containers using the [Pyxis SPANK plugin](https://github.com/NVIDIA/pyxis) which uses [enroot](https://github.com/NVIDIA/enroot) to run unprivileged containers.
+I will describe using Pyxis first because it is easier than using OCI containers.
+
+**Note**: Most EDA tools are not containerized.
+Some won't run in containers and some may run in a container, but not correctly.
+I recommend following the guidance of your EDA vendor and consult with them.
+
+I've seen a couple of main motivations for using containers for EDA tools.
+The first is because orchestration tools like Kubernetes and AWS Batch require jobs to run in containers.
+The other is to have more flexibility managing the run time environment of the tools.
+Since the EDA tools themselves aren't containerized, the container is usually used to manage file system mounts and packages that are used by the tools.
+If new packages are required by a new tool, then it is easy to update and distribute a new version of the container.
+Another reason is to use legacy OS distributions on an instance running a newer distribution.
+
+## Using Pyxis
+
+The enroot and Pyxis packages were developed by NVIDIA to make it easier to run containers on Slurm compute nodes.
+ParallelCluster started installing enroot and Pyxis in version 3.11.1 so that you can [run containerized jobs with Pyxis](https://docs.aws.amazon.com/en_us/parallelcluster/latest/ug/tutorials_11_running-containerized-jobs-with-pyxis.html).
+
+To configure Slurm to use the Pyxis plugin, set the **slurm/ParallelClusterConfig/EnablePyxis** parameter to **true** and create or update your cluster.
+This will configure the head node to use the Pyxis plugin.
+It will also configure your external login nodes to install, configure, and use enroot and the Pyxis plugin.
+
+### Running a containerized job using Pyxis
+
+With Pyxis configured in your cluster, you have new options in srun and sbatch to specify a container image.
+
+```
+# Submitting an interactive job
+srun -N 2 --container-image docker://rockylinux:8 hostname
+
+# Submitting a batch job
+sbatch -N 2 --wrap='srun --container-image docker://rockylinux:8 hostname'
+```
+
+## Using OCI containers
+
+Slurm supports [running jobs in unprivileged OCI containers](https://slurm.schedmd.com/containers.html).
+OCI is the [Open Container Initiative](https://opencontainers.org/), an open governance structure with the purpose of creating open industry standards around container formats and runtimes.
+
+I'm going to document how to add OCI support to your EDA Slurm cluster.
+
+**NOTE**: Rootless docker requires user-specific setup for each user that will run the containers.
+For this reason, it is much easier to use Pyxis.
+
+### Configure rootless docker on login and compute nodes
+
+The login and compute nodes must be configured to use an unprivileged container runtime.
+
+Run the following script as root to install rootless Docker.
+
+```
+/opt/slurm/${ClusterName}/config/bin/install-rootless-docker.sh
+```
+
+The script [installs the latest Docker from the Docker yum repo](https://docs.docker.com/engine/install/rhel/).
+
+The creation of a compute node AMI with rootless docker installed has been automated in the [creation of a custom compute node AMI](custom-amis.md).
+Use one of the build config files with **docker** in the name to create a custom AMI and configure your cluster to use it.
+
+### Per user configuration
+
+Next, [configure Docker to run rootless](https://docs.docker.com/engine/security/rootless/) by running the following script as the user that will be running Docker.
+
+```
+dockerd-rootless-setuptool.sh
+```
+
+Each user that will run Docker must have an entry in `/etc/subuid` and `/etc/subgid`.
+The creates_users_groups_json.py script will create `/opt/slurm/config/subuid` and `/opt/slurm/config/subgid` and the compute nodes will copy them to `/etc/subuid` and `/etc/subgid`.
+
+You must configure docker to use a non-NFS storage location for storing images.
+
+`~/.config/docker/daemon.json`:
+
+```
+{
+    "data-root": "/var/tmp/${USER}/containers/storage"
+}
+```
+
+### Create OCI Bundle
+
+Each container requires an [OCI bundle](https://slurm.schedmd.com/containers.html#bundle).
+
+The bundle directories can be stored on NFS and shared between users.
+For example, you could create an oci-bundles directory on your shared file system.
+
+This shows how to create an ubuntu bundle.
+You can do this as root with the docker service running, but it would be better to run
+it using rootless Docker.
+
+```
+export OCI_BUNDLES_DIR=~/oci-bundles
+export IMAGE_NAME=ubuntu
+export BUNDLE_NAME=ubuntu
+mkdir -p $OCI_BUNDLES_DIR
+cd $OCI_BUNDLES_DIR
+mkdir -p $BUNDLE_NAME
+cd $BUNDLE_NAME
+docker pull $IMAGE_NAME
+docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar
+mkdir rootfs
+tar -C rootfs -xf $IMAGE_NAME.tar
+runc spec --rootless
+runc run containerid1
+```
+
+The same process works for Rocky Linux 8.
+
+```
+export OCI_BUNDLES_DIR=~/oci-bundles
+export IMAGE_NAME=rockylinux:8
+export BUNDLE_NAME=rockylinux8
+mkdir -p $OCI_BUNDLES_DIR
+cd $OCI_BUNDLES_DIR
+mkdir -p $BUNDLE_NAME
+cd $BUNDLE_NAME
+docker pull $IMAGE_NAME
+docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar
+mkdir rootfs
+tar -C rootfs -xf $BUNDLE_NAME.tar
+runc spec --rootless
+runc run containerid2
+```
+
+### Run a bundle on Slurm using OCI container
+
+```
+export OCI_BUNDLES_DIR=~/oci-bundles
+export BUNDLE_NAME=rockylinux8
+
+srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty hostname
+
+srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty bash
+
+sbatch -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --wrap hostname
+```

mkdocs.yml

@@ -15,6 +15,7 @@ nav:
   - 'job_preemption.md'
   - 'rest_api.md'
   - 'onprem.md'
+  - 'containers.md'
  # - 'federation.md'
   - 'delete-cluster.md'
  # - 'implementation.md'

source/cdk/cdk_slurm_stack.py

@@ -53,7 +53,7 @@
 import boto3
 from botocore.exceptions import ClientError
 import config_schema
-from config_schema import get_PARALLEL_CLUSTER_LAMBDA_RUNTIME, get_PARALLEL_CLUSTER_MUNGE_VERSION, get_PARALLEL_CLUSTER_PYTHON_VERSION, get_PC_SLURM_VERSION, get_SLURM_VERSION
+from config_schema import get_PARALLEL_CLUSTER_ENROOT_VERSION, get_PARALLEL_CLUSTER_LAMBDA_RUNTIME, get_PARALLEL_CLUSTER_MUNGE_VERSION, get_PARALLEL_CLUSTER_PYTHON_VERSION, get_PARALLEL_CLUSTER_PYXIS_VERSION, get_PC_SLURM_VERSION, get_SLURM_VERSION
 from constructs import Construct
 from copy import copy, deepcopy
 from hashlib import sha512
@@ -289,6 +289,10 @@ def check_config(self):
                         self.mount_home_src = mount_dict['src']
                         logger.info(f"Mounting /home from {self.mount_home_src} on compute nodes")
 
+        if self.config['slurm']['ParallelClusterConfig']['EnablePyxis'] and not config_schema.PARALLEL_CLUSTER_SUPPORTS_PYXIS(self.PARALLEL_CLUSTER_VERSION):
+            logger.error(f"Cannot EnablePyxis before ParaallelCluster version {config_schema.PARALLEL_CLUSTER_SUPPORTS_PYXIS_VERSION}")
+            config_errors += 1
+
         # Check OS
         if self.config['slurm']['ParallelClusterConfig']['Image']['Os'] not in config_schema.get_PARALLEL_CLUSTER_ALLOWED_OSES(self.config):
             logger.error(f"{self.config['slurm']['ParallelClusterConfig']['Image']['Os']} is not supported in ParallelCluster version {self.PARALLEL_CLUSTER_VERSION}.")
@@ -498,7 +502,7 @@ def check_config(self):
 
         if 'Xio' in self.config['slurm']:
             if self.config['slurm']['ParallelClusterConfig']['Architecture'] != 'x86_64':
-                logger.error("Xio is only supported on x86_64 architecture, not {self.config['slurm']['ParallelClusterConfig']['Architecture']}")
+                logger.error(f"Xio is only supported on x86_64 architecture, not {self.config['slurm']['ParallelClusterConfig']['Architecture']}")
                 config_errors += 1
 
         if config_errors:
@@ -1254,11 +1258,13 @@ def create_parallel_cluster_assets(self):
         # Additions or deletions to the list should be reflected in config_scripts in on_head_node_start.sh.
         files_to_upload = [
             'config/bin/configure-eda.sh',
+            'config/bin/configure-rootless-docker.sh',
             'config/bin/create_or_update_users_groups_json.sh',
             'config/bin/create_users_groups_json.py',
             'config/bin/create_users_groups_json_configure.sh',
             'config/bin/create_users_groups_json_deconfigure.sh',
             'config/bin/create_users_groups.py',
+            'config/bin/install-rootless-docker.sh',
             'config/bin/on_head_node_start.sh',
             'config/bin/on_head_node_configured.sh',
             'config/bin/on_head_node_updated.sh',
@@ -1532,6 +1538,7 @@ def create_parallel_cluster_lambdas(self):
                 'ConfigureEdaScriptS3Url': self.custom_action_s3_urls['config/bin/configure-eda.sh'],
                 'ErrorSnsTopicArn': self.config.get('ErrorSnsTopicArn', ''),
                 'ImageBuilderSecurityGroupId': self.imagebuilder_sg.security_group_id,
+                'InstallDockerScriptS3Url': self.custom_action_s3_urls['config/bin/install-rootless-docker.sh'],
                 'ParallelClusterVersion': self.config['slurm']['ParallelClusterConfig']['Version'],
                 'Region': self.cluster_region,
                 'SubnetId': self.config['SubnetId'],
@@ -2273,6 +2280,7 @@ def get_instance_template_vars(self, instance_role):
                 "cluster_name": cluster_name,
                 "region": self.cluster_region,
                 "time_zone": self.config['TimeZone'],
+                "parallel_cluster_version": self.PARALLEL_CLUSTER_VERSION
             }
             instance_template_vars['default_partition'] = 'batch'
             instance_template_vars['file_system_mount_path'] = '/opt/slurm'
@@ -2287,9 +2295,12 @@ def get_instance_template_vars(self, instance_role):
                 instance_template_vars['accounting_storage_host'] = self.config['slurm']['ParallelClusterConfig']['Slurmdbd']['Host']
             else:
                 instance_template_vars['accounting_storage_host'] = ''
+            instance_template_vars['enable_pyxis'] = self.config['slurm']['ParallelClusterConfig']['EnablePyxis']
             instance_template_vars['licenses'] = self.config['Licenses']
+            instance_template_vars['parallel_cluster_enroot_version'] = get_PARALLEL_CLUSTER_ENROOT_VERSION(self.config)
             instance_template_vars['parallel_cluster_munge_version'] = get_PARALLEL_CLUSTER_MUNGE_VERSION(self.config)
             instance_template_vars['parallel_cluster_python_version'] = get_PARALLEL_CLUSTER_PYTHON_VERSION(self.config)
+            instance_template_vars['parallel_cluster_pyxis_version'] = get_PARALLEL_CLUSTER_PYXIS_VERSION(self.config)
             instance_template_vars['primary_controller'] = True
             instance_template_vars['slurm_uid'] = self.config['slurm']['SlurmUid']
             instance_template_vars['slurmctld_port'] = self.slurmctld_port
@@ -2310,8 +2321,11 @@ def get_instance_template_vars(self, instance_role):
                 instance_template_vars['xio_config'] = self.config['slurm']['Xio']
                 instance_template_vars['xio_config']['ExtraMounts'] = self.config['slurm'].get('storage', {}).get('ExtraMounts', [])
         elif instance_role == 'ParallelClusterExternalLoginNode':
+            instance_template_vars['enable_pyxis']                       = self.config['slurm']['ParallelClusterConfig']['EnablePyxis']
             instance_template_vars['slurm_version']                      = get_SLURM_VERSION(self.config)
+            instance_template_vars['parallel_cluster_enroot_version']    = get_PARALLEL_CLUSTER_ENROOT_VERSION(self.config)
             instance_template_vars['parallel_cluster_munge_version']     = get_PARALLEL_CLUSTER_MUNGE_VERSION(self.config)
+            instance_template_vars['parallel_cluster_pyxis_version']     = get_PARALLEL_CLUSTER_PYXIS_VERSION(self.config)
             instance_template_vars['slurmrestd_port']                    = self.slurmrestd_port
             instance_template_vars['file_system_mount_path']             = f'/opt/slurm/{cluster_name}'
             instance_template_vars['slurm_base_dir']                     = f'/opt/slurm/{cluster_name}'

source/cdk/config_schema.py

@@ -272,6 +272,14 @@ def get_PARALLEL_CLUSTER_MUNGE_VERSION(config):
     parallel_cluster_version = get_parallel_cluster_version(config)
     return PARALLEL_CLUSTER_MUNGE_VERSIONS[parallel_cluster_version]
 
+def get_PARALLEL_CLUSTER_ENROOT_VERSION(config):
+    parallel_cluster_version = get_parallel_cluster_version(config)
+    return PARALLEL_CLUSTER_ENROOT_VERSIONS[parallel_cluster_version]
+
+def get_PARALLEL_CLUSTER_PYXIS_VERSION(config):
+    parallel_cluster_version = get_parallel_cluster_version(config)
+    return PARALLEL_CLUSTER_PYXIS_VERSIONS[parallel_cluster_version]
+
 def get_PARALLEL_CLUSTER_PYTHON_VERSION(config):
     parallel_cluster_version = get_parallel_cluster_version(config)
     return PARALLEL_CLUSTER_PYTHON_VERSIONS[parallel_cluster_version]
@@ -338,6 +346,12 @@ def get_PARALLEL_CLUSTER_LAMBDA_RUNTIME(parallel_cluster_version):
     else:
         return aws_lambda.Runtime.PYTHON_3_12
 
+# Version 3.11.1
+
+PARALLEL_CLUSTER_SUPPORTS_PYXIS_VERSION = parse_version('3.11.1')
+def PARALLEL_CLUSTER_SUPPORTS_PYXIS(parallel_cluster_version):
+    return parallel_cluster_version >= PARALLEL_CLUSTER_SUPPORTS_PYXIS_VERSION
+
 # Version 3.12.0
 
 def PARALLEL_CLUSTER_REQUIRES_FSXZ_OUTBOUND_SG_RULES(parallel_cluster_version):
@@ -527,6 +541,7 @@ def DEFAULT_OS(config):
         #     2 cores
         'm7i.xlarge',
         'r7a.large',
+        'r8g.large',
         #     4 cores
         'c7i.2xlarge',
         'm7a.xlarge',
@@ -538,6 +553,7 @@ def DEFAULT_OS(config):
         # 32 GB:
         #     2 cores
         'r7iz.xlarge',
+        'r8g.xlarge',
         #     4 core(s):
         'm7i.2xlarge',
         'r7a.xlarge',
@@ -555,6 +571,7 @@ def DEFAULT_OS(config):
         #     8 core(s):
         'm7i.4xlarge',
         'r7a.2xlarge',
+        'r8g.2xlarge',
         #     16 core(s): ['m8g.4xlarge']
         'c7i.8xlarge',
         'm7a.4xlarge',
@@ -573,18 +590,22 @@ def DEFAULT_OS(config):
         #     16 cores
         'm7i.8xlarge',
         'r7a.4xlarge',
+        'r8g.4xlarge',
         #     32 cores
         'c7i.16xlarge',
         'm7a.8xlarge',
+        'm8g.8xlarge',
         #     64 cores
         'c7a.16xlarge',
+        'c8g.16xlarge',
 
         # 192 GB:
         #     48 cores
         'c7i.24xlarge',
         'm7a.12xlarge',
         #     96 cores
         'c7a.24xlarge',
+        'c8g.24xlarge',
 
         # 256 GB:
         #     4 cores
@@ -593,8 +614,10 @@ def DEFAULT_OS(config):
         #     32 cores
         'm7i.16xlarge',
         'r7a.8xlarge',
+        'r8g.8xlarge',
         #     64 cores
         'm7a.16xlarge',
+        'm8g.16xlarge',
         #     128 cores
         'c7a.32xlarge',
 
@@ -605,24 +628,29 @@ def DEFAULT_OS(config):
         #     96 cores
         'c7i.48xlarge',
         'm7a.24xlarge',
+        'm8g.24xlarge',
         #     192 cores
         'c7a.48xlarge',
+        'c8g.48xlarge',
 
         # 512 GB:
         #     8 cores
         'x2iedn.4xlarge', # Have newer r7iz
         'x2iezn.4xlarge', # Have newer r7iz
         #     64 cores
         'r7a.16xlarge',
+        'r8g.16xlarge',
         #     128 cores
         'm7a.32xlarge',
 
         # 768 GB:
         #     96 cores
         'm7i.48xlarge',
         'r7a.24xlarge',
+        'r8g.24xlarge',
         #     192 cores
         'm7a.48xlarge',
+        'm8g.48xlarge',
 
         # 1024 GB:
         #     16 cores
@@ -1530,6 +1558,10 @@ def get_config_schema(config):
                 },
                 Optional('Architecture', default=DEFAULT_ARCHITECTURE): And(str, lambda s: s in VALID_ARCHITECTURES),
                 Optional('ComputeNodeAmi'): And(str, lambda s: s.startswith('ami-')),
+                # Recommend to not use EFA unless necessary to avoid insufficient capacity errors when starting new instances in group or when multiple instance types in the group
+                # See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-cluster
+                Optional('EnableEfa', default=False): bool,
+                Optional('EnablePyxis', default=False): bool,
                 Optional('Database'): {
                     Optional('DatabaseStackName'): str,
                     Optional('FQDN'): str,

source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py

@@ -217,6 +217,23 @@ def lambda_handler(event, context):
                             Body   = build_file_content
                         )
 
+                    # Image with rootless Docker installed
+                    template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-docker-{distribution}-{version}-{architecture}".replace('_', '-')
+                    template_vars['ComponentS3Url'] = environ['InstallDockerScriptS3Url']
+                    build_file_s3_key = f"{assets_base_key}/config/build-files/{template_vars['ImageName']}.yml"
+                    if requestType == 'Delete':
+                        response = s3_client.delete_object(
+                            Bucket = assets_bucket,
+                            Key    = build_file_s3_key
+                        )
+                    else:
+                        build_file_content = build_file_template.render(**template_vars)
+                        s3_client.put_object(
+                            Bucket = assets_bucket,
+                            Key    = build_file_s3_key,
+                            Body   = build_file_content
+                        )
+
                     # Image with EDA packages
                     template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-eda-{distribution}-{version}-{architecture}".replace('_', '-')
                     template_vars['ComponentS3Url'] = environ['ConfigureEdaScriptS3Url']

source/resources/parallel-cluster/config/bin/configure-eda.sh

@@ -88,4 +88,6 @@ ansible-playbook $PLAYBOOKS_PATH/eda_tools.yml \
     -i inventories/local.yml \
     -e @$ANSIBLE_PATH/ansible_head_node_vars.yml
 
-popd
+ansible-playbook $PLAYBOOKS_PATH/install-rootless-docker.yml \
+    -i inventories/local.yml \
+    -e @$ANSIBLE_PATH/ansible_head_node_vars.yml