Automatically cluster with AWS execution environments (#2797)

Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: José Valim <jose.valim@gmail.com>

docs/deployment/clustering.md

@@ -12,17 +12,31 @@ You may set `LIVEBOOK_CLUSTER` to one of the following values.
 
 > #### Attention {: .warning}
 >
-> "auto" is only a valid value if you're running Livebook's Docker image on Fly.io or Kubernetes.
+> "auto" is only a valid value if you're running Livebook's Docker image on AWS ECS (including Fargate), Kubernetes, and Fly.io.
 
-Detects the hosting platform and automatically sets up a cluster using DNS configuration. Currently the supported platforms are Fly.io and Kubernetes.
+Detects the hosting platform and automatically sets up a cluster using DNS configuration. See platform specific notes below.
+
+#### AWS ECS & Fargate
+
+If you're running Livebook in the **AWS ECS** environment, the `auto` configuration will automatically cluster based on the ECS Container Metadata HTTP API. The cluster's "deployment" name will be based on a SHA checksum of the ECS Container Image ID. Largely you don't need to care about this, but any Livebook deployment using the same image ID will be clustered together. If you want more containers (say in **AWS Fargate**), increase the `desiredCount` of the family.
+
+While ECS and Fargate won't need any further configuration for clustering, you will need to do network level configuration to allow the containers to talk to other resources (databases, S3, etc), as well as be reached by the public internet, etc. That configuration is outside of the scope of this documentation. If you're having issues connecting, there's a good chance it's either you haven't setup the standard ports required, you haven't correctly setup or configured the security groups, or you haven't correctly configured the HTTP listeners/load balancers.
+
+#### Fly.io
+
+When deploying Livebook to Fly.io, the `auto` configuration automatically connects your cluster using Fly's private networks over IPv6.
+
+#### Kubernetes
+
+When using the Livebook application, you can also choose "auto" for Kubernetes deployments, but those values are automatically replaced by a DNS query, such as `dns:livebook-headless.$(POD_NAMESPACE).svc.cluster.local`, when generating the relevant resource definitions. See [the Kubernetes section in the Docker guides](docker.md#Kubernetes) for an example. 
 
 ### `dns:QUERY`
 
 Sets up a cluster using DNS for queries for A/AAAA records to discover new nodes. Additionally, you must additionally set the following env vars:
 
-  * `LIVEBOOK_NODE=livebook_server@MACHINE_IP`, where `MACHINE_IP` is the machine IP of each deployed node
+- `LIVEBOOK_NODE=livebook_server@MACHINE_IP`, where `MACHINE_IP` is the machine IP of each deployed node
 
-  * If your cloud requires IPv6, also set `ERL_AFLAGS="-proto_dist inet6_tcp"`
+- If your cloud requires IPv6, also set `ERL_AFLAGS="-proto_dist inet6_tcp"`
 
 ## Setting other env vars
 

lib/livebook_web/components/app_components.ex

@@ -125,7 +125,7 @@ defmodule LivebookWeb.AppComponents do
             for more information.
           </div>
           <p class="mt-1 text-sm">
-            Automatic clustering is available when deploying to Fly.io and Kubernetes.
+            Automatic clustering is available when deploying to AWS ECS, Fly.io, and Kubernetes.
           </p>
         </div>
       </div>

rel/server/env.sh.eex

@@ -12,6 +12,17 @@ if [ "$LIVEBOOK_CLUSTER" = "auto" ] && [ ! -z "$FLY_APP_NAME" ]; then
     deployment="$(echo "$FLY_IMAGE_REF" | shasum | cut -c 1-10)"
     export LIVEBOOK_NODE="${FLY_APP_NAME}-${deployment}@${FLY_PRIVATE_IP}"
   fi
+elif [ "$LIVEBOOK_CLUSTER" = "auto" ] && [ ! -z "$ECS_CONTAINER_METADATA_URI" ]; then
+  metadata="$(curl --silent $ECS_CONTAINER_METADATA_URI)"
+  machine_ip="$(echo $metadata | $RELEASE_ROOT/bin/livebook eval 'IO.read(:stdio, :eof) |> JSON.decode!() |> Map.fetch!("Networks") |> hd() |> Map.fetch!("IPv4Addresses") |> hd() |> IO.write()')"
+  image_id="$(echo $metadata | $RELEASE_ROOT/bin/livebook eval 'IO.read(:stdio, :eof) |> JSON.decode!() |> Map.fetch!("ImageID") |> IO.write()')"
+
+  if [ -z "${LIVEBOOK_NODE}" ]; then
+    deployment="$(echo $image_id | shasum | cut -c 1-10)"
+    export LIVEBOOK_NODE="livebook-${deployment}@${machine_ip}"
+  else
+    export LIVEBOOK_NODE="${LIVEBOOK_NODE}@${machine_ip}"
+  fi
 fi
 
 if [ -f "${RELEASE_ROOT}/user/env.sh" ]; then