Amazon ECS

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that helps you to more efficiently deploy, manage, and scale containerized applications. It deeply integrates with the AWS environment to provide an easy-to-use solution for running container workloads within Amazon Web Services.

For more information on Amazon ECS, please visit https://aws.amazon.com/ecs.

Gitana SDK

The code and configuration provided here references the samples provided in the Gitana SDK. We recommend that you pull down this code and consulting the files there for reference:

Gitana SDK - Amazon ECS Examples for Gitana 3.2

Infrastructure

In this article, we'll walk through the configuration of Gitana within ECS. In doing so, we make a best effort to stick to tried and true, best practices with ECS. Many of the definitions and configs provided here are useful for learning and may very well apply to other efforts customers make configure using other AWS features such as CloudFormation templates, EKS or EC2 directly.

In addition to using ECS, we will also use the following AWS services:

  • AWS Document DB (MongoDB)
  • AWS OpenSearch (ElasticSearch)
  • AWS Fargate (for container allocation)
  • AWS S3

Gitana works natively with both of these. This guide provides sample configurations to integrate with both.

Getting Started

We recommend using the Gitana Tools library to auto-generate a base set of configuration files for use in your container framework. This is a good starting point no matter whether you're using Kubernetes, ECS or any other framework.

To begin, open up a terminal session and go into an empty directory.

Then run the following for your specific version of Gitana:

docker pull public.ecr.aws/gitana/admin:3.2.84
docker run -v .:/data public.ecr.aws/gitana/admin:3.2.84 generate-certificate --name properties
docker run -v .:/data public.ecr.aws/gitana/admin:3.2.84 generate-root-properties --cert config/keys/properties.crt

This will generate the following

config/root.properties                          (encrypted root properties)
config/keys/properties.key                      (private key file)
config/keys/properties.pem                      (public key file)
config/keys/properties.crt                      (certificate)

The root.properties file holds the unique security tokens for your cluster. These tokens are encrypted using your properties.crt certificate / public key.

You should now generate your "admin" password. This can be appended to your root properties.

docker run -v .:/data public.ecr.aws/gitana/admin:3.2.84 set-property --file config/root.properties --cert config/keys/properties.crt --property admin.password --value admin

x You should then also create your API extension files:

docker run -v .:/data public.ecr.aws/gitana/admin:3.2.84 generate-configs

This will generate:

config/api/setup.sh
config/api/classes/container.properties
config/api/classes/log4j2-container.xml
config/api/classes/gitana-container-context.xml

Configure Amazon ECS

It is expected that the reader will be familiar with Amazon ECS and Amazon AWS with respect to setting up an ECS cluster. Here are some general notes.

VPC

Set up a VPC within a single region using three availability zones. Make sure that you have both public and private subnets available for each availability zone. We will refer to the region going forward as regionId. You will need to substitute in the region ID that is appropriate for your config.

The VPC ID will be referred to as vpcId.

IAM Roles

You will need to define two IAM roles - ecsTaskExecutionRole and ecsTaskNodeRole.

  • ecsTaskExecutionRole is an IAM role that the ECS orchestration layer will use to execute your task definitions (which largely entails the provisioning and configuration of containers).
  • ecsTaskNodeRole is an IAM role available to your running containers. Gitana will automatically use this role for any interactions with AWS services (unless otherwise configured). This includes connecting to AWS DocumentDB and AWS OpenSearch.

Open Search

Create an OpenSearch cluster within your VPC. By default, your cluster should be configured with SSL enabled.

Gitana will authenticate to your OpenSearch cluster using either the container's IAM role or an explicit IAM user (access/secret key). Configure your OpenSearch cluster to allow for authentication as that IAM user/role and map that authenticated principal to an internal user within the database.

You will need to note the following:

  • The cluster host (i.e. mydocumentdb-mban3wvglsmxiexkfsxngpaunm.eu-west-1.es.amazonaws.com:443)
  • The cluster name (i.e. my-documentdb-cluster)
  • Username and Password

Document DB

Create a DocumentDB cluster within your VPC. By default, your cluster should be configured with SSL enabled.

In order to connect with SSL to the DocumentDB cluster, you will need to download the global-bundle.pem certificate authority file that DocumentDB provides. This certificate authority must be provided to Gitana so that it can connect via TLS to your DocumentDB cluster.

Note that this file is included with the source code linked above but you may need to replace it for your specific VPC/region.

Gitana will connect to your DocumentDB cluster using either the container's IAM role. It will authenticate using a DocumentDB username/password. As such, you will need to set up a username/password for use within DocumentDB or utilize the master user.

You will need to note the following:

  • The cluster host: (i.e. docdb-2024-01-02-19-59-57.cluster-csroc4g7mbq2.eu-west-1.docdb.amazonaws.com)
  • Username and Password

SNS / SQS

We will use Amazon SNS to receive notifications from our API server. These notifications are then placed onto a message queue. In this case, we will use Amazon SQS for our message queue.

Please follow the instructions for setting up Amazon SNS here: Setting up Amazon SNS

Please follow the instructions for setting up Amazon SQS here: Setting up Amazon SQS

For more information, see the sections on Amazon SNS and Amazon SQS as described here: API Server Configuration

You will need to the SNS and SQS details to configure your container.properties and XML files below.

S3 (Binary Storage)

You will need an S3 bucket to store your binary files. These binary files consist of any binary attachment information stored to nodes within Gitana.

Set up a bucket for this purpose. You will need to know the bucket name, the region and the AWS access key and secret key required to access the bucket.

S3 (Configuration Storage)

You will need an S3 bucket for storage of your Gitana configuration files. These files are stored into S3 and are then loaded into the root volume of the Gitana API and UI containers upon startup. A sidebar ("init") container runs ahead of the primary container. It connects to S3, downloads your files and copies them into place.

Set up a bucket and ensure that the ecsTaskNodeRole has authorities to connect to the bucket and read files from it.

Cluster Configuration

Provide a name and password for your Gitana cluster. Also, enable AWS. Provide an AWS tag key/value that is applied to EC2 instances to support EC2 discovery.

Make the following changes to api/classes/container.properties:

cluster.group.name=mycluster
cluster.group.password=mycluster
cluster.port=5800

Enable AWS

cluster.aws.enabled=true
cluster.aws.tag.key=mycluster
cluster.aws.tag.value=mycluster-member

DocumentDB Configuration

Connect Gitana to DocumentDB using TLS/SSL. You will need your DocumentDB host, username and password.

Make the following changes to api/classes/container.properties:

mongodb.hosts=DOCUMENTDB_HOST
mongodb.default.authentication.required=true
mongodb.default.authentication.username=DOCUMENTDB_USERNAME
mongodb.default.authentication.password=DOCUMENTDB_PASSWORD
mongodb.default.engine=aws-document-db
mongodb.default.ssl.enabled=true
mongodb.default.ssl.invalidHostNameAllowed=true
mongodb.default.retryWrites=false
mongodb.default.replicaSetName=rs0
mongodb.default.readPreference=secondaryPreferred

Where DOCUMENTDB_HOST, DOCUMENTDB_USERNAME and DOCUMENTDB_PASSWORD should be supplied based on your AWS Document DB installation in prior steps.

With the configuration shown above, the EC2 container running your task definition will connect to DocumentDB as the ecsTaskNodeRole IAM role. Make sure that this role has sufficient authorities to connect.

In order to connect to DocumentDB using SSL, you will need to install DocumentDB's CA certificate into the API container on startup.

This can be done by adding the following to the file api/setup.sh:

ls $JAVA_HOME/lib/security/cacerts

# download the DocumentDB pem
curl -sS "https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem" > /gitana/global-bundle.pem

# split by delimitter
awk 'split_after == 1 {n++;split_after=0} /-----END CERTIFICATE-----/ {split_after=1}{print > "rds-ca-" n ".pem"}' < /gitana/global-bundle.pem

for CERT in rds-ca-*; do
    alias=$(openssl x509 -noout -text -in $CERT | perl -ne 'next unless /Subject:/; s/.*(CN=|CN = )//; print')
    echo "Importing $alias"
    keytool -import -file ${CERT} -alias "${alias}" -storepass changeit -keystore $JAVA_HOME/lib/security/cacerts -noprompt -trustcacerts
    rm $CERT
done

This code runs on startup of the API container. It downloads the certificate and installs it into the JDK trust store before the API server starts up.

OpenSearch Configuration

Connect Gitana to OpenSearch using TSL/SSL. You will need your OpenSearch host, username and password.

Make the following changes to api/classes/container.properties:

elasticsearch.remote.hosts=OPENSEARCH_HOST
elasticsearch.remote.username=OPENSEARCH_USERNAME
elasticsearch.remote.password=OPENSEARCH_PASSWORD
elasticsearch.remote.ssl=true
elasticsearch.remote.cluster.name=mycluster
elasticsearch.remote.serviceName=es
elasticsearch.remote.region=eu-west-1

Where OPENSEARCH_HOST, OPENSEARCH_USERNAME and OPENSEARCH_PASSWORD should be supplied based on your configuration of Open Search in previous steps.

With the configuration shown above, the EC2 container running your task definition will connect to DocumentDB as the ecsTaskNodeRole IAM role. Make sure that this role has sufficient authorities to connect.

Next, add the following XML configuration to enable authentication to DocumentDB:

Add the following to api/classes/gitana-container-context.xml:

<bean id="elasticSearchCondorHttpClientConfigCallbackFactory" parent="abstractElasticSearchCondorHttpClientConfigCallbackFactory" class="org.gitana.platform.services.elasticsearch.OpenSearchHttpClientConfigCallbackFactory">
    <property name="serviceName"><value>${elasticsearch.remote.serviceName}</value></property>
    <property name="region"><value>${elasticsearch.remote.region}</value></property>
</bean>
Using Direct Credentials

Alternatively, if you wish to connect as a specific IAM user or role, you may optionally provide your AWS access credentials directly. To do so, add the following to api/classes/container.properties:

elasticsearch.remote.accessKey=YOUR_AWS_ACCESS_KEY
elasticsearch.remote.secretKey=YOUR_AWS_SECRET_KEY

Fill in the correct values for YOUR_AWS_ACCESS_KEY and YOUR_AWS_SECRET_KEY.

And adjust your api/classes/gitana-container-context.xml file to be like this:

<bean id="elasticSearchCondorHttpClientConfigCallbackFactory" parent="abstractElasticSearchCondorHttpClientConfigCallbackFactory" class="org.gitana.platform.services.elasticsearch.OpenSearchHttpClientConfigCallbackFactory">
    <property name="accessKey"><value>${elasticsearch.remote.accessKey}</value></property>
    <property name="secretKey"><value>${elasticsearch.remote.secretKey}</value></property>
    <property name="serviceName"><value>${elasticsearch.remote.serviceName}</value></property>
    <property name="region"><value>${elasticsearch.remote.region}</value></property>
</bean>

SNS

Adjust your api/classes/gitana-container-context.xml file to include the following:

<bean id="cloudcmsUIServerApplicationDeployer" class="org.gitana.platform.services.application.deployment.CloudCMSApplicationDeployer" parent="abstractApplicationDeployer">
    <property name="type"><value>${gitana.default.application.deployer.uiserver.type}</value></property>
    <property name="deploymentURL"><value>${gitana.default.application.deployer.uiserver.deploymentURL}</value></property>
    <property name="domain"><value>${gitana.default.application.deployer.uiserver.domain}</value></property>
    <property name="baseURL"><value>${gitana.default.application.deployer.uiserver.baseURL}</value></property>
    <property name="notificationsEnabled"><value>${gitana.default.application.deployer.uiserver.notifications.enabled}</value></property>
    <property name="notificationsProviderType"><value>${gitana.default.application.deployer.uiserver.notifications.providerType}</value></property>
    <property name="notificationsProviderConfiguration">
        <map>
            <entry key="accessKey"><value>${gitana.default.application.deployer.uiserver.notifications.configuration.accessKey}</value></entry>
            <entry key="secretKey"><value>${gitana.default.application.deployer.uiserver.notifications.configuration.secretKey}</value></entry>
            <entry key="region"><value>${gitana.default.application.deployer.uiserver.notifications.configuration.region}</value></entry>
        </map>
    </property>
    <property name="notificationsTopic"><value>${gitana.default.application.deployer.uiserver.notifications.topic}</value></property>
</bean>

Adjust your api/classes/container.properties file to include the following details about your SNS details:

gitana.default.application.deployer.uiserver.notifications.enabled=true
gitana.default.application.deployer.uiserver.notifications.providerType=sns
gitana.default.application.deployer.uiserver.notifications.topic=arn:aws:sns:us-east-1:accountId:queueName
gitana.default.application.deployer.uiserver.notifications.configuration.accessKey=
gitana.default.application.deployer.uiserver.notifications.configuration.secretKey=
gitana.default.application.deployer.uiserver.notifications.configuration.region=

These settings enable the API to publish messages to the Amazon SNS service. In turn, the SNS service should be connected to an SQS simple queue. The messages that the API sends to the SNS endpoint should be routed to the SQS queue (which the UI server then picks up below).

SQS

Your UI task definition should define the following environment variables so that messages from the SQS simple queue will be picked up and processed by the UI cluster.

Ensure that your UI task definition includes the following:

CLOUDCMS_NOTIFICATIONS_ENABLED=true
CLOUDCMS_NOTIFICATIONS_SQS_QUEUE_URL=http://sqs.us-east-1.amazonaws.com/{accountId}/{queueId}
CLOUDCMS_NOTIFICATIONS_SQS_REGION=us-{region}
CLOUDCMS_NOTIFICATIONS_SQS_ACCESS_KEY={accessKey}
CLOUDCMS_NOTIFICATIONS_SQS_SECRET_KEY={secretKey}

These environment variables are included in the task definition within the SDK for reference.

S3 (Binary Storage)

Add the following to your api/classes/gitana-container-context.xml file:

<!-- s3 binary storage -->
<util:map id="defaultBinaryStorageConfiguration">
    <entry key="accessKey" value="${gitana.defaultBinaryStorageConfiguration.accessKey}"/>
    <entry key="secretKey" value="${gitana.defaultBinaryStorageConfiguration.secretKey}"/>
    <entry key="bucketName" value="${gitana.defaultBinaryStorageConfiguration.bucketName}"/>
    <entry key="region" value="${gitana.defaultBinaryStorageConfiguration.region}"/>
</util:map>

And add the following to your api/classes/container.properties file:

org.gitana.platform.services.binary.storage.provider=s3gridfs
gitana.defaultBinaryStorageConfiguration.accessKey={accessKey}
gitana.defaultBinaryStorageConfiguration.secretKey={secretKey}
gitana.defaultBinaryStorageConfiguration.bucketName={bucketName}
gitana.defaultBinaryStorageConfiguration.region={regionId}

Be sure to fill in the values for accessKey, secretKey, bucketName and regionId.

S3 (Configuration Storage)

Once you've made the changes to your configuration files, you can upload them to your S3 bucket.

One way to do this is to run a command like this:

aws s3 sync config/ s3://mycluster-bucket/config

ECS

Create an ECS cluster with Fargate configured for container provisioning.

Fargate is an AWS service that automatically takes care of the provisioning and lifecycle management of containers. Fargate adheres to the prescription provided in your Task Definition.

A Task Definition is similar, in some sense, to the configuration of a Pod in Kubernetes. It defines a facility in which one or more containers may run. Task Definitions define the amount of memory, CPU and other resources that they wish to allocate. In turn, containers may also specify their requirements for memory, CPU and other resources.

Fargate allocates EC2 instances behind the scenes to host these Task Definition runtimes and provisioned containers. The idea is that it shouldn't really matter where your containers reside so long as they have a consistent network environment and low-latency to the AWS DocumentDB and AWS OpenSearch endpoints.

Services

You should set up ECS services within your Cluster for:

  • api
  • ui
  • av
  • clamav

Each service should be set up with a load balancer on a public subnet (if you intend to have your load balancer service requests directly from the outside world). Use a private subnet if you will solve routing from public ingress via a different mechanism.

We recommend using Service Connect to for your ECS services. This automatically takes care of provisioning domain names for your services so that they can see each other. Alternatively, you could use Service Discovery or roll your own using EC2.

In the end, you should have three public load ALB load balancers for API (api-lb), UI (ui-lb) and AV (av-lb). The URL endpoints for these might look like:

http://api-lb-1786465342.eu-west-1.elb.amazonaws.com
http://ui-lb-1786465342.eu-west-1.elb.amazonaws.com
http://av-lb-1786465342.eu-west-1.elb.amazonaws.com

Similarly, you should have three internal routes available:

http://api:80
http://ui:80
http://av:80
http://clamav:3310

Where the api, ui and av services are on public subnets. The clamav service should be a private subnet.

Once these are in place, you can create Route 53 records to make access to the public ALBs more accessible.

Task Definitions

The following task definitions are provided within the Gitana SDK example:

Each of these task definitions should be loaded into their respective service. Be sure to make modifications to each task definition to suit your needs. In some cases, environment variables will need to be filled in. In other cases, you will need to adjust CPU and memory requirements or adjust healthcheck settings to match your precise needs.

As noted previously, we recommend allocating containers using Amazon Fargate.

You will need to make sure that your API containers have low-latency connections to the intended target database and search engine (in this case, AWS Document DB and AWS Open Search, respectively). We recommend keeping these elements within the same VPC.

The same provided here is intended to keep all running components within the same VPC (and indeed, within the same availability zone and subnet). This is to keep things simple. In practice, you may wish to spread your API and UI containers out for purposes of resiliency and fault tolerance. In doing so, you will need to be mindful of maintaining low latency DB and search engine connectivity.

Production Recommendations

What is presented here is quite simple. It should give you the general idea on how to set things up within ECS.

For production deployments, we recommend the following:

Web vs Workers

Split your API web containers from your API worker containers. You can either do this using two separate clusters (with a single service each) or using two services within the same cluster. The web API containers are responsible for handling and serving API traffic. The worker API containers are responsible solely for processing background jobs.

In this way, you can size the services differently and provision CPU and memory that is ideal and minimal for each tier. This saves you money and also delivers the greatest scalability. Web-facing API containers can generally be much leaner and more CPU optimized whereas Worker API containers should have more memory and not be responsible for handling any web traffic.

As you require more request-handling capacity, you can scale up the web API service. As you require more background job processing power, you can scale up the worker API service.

Take a look at the Kubernetes "workers" example for an idea of how to set this up. It isn't ECS, but it should provide some ideas and help to get you started.

Secrets vs S3

In the ECS example provided herein, we've utilized S3 to store files. A sidecar (init) container is used to pull down files on start up. The files are mounted into a /gitana base directory and the API servers pick up things from disk. It works.

However, it may not be ideal. You may want to split certain files out of your file set and store them as Amazon Secrets. Those secrets can be fed into your config as Environment Variables.

Ideal candidates for this are the root.properties file, the properties.key private key file and the Gitana license itself.

For an example of this, see the Kubernetes examples.

,