Merge branch 'release/1.14'

Author: denis-yuen

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,6 +1,11 @@
 **Description**
 A description of the PR, should include a decent explanation as to why this change was needed and a decent explanation as to what this change does
 
+
+**Review Instructions**
+Describe if this ticket needs review and if so, how one may go about it in qa and/or staging environments. 
+For example, a ticket based on Security Hub, Snyk, or Dependabot may not need review since those services will generate new warnings if the issue has not been resolved properly. On the other hand, an infrastructure ticket that results in visible changes to the end-user will definitely require review. Many tickets will likely be between these two extremes, so some judgement may be required. 
+
 **Issue**
 A link to a github issue or SEAB- ticket (using that as a prefix)
 

.github/workflows/docker-image.yml

@@ -29,7 +29,7 @@ jobs:
         python -m pip install --upgrade pip
         pip install requests
     - name: Set up Ruby 2.6
-      uses: actions/setup-ruby@v1
+      uses: ruby/setup-ruby@v1
       with:
         ruby-version: 2.6
     - name: Build environment
@@ -38,8 +38,4 @@ jobs:
     - name: Run script to generate templates
       run: bash install_bootstrap --script
     - name: Run Docker image to validate nginx config
-      run: docker run -v $PWD/config/default.nginx_http.conf:/etc/nginx/conf.d/default.conf:ro -v $PWD/config/default.nginx_http.shared.conf:/etc/nginx/conf.d/default.nginx_http.shared.conf:ro -v $PWD/config/default.nginx_http.security.conf:/etc/nginx/conf.d/default.nginx_http.security.conf:ro nginx:1.13.1 nginx -t -c /etc/nginx/nginx.conf
-    - name: Test image digest script
-      run: |
-        cd scripts/tests
-        python test-webservice-image-digest.py
+      run: docker run -v $PWD/config/nginx-conf/default.nginx_http.conf:/etc/nginx/conf.d/default.conf:ro -v $PWD/config/nginx-conf/default.nginx_http.shared.conf:/etc/nginx/conf.d/default.nginx_http.shared.conf:ro -v $PWD/config/nginx-conf/default.nginx_http.security.conf:/etc/nginx/conf.d/default.nginx_http.security.conf:ro nginx:1.13.1 nginx -t -c /etc/nginx/nginx.conf

DEV-README.md

@@ -11,7 +11,7 @@ There are 3 different sets of metric logs being sent to logstash's elasticsearch
 
 ## Apache HTTP Logs
 
-Currently, install\_bootstrap and docker-compose handles all necessary configuration
+Currently, install\_bootstrap and docker compose handles all necessary configuration
 
 <!-- Long term, will likely move to AWS RDS, making postgres setup simpler
 ## Postgres

README.md

@@ -1,10 +1,8 @@
 # compose\_setup
-This project documents how to setup Dockstore staging and production using composed Docker containers. 
+This project contains configuration template files needed to run Dockstore using AWS Fargate. View the [dockstore-deploy repository](https://github.com/dockstore/dockstore-deploy)
+for more information about how Dockstore is setup using AWS Fargate.
 Log issues and see general documentation at [dockstore](https://github.com/ga4gh/dockstore/issues) and [docs.dockstore.org](https://docs.dockstore.org/) respectively
 
-Port 80 is exposed over http. This port should not be exposed to the public. A separately [configured load
-balancer](https://github.com/dockstore/dockstore-deploy) is responsible for SSL termination and forwarding traffic to this instance. Previously this repo handled the SSL termination with nginx and LetsEncrypt.
-
 If you are looking for how to run Dockstore locally as a developer, you are probably in the wrong place and should take a look at https://github.com/dockstore/dockstore/blob/develop/docker-compose.yml
 
 ## Prerequisities
@@ -17,24 +15,16 @@ If you are looking for how to run Dockstore locally as a developer, you are prob
 
 ## Usage
 
-1. Call the install\_bootstrap script. This templates the contents of `templates` using mustache to the `config` directory while recording your answers for future use. Note that this will also
-rebuild your docker images without affecting existing running containers 
+1. Call the install\_bootstrap script. This templates the contents of `templates` using mustache to the `config` directory while recording your answers for future use.
 
 2. Some additional information on the answers requested in the script
     1. Each integration requires a client id and a secret, it is worth saying that you should not check these in 
     2. The discourse URL is needed to link Dockstore to a discussion forum 
     3. the Google verification code and tag manager ID are used if you want to properly track visitors to Dockstore and what pages they browse to
 
-3. The bootstrap script can also rebuild your Docker images. Keep in mind the following handy commands:
-    1. `install_bootstrap --script` will template and build everything using your previous answers (useful for quick iteration) 
-    2. `docker-compose down` will bring all containers down safely 
-    3. `nohup docker-compose up --force-recreate --remove-orphans >/dev/null 2>&1 &` will re-create all containers known to docker-compose and delete those volumes that no longer are associated with running containers
-    4. `docker system prune` for cleaning out old containers and images
-    5. To watch the logs `docker-compose logs --follow` while debugging
-
-4. After following the instructions in the bootstrap script and starting up the site with `docker-compose`, you can browse to the Dockstore site hosted at port 443 by default. `https://<domain-name>` if you specified https or `http://<domain-name>:443` if you did not. 
+3. After following the instructions in the bootstrap script and starting up the site with AWS Fargate, you can browse to the Dockstore site hosted at port 443 by default using `https://<domain-name>`. 
 
-The current setup relies upon an externally hosted database (currently AWS RDS) and externally hosted search (currently AWS Elasticsearch). 
+The current setup relies upon an externally hosted container orchestration service (current AWS ECS with Fargate), externally hosted database (currently AWS RDS) and externally hosted search (currently AWS Elasticsearch). 
 
 ### Loading Up a Database ###
 
@@ -45,15 +35,15 @@ Note that database migration is run once during the startup process and is contr
 
 ## Logging Usage
 
-If using with logstash in a container (for development), use `-f docker-compose.yml -f docker-compose.dev.yml` flags after each `docker-compose` command to merge docker-compose files (e.g. `docker-compose -f docker-compse.yml -f docker-compose.dev.yml build`) 
+If using with logstash in a container (for development), use `-f docker-compose.yml -f docker-compose.dev.yml` flags after each `docker compose` command to merge docker-compose files (e.g. `docker compose -f docker-compse.yml -f docker-compose.dev.yml build`)
 
 For example to deploy just logging 
 
 ```
-docker-compose  -f docker-compose.dev.yml build
-nohup docker-compose -f docker-compose.dev.yml up --force-recreate --remove-orphans >/dev/null 2>&1 &
-docker-compose -f docker-compose.dev.yml down
-docker-compose -f docker-compose.dev.yml kill
+docker compose  -f docker-compose.dev.yml build
+nohup docker compose -f docker-compose.dev.yml up --force-recreate --remove-orphans >/dev/null 2>&1 &
+docker compose -f docker-compose.dev.yml down
+docker compose -f docker-compose.dev.yml kill
 ```
 
 ### Kibana Dashboard Setup ###
@@ -72,3 +62,10 @@ npm run install-git secrets
 This should install git secrets into your local repository and perform a scan. 
 If secrets are found, the run will error and output the potential secret to stdout.
 If you believe the scan is a false-positive, add the line glob to .gitallowed.
+
+## Handy docker-compose commands:
+    1. `install_bootstrap --script` will template and build everything using your previous answers (useful for quick iteration) 
+    2. `docker compose down` will bring all containers down safely
+    3. `nohup docker compose up --force-recreate --remove-orphans >/dev/null 2>&1 &` will re-create all containers known to docker-compose and delete those volumes that no longer are associated with running containers
+    4. `docker system prune` for cleaning out old containers and images
+    5. To watch the logs `docker compose logs --follow` while debugging

docker-compose.yml

@@ -28,7 +28,6 @@
 "EXTERNAL_GOOGLE_CLIENT_PREFIX1":"replaceme",
 "FEATURED_CONTENT_URL":"foobar",
 "FEATURED_NEWS_URL":"replaceme",
-"GALAXY_PLUGIN_VERSION":"0.0.6",
 "GITHUB_APP_ID": "1234",
 "GITHUB_APP_NAME": "foobar",
 "GITHUB_APP_PRIVATE_KEY_FILE": "/replaceme",
@@ -38,24 +37,16 @@
 "GITLAB_CLIENT_SECRET":"foobar",
 "GOOGLE_CLIENT_ID":"potato",
 "GOOGLE_CLIENT_SECRET":"potato",
-"HTTPS":false,
-"IS_FARGATE_DEPLOY":false,
 "LOGSTASH":false,
 "LOGSTASH_HOST":"replaceme",
+"METRICS_BUCKET_NAME":"replaceme",
 "NEXTFLOW_PARSING_LAMBDA_VERSION":"n/a",
-"BD_CATALYST_SEVEN_BRIDGES_IMPORT_URL":"replaceme",
-"BD_CATALYST_TERRA_IMPORT_URL":"replaceme",
-"AUTHORIZER_TYPE":"replaceme",
-"EXTERNAL_GOOGLE_CLIENT_PREFIX1":"replaceme",
-"SAM_PATH":"replaceme",
-"DATABASE_GENERATED":false,
 "ORCID_CLIENT_ID":"replaceme",
 "ORCID_CLIENT_SECRET":"replaceme",
 "ORCID_SCOPE":"replaceme",
 "ORCID_URL":"replaceme",
 "POSTGRES_DBPASSWORD":"replaceme",
 "PRODUCTION":false,
-"PUBLIC_LAUNCHER_IP_ADDRESS":"",
 "QUAY_CLIENT_ID":"foobared",
 "QUAY_CLIENT_SECRET":"foobared",
 "SAM_PATH":"replaceme",

dockstore_launcher_config/compose.config

@@ -19,26 +19,19 @@ MSG
 
 function template()
 {
-    mkdir -p config
+    # Place the config files into specific directories so they can be mounted to container paths as bind mounts
+    NGINX_CONF_DIRECTORY="config/nginx-conf"
+    NGINX_HTML2_DIRECTORY="config/nginx-html2"
+    WEBSERVICE_DIRECTORY="config/webservice"
 
-    NGINX_CONF_DIRECTORY="config"
-    NGINX_HTML2_DIRECTORY="config"
-    WEBSERVICE_DIRECTORY="config"
+    mkdir -p ${NGINX_CONF_DIRECTORY}
+    mkdir -p ${NGINX_HTML2_DIRECTORY}
+    mkdir -p ${WEBSERVICE_DIRECTORY}
 
-    if [ $IS_FARGATE_DEPLOY == "true" ]; then
-        mkdir -p config/webservice
-        mkdir -p config/nginx-conf
-        mkdir -p config/nginx-html2
-        # Place the config files into specific directories so they can be mounted to container paths as bind mounts
-        NGINX_CONF_DIRECTORY="${NGINX_CONF_DIRECTORY}/nginx-conf"
-        NGINX_HTML2_DIRECTORY="${NGINX_HTML2_DIRECTORY}/nginx-html2"
-        WEBSERVICE_DIRECTORY="${WEBSERVICE_DIRECTORY}/webservice"
+    # Download index.html for UI we will be serving
+    wget -qO ${NGINX_HTML2_DIRECTORY}/index.html https://gui.dockstore.org/${UI2_HASH}/index.html
+    wget -qO ${NGINX_HTML2_DIRECTORY}/manifest.json https://gui.dockstore.org/${UI2_HASH}/manifest.json
 
-        wget -qO ${NGINX_HTML2_DIRECTORY}/index.html https://gui.dockstore.org/${UI2_HASH}/index.html
-        wget -qO ${NGINX_HTML2_DIRECTORY}/manifest.json https://gui.dockstore.org/${UI2_HASH}/manifest.json
-    fi
-
-    mustache dockstore_launcher_config/compose.config templates/Dockerfile_ui2.template > config/Dockerfile_ui2
     mustache dockstore_launcher_config/compose.config templates/robots.txt.template > ${NGINX_HTML2_DIRECTORY}/robots.txt
     mustache dockstore_launcher_config/compose.config templates/default.nginx_http.conf.template > ${NGINX_CONF_DIRECTORY}/default.nginx_http.conf
     mustache dockstore_launcher_config/compose.config templates/default.nginx_http.shared.conf.template > ${NGINX_CONF_DIRECTORY}/default.nginx_http.shared.conf
@@ -59,8 +52,6 @@ function template()
     mustache dockstore_launcher_config/compose.config templates/jvm.options.logstash.template > config/jvm.options.logstash
     mustache dockstore_launcher_config/compose.config templates/postgres_backup.sh.template > scripts/postgres_backup.sh
     chmod a+rx scripts/postgres_backup.sh
-    # .env file
-    mustache dockstore_launcher_config/compose.config templates/env.template > .env
 
     mkdir -p config/rules
     for f in $(ls templates/rules/); do mustache dockstore_launcher_config/compose.config templates/rules/$f > config/rules/$f; done
@@ -73,13 +64,4 @@ fi
 
 template
 
-if [ $IS_FARGATE_DEPLOY == "false" ]; then
-    # We need to set the environment variable for the image digest
-    source .env
-    DOCKSTORE_IMAGE_DIGEST=$(scripts/webservice-image-digest.py $DOCKSTORE_VERSION)
-    echo "DOCKSTORE_IMAGE_DIGEST=$DOCKSTORE_IMAGE_DIGEST" >> .env
-
-    docker-compose build
-fi
-
 echo "Exiting now."

install_bootstrap

@@ -7,5 +7,5 @@ set -o nounset
 # Restarts stack by running commands from /home/ubuntu/compose_setup
 
 cd /home/ubuntu/compose_setup
-docker-compose down
-nohup docker-compose up --force-recreate --remove-orphans >/dev/null 2>&1 &
+docker compose down
+nohup docker compose up --force-recreate --remove-orphans >/dev/null 2>&1 &