0.13.0

[stack_processor]: Fix imports. Fix deadlocks. Add validation. Improve error messages @aknysh (#53)

what

• Fix imports
• Fix deadlocks
• Add validation
• Improve error messages

why

• Use absolute import paths (from the base path, which is usually stacks) even inside subfolders.
  It's more clear to use

import:
  - catalog/services/*.yaml

than

import:
  - services/*.yaml

if /services/*.yaml is a subfolder of catalog.

• Fix deadlock - if any import pointed to an incorrect path, the WaitGroup was waiting forever causing process hang/freeze. Now it checks if the import path exists and points to any files, and throws an error otherwise

• Add validation and improve error messages for situations like these:

Invalid import in config file ../stacks/catalog/eks-app-defaults.yaml. 
No config files found for import: catalog/services/s1/*.yaml

Invalid import in config file ../stacks/uw2-globals.yaml. 
No matches found for import: 'globals2.yaml'

Invalid import in config file ../stacks/uw2-globals.yaml. 
The file imports itself in import: 'uw2-globals.yaml'

0.12.0

[stack_processor]: Fix concurrency issues in gorotines. Treat all imports as globs @aknysh (#50)

what

• Fix concurrency issues in gorotines
• Treat all imports as globs

why

• Fix concurrency issues in gorotines - was missed in #49
• Treat all imports as globs - allow specifying the entire folder to be imported as YAML config instead of listing each file separately

test

This import

import:
   - services/*.yaml

produces these outputs

         service-1:
              backend:
                acl: bucket-owner-full-control
                bucket: eg-uw2-root-tfstate
                dynamodb_table: eg-uw2-root-tfstate-lock
                encrypt: true
                key: terraform.tfstate
                region: us-west-2
                role_arn: arn:aws:iam::XXXXXXXXXXXX:role/eg-gbl-root-terraform
              backend_type: s3
              deps:
              - catalog/eks-app-defaults
              - catalog/eks-defaults
              - catalog/services/service-1-defaults
              - globals
              - uw2-dev
              - uw2-globals
              env:
                ENV_TEST_1: test1
                ENV_TEST_2: test2
                ENV_TEST_3: test3
              settings:
                spacelift:
                  autodeploy: false
                  workspace_enabled: true
                version: 0
              stacks:
              - catalog/eks-app-defaults
              - catalog/eks-defaults
              - catalog/services/service-1-defaults
              - uw2-dev
              - uw2-prod
              - uw2-staging
              - uw2-uat
              vars:
                environment: uw2
                name: service-1
                namespace: eg
                region: us-west-2
                stage: dev
        service-2:
              backend:
                acl: bucket-owner-full-control
                bucket: eg-uw2-root-tfstate
                dynamodb_table: eg-uw2-root-tfstate-lock
                encrypt: true
                key: terraform.tfstate
                region: us-west-2
                role_arn: arn:aws:iam::XXXXXXXXXXXX:role/eg-gbl-root-terraform
              backend_type: s3
              deps:
              - catalog/eks-app-defaults
              - catalog/eks-defaults
              - catalog/services/service-2-defaults
              - globals
              - uw2-dev
              - uw2-globals
              env:
                ENV_TEST_1: test1
                ENV_TEST_2: test2
                ENV_TEST_3: test3
              settings:
                spacelift:
                  autodeploy: false
                  workspace_enabled: true
                version: 0
              stacks:
              - catalog/eks-app-defaults
              - catalog/eks-defaults
              - catalog/services/service-2-defaults
              - uw2-dev
              - uw2-prod
              - uw2-staging
              - uw2-uat
              vars:
                environment: uw2
                name: service-2
                namespace: eg
                region: us-west-2
                stage: dev

        imports:
        - catalog/eks-app-defaults
        - catalog/eks-defaults
        - catalog/rds-defaults
        - catalog/s3-defaults
        - catalog/services/service-1-defaults
        - catalog/services/service-2-defaults
        - globals
        - uw2-globals

0.11.1

🚀 Enhancements

[spacelift_stack_processor]: Fix concurrency issues @aknysh (#49)

what

• [spacelift_stack_processor]: Fix concurrency issues in ProcessYAMLConfigFile function

why

• The locking in ProcessYAMLConfigFile function was implemented incorrectly. The function is a recursive function calling itself multiple times and updating an external ("global") map (passed to the function as argument). The lock was implemented as a local variable, sometimes resulting in the map being updated by two gorotines at the same time when processing many stacks/imports, throwing this error:

fatal error: concurrent map writes
goroutine 196 [running]:
runtime.throw(0xc3fe23, 0x15)
	runtime/panic.go:1117 +0x72 fp=0xc000119eb0 sp=0xc000119e80 pc=0x436352
runtime.mapassign_faststr(0xb68580, 0xc00016d5c0, 0xc0008f0180, 0xf, 0xc00016d5c0)
	runtime/map_faststr.go:211 +0x3f1 fp=0xc000119f18 sp=0xc000119eb0 pc=0x414491
github.com/cloudposse/terraform-provider-utils/internal/stack.ProcessYAMLConfigFile.func1(0xc0008f0780, 0xc00016d5c0, 0xc0008ed170, 0xc0008f0758, 0xc0006b64e0, 0xc0008f0180, 0xf, 0xc0000e8c00, 0x24)
	github.com/cloudposse/terraform-provider-utils/internal/stack/stack_processor.go:132 +0x185 fp=0xc000119f98 sp=0xc000119f18 pc=0x7d2a85
runtime.goexit()
	runtime/asm_amd64.s:1371 +0x1 fp=0xc000119fa0 sp=0xc000119f98 pc=0x46c401
created by github.com/cloudposse/terraform-provider-utils/internal/stack.ProcessYAMLConfigFile
	github.com/cloudposse/terraform-provider-utils/internal/stack/stack_processor.go:120 +0x39c

related

• Closes #48

0.11.0

[spacelift_stack_processor]: Add `depends_on` to Spacelift YAML stack config @aknysh (#43)

what

• [spacelift_stack_processor]: Add depends_on to Spacelift YAML stack config

why

• For each stack, in YAML stack config, allow specifying the stack dependencies on other components in the same environment and on other stacks
• Automatically detects if the dependency is a stack or a component in the current stack throws an error if it's not a valid stack and not a valid component)
• depends-on then added to labels for the Spacelift stacks
• Will be used in Spacelift workflows (using Rego policies) to trigger the dependent stacks after the "parent" stacks finish running

test

    aurora-postgres-2:
      component: aurora-postgres
      settings:
        spacelift:
          workspace_enabled: true
          autodeploy: true
          labels:
            - "deps:config/secrets/dev-internal-secrets.yml"
          depends_on:
            - vpc
            - dns-delegated
            - gbl-dns-dns-primary

labels:
    - deps:stacks/catalog/rds-defaults.yaml
    - deps:stacks/globals.yaml
    - deps:stacks/uw2-dev.yaml
    - deps:stacks/uw2-globals.yaml
    - deps:config/secrets/dev-internal-secrets.yml
    - depends-on:uw2-dev-vpc
    - depends-on:uw2-dev-dns-delegated
    - depends-on:gbl-dns-dns-primary
    - folder:component/aurora-postgres-2
    - folder:uw2/dev

If any of the entries in depends_on is not a valid stack and not a valid component in the current stack, the following errors will be thrown:

Error: Component 'aurora-postgres-2' in stack 'uw2-dev' specifies 'depends_on' dependency 'vpc3', 
but 'vpc3' is not a stack and not a terraform component in 'uw2-dev' stack

Error: Component 'aurora-postgres-2' in stack 'uw2-dev' specifies 'depends_on' dependency 'gbl-ops-dns-primary', 
but 'gbl-ops-dns-primary' is not a stack and not a terraform component in 'uw2-dev' stack

0.10.0

[spacelift_stack_processor]: Add explicit labels to Spacelift stacks @aknysh (#42)

what

• [spacelift_stack_processor]: Add explicit labels to Spacelift stacks defined in YAML configs

why

• Allow adding custom labels to each stack's Spacelift config
• This will allow
  • adding explicit dependencies (files) to each stack (if the files change, the stack will be triggered)
  • creating complex workflows where one Spacelift stack depends on others

test

Given this config with the custom label:

aurora-postgres-2:
  settings:
    spacelift:
      workspace_enabled: true
      labels:
        - "deps:config/secrets/dev-internal-secrets.yml"

the provider generates the following list of all labels for the component (including the custom label):

labels:
  - import:stacks/catalog/eks-defaults.yaml
  - import:stacks/catalog/rds-defaults.yaml
  - import:stacks/catalog/s3-defaults.yaml
  - import:stacks/globals.yaml
  - import:stacks/uw2-globals.yaml
  - stack:stacks/catalog/rds-defaults.yaml
  - stack:stacks/globals.yaml
  - stack:stacks/uw2-dev.yaml
  - stack:stacks/uw2-globals.yaml
  - stack:stacks/uw2-prod.yaml
  - stack:stacks/uw2-staging.yaml
  - stack:stacks/uw2-uat.yaml
  - deps:stacks/catalog/rds-defaults.yaml
  - deps:stacks/globals.yaml
  - deps:stacks/uw2-dev.yaml
  - deps:stacks/uw2-globals.yaml
  - deps:config/secrets/dev-internal-secrets.yml
  - folder:component/aurora-postgres-2
  - folder:uw2/dev

0.9.0

[spacelift_stack_processor]: Update Spacelift labels @aknysh (#41)

what

• [spacelift_stack_processor]: Update Spacelift labels

why

• For each component (which is a Spacelift stack), we need to apply only the folder: env/stage label. No need for folder: env label since if both are applied, Spacelift double-counts components in each folder (and no component exists directly in env, they are under env/stage)

0.8.0

Add `utils_spacelift_stack_config` data source. Use `goroutines` for concurrent YAML stacks processing @aknysh (#40)

what

• Add utils_spacelift_stack_config data source
• Add examples/data-sources/utils_spacelift_stack_config example
• Use gorotines for concurrent YAML stacks processing
• Directly convert infrastructure YAML stack configs into Spacelift stack configs

why

• Speed up YAML stack processing (in many cases, from tens of minutes to tens of seconds)
• Go binary is much faster than Terraform on converting infrastructure stacks into Spacelift stacks when the complexity of computation is O(n^2) or O(n^3) (as we have in terraform-spacelift-cloud-infrastructure-automation)
• Use goroutines and WaitGroup to process a list of infrastructure YAML stack configs concurrently to speed up processing
• The old implementation was too slow - Spacelift terraform plan on about 260 infrastructure YAML stacks took 40-50 minutes
• In the new implementation, which does all processing concurrently (in parallel on all CPU cores) and directly converts infrastructure stacks into Spacelift stacks, Spacelift terraform plan takes about 17-20 seconds on 260 YAML stacks
• Backwards compatible with the previous releases

Plan: 1697 to add, 0 to change, 0 to destroy.

0.7.0

Calculate component dependencies from stack imports. Add `deps` output for each component @aknysh (#32)

what

• Calculate component dependencies from stack imports
• Add deps output for each component

why

• Reduce unnecessary stack runs in CI/CD systems like Spacelift

• The provider already calculates these parameters:

  • imports - a list of all imports (all levels) for the stack
  • stacks - a list of all stacks in the infrastructure where the component is defined

• Both imports and stacks are not 100% suitable to correctly determine the YAML config files that a component depends on

  • imports is too broad. The provider returns all direct and indirect imports for the stack, even those that don't define any variables for the component. This will trigger the component's stack in Spacelift even if the unrelated imports are modified resulting unrelated stack runs in Spacelift
  • stacks is too broad and too narrow at the same time. On the one hand, it detects all stacks in the infrastructure where the component is defined, but we don't need to trigger a particular stack in Spacelift if some other top-level YAML stack configs are modified. On the other hand, it misses the cases where a YAML stack config file specifies global variables, which are applied to the component as well

For example, in examples/data-sources/utils_stack_config_yaml/stacks/catalog/eks-defaults.yaml we define eks-iam terraform component. The provider returns the following:

stacks:
  - catalog/eks-defaults
  - globals
  - uw2-dev
  - uw2-globals
  - uw2-prod
  - uw2-staging
  - uw2-uat

imports:
  - catalog/eks-defaults
  - catalog/rds-defaults
  - catalog/s3-defaults
  - globals
  - uw2-globals

imports returns the unrelated catalog/s3-defaults and catalog/rds-defaults which will unnecessary trigger this component stack in Spacelift.

stacks returns all top-level YAML stack configs in the infrastructure all of which will unnecessary trigger this component stack in Spacelift.

We need to return only those YAML config files if any of the following conditions is true:

1. The imported config file has the global vars section and it's not empty
2. The imported config file has the component type section, which has a vars section which is not empty
3. The imported config file has the "components" section, which has the component type section, which has the component section
4. The imported config file has the "components" section, which has the component type section, which has the base component section

For example:

vars:
  stage: dev
terraform:
  vars: {}
components:
  terraform:
    aurora-postgres:
      vars:
        instance_type: db.r4.large
        cluster_size: 1
    aurora-postgres-2:
      component: aurora-postgres
      vars:
        instance_type: db.r4.xlarge
  helmfile: {}

vars and terraform.vars are global vars and component-type vars sections
components is the components section
components.terraform is the component type section (as is components.helmfile) - those are component types and get processed similarly
components.terraform.aurora-postgres is the base component section
components.terraform.aurora-postgres-2 is the component section

The changes in this PR return the correct YAML config dependencies for the components, e.g. for eks-iam:

eks-iam:
  deps:
    - catalog/eks-defaults
    - globals
    - uw2-globals

because globals and uw2-globals have global variables for the component, and catalog/eks-defaults defines the component itself.

This will trigger the Spacelift component stack only if these files are modified (in addition to the component project files and the top-level stack, e.g. uw2-prod, themselves).

0.6.0

Change go version in release action @nitrocode (#39)

what

• Updates github release action to use 1.16

Do the go.sum and go.mod files also need to be updated ?

why

• The go releaser worked for me locally because I'm using go 1.16 but did not work in github action as go 1.15 still does not allow darwin arm releases.

references

• Previous PR #38
• Closes #33

0.5.0

Use go 1.6.3 @nitrocode (#38)

what

• Update to go 1.6.3

why

• Support darwin arm

See https://github.com/cloudposse/terraform-provider-utils/runs/2536496044?check_suite_focus=true

      • building binaries
         • DEPRECATED: skipped darwin/arm64 build on Go < 1.16 for compatibility, check https://goreleaser.com/deprecations/#builds-for-darwinarm64 for more info.

references

• Closes #33