Skip to content

Commit e1277eb

Browse files
aknyshNuru
aknysh
and
Nuru
authored
Calculate component dependencies from stack imports. Add deps output for each component (#32)
* [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * [stack_processor]: Add component config dependencies * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Nuru <Nuru@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Nuru <Nuru@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Nuru <Nuru@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Nuru <Nuru@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Nuru <Nuru@users.noreply.github.com> Co-authored-by: Nuru <Nuru@users.noreply.github.com>
1 parent d82a88d commit e1277eb

File tree

15 files changed

+405
-151
lines changed

15 files changed

+405
-151
lines changed

‎.github/auto-release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ template: |
4646
4747
replacers:
4848
# Remove irrelevant information from Renovate bot
49-
- search: '/---\s+^#.*(Renovate configuration|Configuration)(?:.|\n)*?This PR has been generated .*/gm'
49+
- search: '/(?<=---\s+)+^#.*(Renovate configuration|Configuration)(?:.|\n)*?This PR has been generated .*/gm'
5050
replace: ''
5151
# Remove Renovate bot banner image
5252
- search: '/\[!\[[^\]]*Renovate\][^\]]*\](\([^)]*\))?\s*\n+/gm'

‎.github/workflows/auto-release.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ on:
44
push:
55
branches:
66
- main
7+
- master
8+
- production
79

810
jobs:
911
publish:
@@ -14,7 +16,7 @@ jobs:
1416
id: get-merged-pull-request
1517
with:
1618
github_token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}
17-
# Drafts your next Release notes as Pull Requests are merged into "master"
19+
# Drafts your next Release notes as Pull Requests are merged into "main"
1820
- uses: release-drafter/release-drafter@v5
1921
if: "!contains(steps.get-merged-pull-request.outputs.labels, 'no-release')"
2022
with:

‎CHANGELOG.md

Lines changed: 59 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,75 @@
1+
## 0.7.0
2+
3+
- Calculate component dependencies from stack imports
4+
- Added `deps` output for each component
5+
- Reduced unnecessary stack runs in CI/CD systems like Spacelift
6+
7+
- The provider already calculates these parameters:
8+
- `imports` - a list of all imports (all levels) for the stack
9+
- `stacks` - a list of all stacks in the infrastructure where the component is defined
10+
11+
- Both `imports` and `stacks` are not 100% suitable to correctly determine the YAML config files that a component depends on:
12+
- `imports` is too broad. The provider returns all direct and indirect imports for the stack, even those that don't define any variables for the
13+
component. This will trigger the component's stack in Spacelift even if the unrelated imports are modified, resulting in unnecessary stack runs in
14+
Spacelift
15+
- `stacks` is too broad and too narrow at the same time. On the one hand, it detects all stacks in the infrastructure where the component is
16+
defined, but we don't need to trigger a particular stack in Spacelift if some other top-level YAML stack configs are modified. On the other
17+
hand, it misses the cases where a YAML stack config file specifies global variables, which are applied to the component as well
18+
19+
BACKWARDS INCOMPATIBILITIES / NOTES:
20+
21+
## 0.6.0
22+
23+
- Updates github release action to use `go` version 1.16
24+
25+
BACKWARDS INCOMPATIBILITIES / NOTES:
26+
27+
## 0.4.2
28+
29+
- Added `process_stack_deps` input var to the stack_config data source (configurable, `false` by default). Not all provider invocations need to
30+
process all stack dependencies for the components (e.g. Spacelift module needs it, remote-state does not). Makes invocations without processing
31+
stack dependencies 2-3 times faster.
32+
33+
BACKWARDS INCOMPATIBILITIES / NOTES:
34+
35+
## 0.4.1
36+
37+
- Read and parse only YAML files in stack processor (non-YAML files in the stacks folder cause the YAML parser to panic)
38+
39+
BACKWARDS INCOMPATIBILITIES / NOTES:
40+
41+
## 0.4.0
42+
43+
- Added imports to stack output
44+
45+
- Added stacks to each component output
46+
47+
- `imports` attribute shows all imported stacks for the current stack - can be used in CI/CD pipelines to determine stack dependencies
48+
49+
- `stacks` attribute shows all the stacks the component (and its base component, if present) is declared in - can be used in CI/CD pipelines (e.g.
50+
Spacelift) to determine all stacks that the component depends on, and to provision triggers for all the dependencies (once any of the stack config
51+
files changes, the component's job will be triggered)
52+
53+
BACKWARDS INCOMPATIBILITIES / NOTES:
54+
155
## 0.3.0
256

3-
- Workaround for a deep-merge bug in `mergo.Merge()`. When deep-merging slice of maps in a `for` loop,
4-
`mergo` modifies the source of the previous loop iteration if it's a complex map and `mergo` gets a pointer to it,
5-
not only the destination of the current loop iteration.
57+
- Workaround for a deep-merge bug in `mergo.Merge()`. When deep-merging slice of maps in a `for` loop,
58+
`mergo` modifies the source of the previous loop iteration if it's a complex map and `mergo` gets a pointer to it, not only the destination of the
59+
current loop iteration.
660

761
- Added `settings` sections to `data_source_stack_config_yaml` data source to provide settings for Terraform and helmfile components
862

963
- Added `env` sections to `data_source_stack_config_yaml` data source to provide ENV vars for Terraform and helmfile components
1064

1165
BACKWARDS INCOMPATIBILITIES / NOTES:
1266

13-
1467
## 0.2.0
1568

1669
- Added `data_source_stack_config_yaml` data source to process YAML stack configurations for Terraform and helmfile components
1770

1871
BACKWARDS INCOMPATIBILITIES / NOTES:
1972

20-
2173
## 0.1.0
74+
75+
- Initial release

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
12
<!-- markdownlint-disable -->
23
# terraform-provider-utils [![Latest Release](https://img.shields.io/github/release/cloudposse/terraform-provider-utils.svg)](https://github.com/cloudposse/terraform-provider-utils/releases/latest) [![Slack Community](https://slack.cloudposse.com/badge.svg)](https://slack.cloudposse.com) [![Discourse Forum](https://img.shields.io/discourse/https/ask.sweetops.com/posts.svg)](https://ask.sweetops.com/)
34
<!-- markdownlint-restore -->
@@ -29,7 +30,6 @@
2930

3031
Terraform provider for various utilities (deep merging, stack configuration management), and to add additional missing functionality to Terraform
3132

32-
3333
---
3434

3535
This project is part of our comprehensive ["SweetOps"](https://cpco.io/sweetops) approach towards DevOps.

‎examples/data-sources/utils_stack_config_yaml/stacks/eks/eks-defaults.yaml renamed to ‎examples/data-sources/utils_stack_config_yaml/stacks/catalog/eks-defaults.yaml

File renamed without changes.

‎examples/data-sources/utils_stack_config_yaml/stacks/catalog/rds-defaults.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
components:
2+
terraform:
3+
4+
aurora-postgres:
5+
vars:
6+
instance_type: db.r4.large
7+
cluster_size: 1
8+
env:
9+
ENV_TEST_4: test4
10+
ENV_TEST_5: test5
11+
ENV_TEST_6: test6
12+
ENV_TEST_7: test7

‎examples/data-sources/utils_stack_config_yaml/stacks/catalog/s3-defaults.yaml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
components:
2+
terraform:
3+
4+
s3-bucket:
5+
vars:
6+
user_enabled: false
7+
acl: "private"
8+
grants: null
9+
policy: ""
10+
force_destroy: false
11+
versioning_enabled: true
12+
allow_encrypted_uploads_only: false
13+
lifecycle_rule_enabled: false
14+
noncurrent_version_glacier_transition_days: 30
15+
noncurrent_version_deeparchive_transition_days: 60
16+
noncurrent_version_expiration_days: 90
17+
standard_transition_days: 30
18+
glacier_transition_days: 60
19+
deeparchive_transition_days: 90
20+
enable_glacier_transition: false
21+
enable_deeparchive_transition: false
22+
enable_standard_ia_transition: false
23+
enable_current_object_expiration: false
24+
expiration_days: 90
25+
lifecycle_tags: {}
26+
block_public_acls: true
27+
block_public_policy: true
28+
ignore_public_acls: true
29+
restrict_public_buckets: true

‎examples/data-sources/utils_stack_config_yaml/stacks/uw2-dev.yaml

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import:
22
- uw2-globals
3-
- eks/eks-defaults
3+
- catalog/eks-defaults
4+
- catalog/s3-defaults
5+
- catalog/rds-defaults
46

57
vars:
68
stage: dev
@@ -64,16 +66,6 @@ components:
6466
branch: ""
6567
triggers: []
6668

67-
aurora-postgres:
68-
vars:
69-
instance_type: db.r4.large
70-
cluster_size: 1
71-
env:
72-
ENV_TEST_4: test4
73-
ENV_TEST_5: test5
74-
ENV_TEST_6: test6
75-
ENV_TEST_7: test7
76-
7769
aurora-postgres-2:
7870
component: aurora-postgres
7971
vars:
@@ -89,6 +81,11 @@ components:
8981
ENV_TEST_2: test2_override2
9082
ENV_TEST_8: test8
9183

84+
documents-bucket:
85+
component: s3-bucket
86+
vars:
87+
name: "documents"
88+
9289
helmfile:
9390

9491
datadog:

‎examples/data-sources/utils_stack_config_yaml/stacks/uw2-prod.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import:
22
- uw2-globals
3-
- eks/eks-defaults
3+
- catalog/eks-defaults
4+
- catalog/s3-defaults
5+
- catalog/rds-defaults
46

57
vars:
68
stage: prod
@@ -89,6 +91,11 @@ components:
8991
ENV_TEST_2: test2_override2
9092
ENV_TEST_8: test8
9193

94+
documents-bucket:
95+
component: s3-bucket
96+
vars:
97+
name: "documents"
98+
9299
helmfile:
93100

94101
datadog:

‎examples/data-sources/utils_stack_config_yaml/stacks/uw2-staging.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import:
22
- uw2-globals
3-
- eks/eks-defaults
3+
- catalog/eks-defaults
4+
- catalog/s3-defaults
5+
- catalog/rds-defaults
46

57
vars:
68
stage: staging
@@ -89,6 +91,11 @@ components:
8991
ENV_TEST_2: test2_override2
9092
ENV_TEST_8: test8
9193

94+
documents-bucket:
95+
component: s3-bucket
96+
vars:
97+
name: "documents"
98+
9299
helmfile:
93100

94101
datadog:

0 commit comments

Comments
 (0)