Releases · cloudposse/atmos

@samtholiya

Implement schema validation for vendoring and CLI schemas @samtholiya (#1147)

## what

Add atmos validate schema command to verify the YAML structure of the atmos CLI

why

Ensures that configuration files follow the correct YAML structure
Prevents misconfigurations and errors caused by invalid or malformed YAML
Adds an extra layer of validation by enforcing schema compliance when applicable
Improves reliability and usability of the CLI by catching issues early

Fix `atmos terraform clean` command for Terraform components in sub-directories @haitham911 (#1195)

what

optimize atmos terraform clean to find all component paths and delete the relevant files
unit test for clean component in subfolders

why

Support components in subfolders

Fix initial stack name when using the two-arg form of `!terraform.output` @ohaibbq (#1226)

what

Fix initial stack name when using the two-arg form of !terraform.output

why

Given

# atmos.yaml
stacks:
  name_template: "{{.settings.context.project_id}}--{{.settings.context.product}}"

# stacks/apps/app/staging.yaml
settings:
  context:
    project_id: staging
    product: app
components:
  terraform:
    apps/app:
      vars:
        project_number: "{{ .settings.context.project_number }}"
        location: "{{ .settings.context.location }}"
        # Pass the `connection-name` output from our `postgres-instance` component to the jii-texting app
        cloudsql_instance: !terraform.output postgres-instance connection_name
        # The 'name' output uses the name of the first bucket, so ensure that the ETL bucket is the first one in the list
        etl_bucket_name:  !terraform.output gcs-buckets name
  postgres-instance:
    # ...

!terraform.output will fail in describe_stacks.go with the following message:

ERRO no config found for the component 'postgres-instance' in the stack manifest 'app/staging'

@aknysh

Update `atmos.Component` template function for `helmfile` components @aknysh (#1217)

what

Update atmos.Component template function for helmfile components
Add tests
Update docs

why

atmos.Component template function allows reading any section or attribute from the component (vars, settings, etc.), including the outputs section (Terraform state). Helmfile components don't have Terraform state. This PR adds detecting the component type (terraform or helmfile) when executing atmos.Component, and don't try to read the remote state for the helmfile components

components:
  # Terraform components
  terraform:
    component-1:
      vars:
        foo: "foo"
        bar: "bar"
        baz: "baz

    component-2:
      vars:
        foo: '{{ (atmos.Component "component-1" .stack).outputs.foo }}'
        bar: '{{ (atmos.Component "component-1" .stack).outputs.bar }}'
        baz: '{{ (atmos.Component "component-1" .stack).outputs.bar }}--{{ (atmos.Component "component-1" .stack).outputs.baz }}'

  # Helmfile components
  helmfile:
    component-3:
      vars:
        foo: '{{ (atmos.Component "component-1" .stack).vars.foo }}'
        bar: '{{ (atmos.Component "component-1" .stack).vars.bar }}'
        baz: '{{ (atmos.Component "component-1" .stack).vars.baz }}'

    component-4:
      vars:
        foo: '{{ (atmos.Component "component-1" .stack).outputs.foo }}'
        bar: '{{ (atmos.Component "component-3" .stack).vars.bar }}'
        # Helmfile components don't have `outputs` (terraform output) - this should result in `<no value>` and not throwing errors
        baz: '{{ (atmos.Component "component-3" .stack).outputs.baz }}'

Refactor YAML Function Processing in `atmos.yaml`, Support `!env`, `!exec`,`!include`,!`repo-root` functions in `atmos.yaml` configuration files @haitham911 (#1202)

what

Refactor YAML Function Processing in atmos.yaml
Support !env, !exec,!include,!repo-root functions in atmos.yaml configuration files

why

Added support for !exec directive in configuration files enables dynamic configuration like GITHUB_TOKEN: !exec <gh auth token>
Added support for !env directive in configuration files enables dynamic configuration
Added support for !include directive in configuration files enables dynamic configuration in atmos.yaml
Added support for !repo-root to retrieve the root directory of the Atmos repository. If the Git root is not found, it will return a default value if specified

Component Updater Docs @milldr (#1211)

what

Use Steps and StepNumber conventions for component updater documentation

why

This documentation was difficult to follow. Using these conventions makes it easier to follow

Replace `myapp` with `mock` component in tests @milldr (#1213)

what

Delete all occurrences of myapp component
Replaced with mock component

why

The myapp component was built with weather API. We were abusing this API for our tests, which isn't even necessary for what we want to test. The simple mock with inputs and outputs is sufficient.

@aknysh

Add `atmos terraform generate planfile` command @aknysh (#1214)

what

Add atmos terraform generate planfile command
Add unit tests
Update docs

why

Use the atmos terraform generate planfile command to generate a planfile for an Atmos Terraform/OpenTofu component in a stack in JSON or YAML formats.

Usage

Execute the terraform generate planfile command like this:

atmos terraform generate planfile <component> -s <stack> [options]

This command generates a planfile for an Atmos terraform component in a stack.

Examples

atmos terraform generate planfile component1 -s plat-ue2-dev
atmos terraform generate planfile component1 -s plat-ue2-prod --format=json
atmos terraform generate planfile component1 -s plat-ue2-prod --format=yaml
atmos terraform generate planfile <component> -s <stack> --file=planfile.json
atmos terraform generate planfile <component> -s <stack> --format=yaml --file=planfiles/planfile.yaml
atmos terraform generate planfile <component> -s <stack> --file=/Users/me/Documents/atmos/infra/planfile.json

Validate Terraform/OpenTofu planfiles using Checkov

You can generate a planfile for a component in a stack and validate it using Checkov.

atmos terraform generate planfile <component> -s <stack>
checkov --file components/terraform/<component>/<stack>-<component>.planfile.json --framework terraform_plan

Refer to Evaluate Checkov Policies on Terraform Plan for more information.

Notify Not Running in a Git Repo @milldr (#990)

what

Warn when not running from a Git repo

why

Atmos is intended to be run from within a project -- typically a Git repo
Alternatively you can specify the base path with an env var

Warn on specific TF Environment Variables @milldr (#1206)

what

Only warn on these environment variables:

"TF_CLI_ARGS"
"TF_VAR_"
"TF_CLI_ARGS_"
"TF_WORKSPACE"

Deleted duplicated warning on atmos terraform shell

why

Previously we warned on all TF_* environment variables as requested. This is far too noisy and unnecessary for the majority of use cases.
Inspired by terraform-exec, we're going to adopt similar convention.

chore: Log Level for TTY Warning @milldr (#1212)

what

Change the TTY warning in the following to debug

+ atmos vendor pull
INFO Vendoring from '/atmos/vendor.yaml'
WARN No TTY detected. Falling back to basic output. This can happen when no terminal is attached or when commands are pipelined.
INFO ✓ terraform-aws-vpc (v5.19.0)
INFO Vendored 1 components.

why

Change TTY warning to log.Debug as it's not important enough to surface at the warn level.

@aknysh

Add `atmos terraform generate planfile` command @aknysh (#1214)

## what

Add atmos terraform generate planfile command
Add unit tests
Update docs

why

Use the atmos terraform generate planfile command to generate a planfile for an Atmos Terraform/OpenTofu component in a stack in JSON or YAML formats.

Usage

Execute the terraform generate planfile command like this:

atmos terraform generate planfile <component> -s <stack> [options]

This command generates a planfile for an Atmos terraform component in a stack.

Examples

atmos terraform generate planfile component1 -s plat-ue2-dev
atmos terraform generate planfile component1 -s plat-ue2-prod --format=json
atmos terraform generate planfile component1 -s plat-ue2-prod --format=yaml
atmos terraform generate planfile <component> -s <stack> --file=planfile.json
atmos terraform generate planfile <component> -s <stack> --format=yaml --file=planfiles/planfile.yaml
atmos terraform generate planfile <component> -s <stack> --file=/Users/me/Documents/atmos/infra/planfile.json

Validate Terraform/OpenTofu planfiles using Checkov

You can generate a planfile for a component in a stack and validate it using Checkov.

atmos terraform generate planfile <component> -s <stack>
checkov --file components/terraform/<component>/<stack>-<component>.planfile.json --framework terraform_plan

Refer to Evaluate Checkov Policies on Terraform Plan for more information.

Summary by CodeRabbit

New Features
- Introduced a new CLI command to generate Terraform planfiles for specified components, supporting JSON and YAML output formats.
Documentation
- Added comprehensive documentation for the new planfile generation command, including usage instructions, examples, and validation guidance.
- Updated examples and integration docs to reflect the latest CLI version and improved template usage in documentation.
Bug Fixes
- Improved quoting in template functions and documentation examples to ensure correct string rendering in JSON outputs.
Tests
- Added extensive tests for the new planfile generation feature, covering standard operation and error scenarios.
Chores
- Upgraded multiple dependencies and bumped the CLI version to 1.172.0.
- Enhanced style consistency in code comments.

@milldr

feat(DEV-2781): Notify Not Running in a Git Repo @milldr (#990)

## what - Warn when not running from a git repo

why

Atmos is intended to be run from within a project -- typically a git repo
Alternatively you can specify the base path with an env var

references

DEV-2781

Summary by CodeRabbit

New Features
- Added a warning message when running the CLI outside a Git repository without an Atmos configuration, helping users identify misconfigurations.
- Enhanced Atmos configuration validation with Git repository presence checks.
Bug Fixes
- Improved detection of Git repository presence to ensure accurate warnings and error handling.
Tests
- Introduced new test cases covering CLI behavior across Git repository and configuration scenarios.
- Added snapshot files to validate expected CLI output and warning messages.

Warn on specific TF Environment Variables @milldr (#1206)

## what - Only warn on these environment variables:

"TF_CLI_ARGS"
"TF_VAR_"
"TF_CLI_ARGS_"
"TF_WORKSPACE"

Deleted duplicated warning on atmos terraform shell

why

Previously we warned on all TF_* environment variables as requested. This is far too noisy and unnecessary for the majority of use cases.
Inspired by terraform-exec, we're going to adopt similar convention.

references

#1042 (comment)

No warning

Warning with 1 input

Warning with 2 inputs

Summary by CodeRabbit

Bug Fixes
- Improved detection of environment variables that may interfere with Terraform by warning only on specific prefixes, reducing unnecessary warnings.
- Enhanced warning messages with clearer, structured logging for easier troubleshooting.

🚀 Enhancements

chore: Log Level for TTY Warning @milldr (#1212)

## what - Change the TTY warning in the following to debug ``` + atmos vendor pull INFO Vendoring from '/atmos/vendor.yaml' WARN No TTY detected. Falling back to basic output. This can happen when no terminal is attached or when commands are pipelined. INFO ✓ terraform-aws-vpc (v5.19.0) INFO Vendored 1 components. ```

why

Change TTY warning to log.Debug as it's not important enough to surface at the warn level.

references

SweetOps: https://sweetops.slack.com/archives/C031919U8A0/p1744819993354959

Summary by CodeRabbit

Style
- Adjusted log message severity for "No TTY detected" from warning to debug, reducing unnecessary warning messages during CLI operations.
Tests
- Updated test snapshots to reflect the new debug log level for TTY detection messages.
- Renamed a test case and modified its configuration to improve reliability and accuracy of test results.

@aknysh

Update `!store` YAML function. Introduce `atmos.Store` template function @aknysh (#1204)

what

Update !store YAML function
Introduce atmos.Store template function
Add tests
Update docs

why

`!store` YAML function

In the !store YAML function, allow using YQ Expressions to retrieve individual values

To retrieve individual values from complex types such as maps and lists, or do any kind of filtering or querying,
you can utilize YQ expressions.

For example:

Retrieve the first item from a list

subnet_id1: !store <store_name> <stack> <component> <key> | query .private_subnet_ids[0]

Read a key from a map

username: !store <store_name> <stack> <component> <key> | query .config_map.username

`atmos.Store` template function

The atmos.Store template function allows reading the values from a remote store (e.g. SSM Parameter Store, Artifactory, Redis, etc.) into Atmos stack manifests

For example, we can read the vpc_id output of the vpc component in the current stack from the SSM Parameter Store
configured in atmos.yaml as ssm simply by doing:

components:
  terraform:
    cluster:
      vars:
        vpc_id: '{{ atmos.Store "ssm" .stack "vpc" "vpc_id" }}'

To access the configuration of a component in a different stack, you can specify the stack name as the second argument.
For example, here we're reading the vpc_id output of the vpc component in the staging stack:

components:
  terraform:
    cluster:
      vars:
        vpc_id: '{{ atmos.Store "ssm" "staging" "vpc" "vpc_id" }}'

Enhance Atmos CLI: Add Support for Custom Base Path and Config Paths @haitham911 (#1091)

what

Introduce global command-line flags (--base-path, --config, and --config-path) to the Atmos CLI tool.
If the --config or --config-path flag is changed, Atmos will load configurations only from the specified paths.
The --config flag specifies an Atmos configuration file, such as dir1/config.yaml or dir2/config2.yaml, updating cli_config_path to dir1;dir2
The --config-path flag specifies directories containing Atmos configuration files. files names should be one of (atmos.yaml, .atmos.yaml,atmos.yml, .atmos.yml), such as dir3/ or dir4/, updating cli_config_path to dir3;dir4
Support handling multiple configurations from both --config and --config-path simultaneously
Integrate embedded configuration support

why

These changes make it easier to override Atmos settings dynamically—especially useful in automation, such as CI/CD pipelines and local testing. By specifying --config or --config-path
Load alternate configurations without modifying global settings.
Apply environment-specific settings for testing or deployments.
Manage multiple Atmos configurations .
Simplify local development by easily switching between configuration setups.

10000

Refactor `go-getter` utility for better testability @samtholiya (#1146)

what

Refactor go-getter utility for better testability
github_token, bitbucket_token and gitlab_token can now be set via atmos configuration

why

Enables unit testing by allowing mocks
Makes the file download mechanism extensible without modifying core logic

@aknysh

Update `providers` and `hooks` sections @aknysh (#1188)

what

Update providers and hooks sections
Add unit tests

why

The providers and hooks sections were not inherited from the base components
Now the following configuration is supported

components:
  terraform:
    component-1:
      providers:
        aws:
          assume_role:
            role_arn: "Base component IAM Role ARN"
        context:
          enabled: true
          delimiter: "-"
          property_order:
            - product
            - region
            - environment
            - name

    component-2:
      metadata:
        inherits:
          - component-1
      providers:
        aws:
          assume_role:
            role_arn: "Derived component IAM Role ARN"

atmos describe component component-2 -s nonprod --query .providers returns the correct providers configuration, deep-merged from the base and derived components

aws:
    assume_role:
        role_arn: Derived component IAM Role ARN
context:
    delimiter: '-'
    enabled: true
    property_order:
        - product
        - region
        - environment
        - name

update !store docs @mcalhoun (#1186)

what

Update the docs for the !store command to reflect the correct parameter order

why

In the previous version, we had component and stack switched and the docs did not match the implementation

Replace refs to "master" branch, fix "!store" doc formatting @Nuru (#1183)

what

Replace references to "master" branch with references to "main" branch

why

Repositories have renamed the default branch to "main", and edit references are not redirected

Implement `atmos list values` @Cerebrovinny (#1036)

what

Add atmos list values CLI command to list component values across stacks
Add atmos list metadata CLI command to list component metadata across stacks
Add atmos list settings CLI command to list component settings across stacks
Update docs
Add unit tests

why

Query and filter component configurations across multiple stacks using YQ expressions
Compare values, metadata, and settings across different environments without having to manually inspect stack files
Output results in various formats (table, JSON, YAML, CSV, TSV) for easy integration with other tools

Examples

atmos list values

# List all values for a component
atmos list values vpc

# Filter with YQ query
atmos list values vpc --query .vars

# Get specific field across stacks
atmos list values vpc --query .vars.region

# Match specific stacks
atmos list values vpc --stack '*-prod-*'

# Filter specific stacks with query
atmos list values vpc --query .vars.tags --stack '*-ue2-*'

# Include abstract components
atmos list values vpc --abstract

# Output in different formats
atmos list values vpc --format json
atmos list values vpc --format yaml
atmos list values vpc --format csv
atmos list values vpc --format tsv

# Limit number of columns
atmos list values vpc --max-columns 3

atmos list metadata

# List all metadata 
atmos list metadata

# Filter with YQ query
atmos list metadata --query .type

# Get specific metadata field
atmos list metadata --query .inherits

# Match specific stacks
atmos list metadata --stack 'dev-*'

# Filter specific stacks with query
atmos list metadata --query .type --stack '*-ue2-*'

# Output as JSON
atmos list metadata --format json

atmos list settings

# List all settings
atmos list settings

# Filter with YQ query
atmos list settings --query .terraform

# Get specific settings field
atmos list settings --query .terraform.backend

# Match specific stacks
atmos list settings --stack 'staging-*'

# Filter specific stacks with query
atmos list settings --query .terraform.backend --stack '*-prod-*'

# Output as YAML
atmos list settings --format yaml

@aknysh

Update `!include` YAML function. Add unit tests. Update docs @aknysh (#1179)

what

Update !include YAML function
Update Atmos JSONSchema for stack manifests validation (to take into account the !include functions)
Add unit tests
Update docs
- https://atmos.tools/core-concepts/stacks/yaml-functions/include

why

Make Atmos to resolve the !include functions during the initial phase of YAML stack manifest processing (in contrast to the other YAML functions, which are processed during the very last stage of stack processing).

At this initial stage, the !include functions evaluate and substitute the contents of included files, converting them into the appropriate data types (e.g., string, boolean, map, or list) before assigning them to the corresponding sections or attributes in the Atmos stack configurations.

Key implications of this behavior:

The !include functions facilitate stack configuration reuse by enabling modularization of common configurations.
These configurations can reside in separate local or remote files and be referenced in multiple stack definitions,
promoting a DRY approach.
The functions preprocess YAML files before Atmos resolves stacks and components.
This enables configuration Inheritance and deep-merging of stack parameters after the !include directives are expanded.
When transitioning from plain Terraform/OpenTofu to Atmos, !include allows seamless reuse of existing variable
files (varfiles) by referencing them in the vars sections of Atmos components:
```
  my-component:
    vars: !include my-component.tfvars
```

Support the `--config` flag as an array of strings for `atmos validate editorconfig` @samtholiya (#1173)

what

Support the --config flag as an array of strings for atmos validate editorconfig

why

Be consistent with our config flag. With pr #1091 config would be an array of string at the root. And this should be reflected in validate editorconfig

Add golangci-lint rule to deprecate the `processArgsAndFlags` function @samtholiya (#1178)

what

We are restricting the usage of os.Getenv
We are also depricating usage/enhancing function processArgsAndFlags

why

we should start using viper.BindEnv for new env variables

Fix bug related to loading and merging atmos configurations files @haitham911 (#1175)

what

Fixed an issue where multiple configuration files were not loading correctly.

why

Previously, when loading files from multiple paths, only the first file found was loaded, ignoring the rest. This update ensures all configurations are properly merged
Now, all configuration files from each specified path are loaded and merged into the Atmos config

@haitham911

🚀 Enhancements

Fix bug related to loading and merging atmos configurations files @haitham911 (#1175)

## What - Fixed an issue where multiple configuration files were not loading correctly. - Now, all configuration files from each specified path are loaded and merged into the Atmos config.

Why

Previously, when loading files from multiple paths, only the first file found was loaded, ignoring the rest. This update ensures all configurations are properly merged.

references

Summary by CodeRabbit

Refactor
- Enhanced the configuration loading mechanism to improve error handling and ensure accurate processing of YAML configuration files.
Tests
- Added new tests for configuration merging, including scenarios for missing configuration files and merging multiple configuration files.

@Listener430

Update dry run for `atmos vendor pull` to support `ssh` + detailed SCP urls alignment @Listener430 (#1076)

what

This PR upgrades the output of dry run mode for vendor pull command.
After the change is applied, the dry run mode shows

details on the SCP-style links converion,
Injected tokens
urls being vendored in a santized format
yet there's no actual files download.

Dry run output

Non-dry run output (actual vendoring)

Add SSM param assume role @mcalhoun (#1170)

what

Add the ability to assume a role when reading/writing AWS SSM param store

why

In order to be able to read values cross-account

Refactor `list components` and fix lints @Cerebrovinny (#1162)

what && why

Refactor list components for new lint standards

Convert SCP-style URLs into proper SSH URLs for `atmos vendor pull`@Listener430 (#1149)

what

Convert SCP-style URLs into proper SSH URLs for atmos vendor pull

why

Sometimes vendoring urls are provided in a non-standard, SCP-style Git URLs formt which omits a scheme and use a colon for separation. In order Go’s URL parser can process them, they have to be converted into fully qualified URLs (using SSH or HTTPS).
Vendoring now honors tokens for Gitlab and Bitbucket for https vendoring
Masking of sensative data in debug statements in Custom Detector
Links without explicit scheme were indication were not handled correctly, e.g. this one failed
git::git@github.com:cloudposse/terraform-null-label.git?ref={{.Version}}
credentials for http vendoring were read from the token only for github, but not fot bitbucket and gitlab

Testing

Use this to run only test cases relevant for this PR
$ go test -v -run '^TestCLICommands/(atmos_vendor_pull_using_SSH|atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage)$' github.com/cloudposse/atmos/tests

non-standard SCP-style links handling

Token injections were tested wtih bitbucket and gitlab (http) for private and public repos + ssh vendoring for both.
Listing them here as there are no dedicated tests/repos available for testing at bitbucket/gitlab.

Fix Vendoring Issues with Globs and Symlinks @Listener430 (#984)

what

Fix Vendoring Issues with Globs and Symlinks

why

double star globs were not correctly matching multiple segments in all cases
vendoring without a shallow depth is 2x slower
the //. is an esoteric expression to copy all files from the root. Rather than expect users to know this, we default it where it makes sense.

Case 1. Globs

    included_paths:
      - "**/{demo-library,demo-stacks}/**/*.{tf,md}"
    excluded_paths:
      - "**/demo-library/**/*.{tfvars,tf}"

Case 2. Globs without double stars upfront

included_paths:
        - "/weather/*.md"

Case 3. Shallow globs and folder exclusion

included_paths:
       - "**/demo-localstack/*"
       - "**/demo-library/**"
     excluded_paths:
       - "**/demo-library/**/stargazers/**"
       - "**/demo-library/**/*.tf"

Releases: cloudposse/atmos

v1.174.0

why

what

why

what

why

Given

Contributors

v1.173.0

what

why

what

why

what

why

what

why

Contributors

v1.172.0

what

why

Usage

Examples

Validate Terraform/OpenTofu planfiles using Checkov

what

why

what

why

what

why

Contributors

v1.172.0-rc.1

why

Usage

Examples

Validate Terraform/OpenTofu planfiles using Checkov

Summary by CodeRabbit

Contributors

v1.172.0-rc.0

why

references

Summary by CodeRabbit

why

references

No warning

Warning with 1 input

Warning with 2 inputs

Summary by CodeRabbit

🚀 Enhancements

why

references

Summary by CodeRabbit

Contributors

v1.171.0

what

why

!store YAML function

atmos.Store template function

what

why

what

why

Contributors

v1.170.0

what

why

what

why

what

why

what

why

Examples

atmos list values

atmos list metadata

atmos list settings

Contributors

v1.169.0

what

`!store` YAML function

`atmos.Store` template function