[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(storage): Azure backend using thanos.io/objstore #11315

Merged
merged 20 commits into from
Oct 25, 2024
Merged

feat(storage): Azure backend using thanos.io/objstore #11315

merged 20 commits into from
Oct 25, 2024

Conversation

JoaoBraveCoding
Copy link
Collaborator
@JoaoBraveCoding JoaoBraveCoding commented Nov 24, 2023 •
edited by ashwanthgoli
Loading

What this PR does / why we need it*:

This PR adds support to use the thanos.io/objstore backend for the Azure provider for all components excluding the Ruler since it's similar to the work done in #11132.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

  • We tried to the best of our efforts to preserve feature parity so the new backend still uses the ObjectClient interface
  • Fix how some CLI variables we being passed (some HTTP CLI variables were under .http while others weren't)
  • Thanos.io/objstore doesn't support all the CLI flags Loki supports nowadays (see snippet below)
  • This PR was inspired by the way Mimir uses thanos.io/objstore with the goal of making this part of the codebase as similar as possible

CLI Table

azure.account-name
azure.account-key
azure.connection-string
azure.container-name
azure.endpoint-suffix
azure.user-assigned-id
azure.max-retries
azure.max-retry-delay

NOT SUPPORTED AT THE MOMENT
azure.environment
azure.use-managed-identity
azure.use-federated-token
azure.chunk-delimiter
azure.request-timeout
azure.download-buffer-size
azure.upload-buffer-size
azure.download-buffer-size
azure.min-retry-delay
azure.use-service-principal
azure.tenant-id
azure.client-id
azure.client-secret

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@github-actions GitHub Actions
Copy link
Contributor

Trivy scan found the following vulnerabilities:

@JoaoBraveCoding JoaoBraveCoding changed the title storage: Azure backend using thanos.io/objstore feat(storage): Azure backend using thanos.io/objstore Mar 14, 2024
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Mar 15, 2024
@JoaoBraveCoding JoaoBraveCoding marked this pull request as ready for review March 18, 2024 15:59
@JoaoBraveCoding JoaoBraveCoding requested a review from a team as a code owner March 18, 2024 15:59
@periklis periklis mentioned this pull request Jul 15, 2024
Open
7 tasks
JoaoBraveCoding
JoaoBraveCoding commented Oct 24, 2024
View reviewed changes
Copy link
Collaborator Author
@JoaoBraveCoding JoaoBraveCoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, the adapter was a great idea!

JoaoBraveCoding
JoaoBraveCoding commented Oct 24, 2024
View reviewed changes
Copy link
Collaborator Author
@JoaoBraveCoding JoaoBraveCoding left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On pkg/storage/bucket/client.go we should add to func (cfg *StorageBackendConfig) Validate() error {

	if err := cfg.Azure.Validate(); err != nil {
		return errors.Wrap(err, "invalid Azure Storage config")
	}

JoaoBraveCoding
JoaoBraveCoding commented Oct 24, 2024
View reviewed changes
Copy link
Collaborator Author
@JoaoBraveCoding JoaoBraveCoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@JoaoBraveCoding
Copy link
Collaborator Author

Tested and it works 👍

chaudum
chaudum reviewed Oct 25, 2024
View reviewed changes
pkg/storage/bucket/object_client_adapter.go Outdated
"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/pkg/errors"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder why the linter did not catch this empty newline. Please run make format

pkg/storage/bucket/object_client_adapter.go Outdated
Comment on lines 39 to 43
func WithRetryableErrFunc(f func(err error) bool) func(*ObjectClientAdapter) {
return func(o *ObjectClientAdapter) {
o.isRetryableErr = f
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Personally I find this is quite an unusual API.
You are supposed to call bucket.WithRetryableErrFunc(func(err error) bool {})(client), right?

chaudum
chaudum approved these changes Oct 25, 2024
View reviewed changes
pkg/storage/bucket/object_client_adapter.go Outdated
return o
}

type options func(*ObjectClientAdapter)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type options func(*ObjectClientAdapter)
type ClientOptions func(*ObjectClientAdapter)

@ashwanthgoli ashwanthgoli merged commit 5824e3d into grafana:main Oct 25, 2024
60 checks passed
@JoaoBraveCoding JoaoBraveCoding deleted the log-4550-azure branch November 8, 2024 11:37
@loki-gh-app loki-gh-app bot mentioned this pull request Nov 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chaudum chaudum chaudum approved these changes

Assignees
No one assigned
Labels
size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JoaoBraveCoding @chaudum @ashwanthgoli