[prometheus] Fix grafana sqlite locking errors #13063
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Vadim Lazovsky <vadim.lazovsky@flant.com>
vladimirGuryanov
approved these changes
Apr 17, 2025
|
🟢 Workflow detailsYandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: 🟢 |
Merged
github-actions bot
pushed a commit
that referenced
this pull request
Apr 17, 2025
Signed-off-by: Vadim Lazovsky <vadim.lazovsky@flant.com> Co-authored-by: Vadim Lazovsky <vadim.lazovsky@flant.com>
deckhouse-BOaTswain
added a commit
that referenced
this pull request
Apr 17, 2025
|
Cherry pick PR 13067 to the branch release-1.69 successful! |
|
/backport 1.68 |
|
Failure: cherry-pick commit f4e8c7f to the branch release-1.68 failed. See Job for details. |
lazovskiy
added a commit
that referenced
this pull request
Apr 17, 2025
Signed-off-by: Vadim Lazovsky <vadim.lazovsky@flant.com> Co-authored-by: Vadim Lazovsky <vadim.lazovsky@flant.com>
4 tasks
vladimirGuryanov
pushed a commit
that referenced
this pull request
Apr 17, 2025
This was referenced Apr 17, 2025
Merged
morhayn
pushed a commit
that referenced
this pull request
May 14, 2025
Signed-off-by: Vadim Lazovsky <vadim.lazovsky@flant.com> Co-authored-by: Vadim Lazovsky <vadim.lazovsky@flant.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Grafana uses a SQLite database as a backend for storing data. There is only one database file, and it appears that Grafana executes transactions from the different code parts concurrently, hence leading to high database access contention. If some part of the code cannot take an exclusive lock, it fails and retries. When there are a lot of dashboards, alerts, etc., this particular code may exhaust all retries and fail, leading to errors.
There is a consistency mechanism better than exclusive locking - WAL.
Why do we need it, and what problem does it solve?
It has been found that dashboards with alerts cannot be provisioned, or the alerts are never executed due to the issue described above.
Why do we need it in the patch release (if we do)?
We do, as several setups use alerts on the dashboards, and confirmed to have this issue.
Checklist
Changelog entries