Add template functions to support time handling in Loki alerts. #16619

iamhalje · 2025-05-20T16:56:58Z

Currently, Prometheus template functions include parseDuration, which parses a duration string (e.g. "1h2m10ms") and returns the total duration in seconds as a float. However, it is often more useful to work directly with Go time.Duration objects or time.Time operations in templates for alerting and logging purposes.

For example, in Loki alerting, when adding a StartsAt (need add to Loki) function for alert start time, using parseDurationTime allows returning the time in the needed format, such as:

{{ (StartsAt.Add (parseDurationTime "30m")).UnixMilli }}

Adding parseDurationTime enhances Prometheus templating by providing rich duration/time operations in templates, improving alerting capabilities with use cases in Loki.

Related Issues and Pull Requests

grafana/loki#4980
prometheus/alertmanager#3816

Signed-off-by: Dmitry Ponomaryov <me@halje.ru>

beorn7 · 2025-06-04T15:04:16Z

I cannot really judge the usefulness of this feature in the context of Prometheus templating. (For example, I don't understand how Prometheus templating is used in Loki alerting.) I don't know who is currently most qualified to act as a guardian of the templating. Maybe @grobinson-grafana and @gotjosh as the Alertmanager experts have an opinion here. Maybe @slim-bean can help explaining the connection to Loki.

beorn7

Just some code level comments. This should not imply that I think we should add this feature. I leave this to the experts, see other comment.

template/template.go

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

the previous name 'parseDurationTime' was unclear. The new name 'parseGoDuration' makes it explicit that this function uses Go's standard `time.ParseDuration`, which does not support Prometheus-style durations like `3d` or `1w`. Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

…prometheus into add-template-parsedurationtime

iamhalje · 2025-06-04T17:15:08Z

@beorn7 thanks for reaction on this request

overall, i just want to say that this is an attempt to approach the long-standing issue of adding time-based information into Loki alerts - even if it's just from one possible angle. the problem has been around for over five years, and this is initial step toward making such functionality possible

for reference, this is how things currently work in Loki:
https://github.com/grafana/loki/blob/main/pkg/ruler/compat.go#L317

Loki relies entirely on prometheus/templating within the ruler (alert component) to tempating and take dynamic data in alerts, but this mechanism lacks access to time-related values

it's absolutely the right move to involve the developers at this point - thank you for that. i just wanna to stress again that this PR alone won't be sufficient to solve the entire problem, but perhaps it will prompt the Grafana Loki team to consider adding something similar or native on their side.

Thanks again!

verdel · 2025-06-06T11:32:03Z

As @iamhalje mentioned above, Loki uses prometheus/templating for AlertRule templating, which lacks functions for retrieving the current timestamp and time duration.

As a result, if prometheus/templating were to support getting time.Time and time.Duration, it would be possible, for example, to add a link to a Grafana dashboard with a specific timestamp in the AlertRule annotations

"annotations": {
  "title": "Alert Summary text",
  "description": "Detailed alert description",
  "log_url": "https://grafana.example.com ...from={{ now.Add (parseGoDuration "-2m") }}, to={{ now }}"
}

I propose adding additional function now, to help resolve time-related issues in AlertRule templating for Loki and Prometheus.

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje · 2025-06-06T19:45:20Z

@verdel ideally, Loki itself would track the exact alert firing time and provide a variable in the templating context representing this specific alert start time. This would allow generating dashboard URLs with precise time ranges relative to when the alert actually began

however, the current approach (let’s call it nowTime) is still a practical solution:

groups:
- name: alert
  rules:
    for: 15m
    annotations:
      dashboard: "https://grafana.example.com/dashboard?from={{ (nowTime.Add (parseGoDuration \"-30m\")).UnixMilli }}&to={{ (nowTime.Add (parseGoDuration \"30m\")).UnixMilli }}"

since we can use time.Duration offsets relative to nowTime, we can select any desired time window and pass it to the alert template as needed. that said, simply returning nowTime as the current time is still useful and requires no changes on Loki side

this way, alert templates can use nowTime to get at least the current timestamp, and we can add alert firing times later as needed

@beorn7 await for feedback from @prometheus/@grafana teams for adding this functions in templating. thanks

verdel · 2025-06-06T22:42:46Z

@verdel ideally, Loki itself would track the exact alert firing time and provide a variable in the templating context representing this specific alert start time. This would allow generating dashboard URLs with precise time ranges relative to when the alert actually began

When an instance of Expander is created, a timestamp is passed to it. If I’m not mistaken, in that case, it’s possible to change the nowTime function from time.Now() to timestamp.Time(), and we will get the exact timestamp that Loki provides during the alert rule evaluation process.

But in that case, the function name nowTime doesn’t quite reflect its functionality. It might be necessary to rename it.

dashboard: "https://grafana.example.com/dashboard?from={{ nowTime.Add (parseGoDuration "-30m").UnixMilli }}&to={{ nowTime.Add (parseGoDuration "30m").UnixMilli }}"

A small note: UnixMilli should be called not on the output of parseGoDuration, but on the result of nowTime.Add, like this:

{{ (nowTime.Add (parseGoDuration "-30m")).UnixMilli }}

It’s also important to note that Loki uses vendoring for third-party modules, and currently, the version of the Prometheus module it uses is slightly behind the latest version in that repository.

verdel · 2025-06-07T06:35:33Z

@beorn7, the reason for creating this PR might become a bit clearer if you read the discussion in my PR in the Alertmanager repository.

Since Prometheus/Loki doesn’t support performing time operations when preparing additional information in an AlertRule, I pass a template into the annotation, which is then re-templated on the Alertmanager side.

It looks like this:

AlertRule

annotations:
    log_url: `https://grafana.test.com/explore?schemaVersion=1&"range":{"from":"%from%","to":"%to%"}}}&orgId=1`,

Alertmanager Template

{{- if (index .Alerts 0).Annotations.log_url }}:books: [Logs]({{ (index .Alerts 0).Annotations.log_url | reReplaceAll "%from%" (printf "%d000" ((index .Alerts 0).StartsAt.Add -120000000000).Unix) | reReplaceAll "%to%" (printf "%d000" (index .Alerts 0).StartsAt.Unix)}}) · {{ end -}}

But since Alertmanager still doesn’t have a function to work with durations, I still have to use a static value in the argument of the .Add function.

Additionally, using a template inside .Annotation breaks the display of information in the Alertmanager UI itself.

We discussed with @grobinson-grafana that this approach—passing a template into an AlertRule—is not the right solution, and that the value should be generated on the Prometheus/Loki side, not in Alertmanager.

If we add these two time-handling functions to the AlertRule templating in Prometheus, it would solve the problem.

beorn7 · 2025-06-10T19:02:50Z

In view of all the @-mentioning here, I should re-emphasize that I'm not qualified to vet the validity of this feature.

verdel · 2025-06-10T23:19:24Z

@beorn7, is there any other way to notify the maintainers responsible for the area related to our issue that we need their help reviewing this PR?

template/template.go

beorn7 · 2025-06-11T09:41:15Z

@beorn7, is there any other way to notify the maintainers responsible for the area related to our issue that we need their help reviewing this PR?

In principle, the PR alone should eventually trigger a review. @-mentioning people that you think are qualified might help. You could also try to catch people on the CNCF Slack. But these are all using increasing invasiveness, and reviewers are usually overloaded already.

beorn7 · 2025-06-11T09:45:56Z

Looking at templating in general (of which I am not an expert, as said): There is toTime, which converts a string or a number representing a Unix timestamp in seconds into the Go time.Time type. I would say that a corresponding toDuration would make sense and would probably be less controversial.

I'm not sure about the nowTime in general, but I would, in the same spirit, call it just now and let it return a Unix timestamp in seconds, which you can then convert into a Go time.Time with toTime if needed.

Final point: All of this must also be documented in https://github.com/prometheus/prometheus/blob/main/docs/configuration/template_reference.md .

iamhalje · 2025-06-11T10:47:07Z

@beorn7 @verdel today/tomorrow i will check all your suggestions in the Loki itself, will tell you result and will try to call the Loki developers, thanks

…seGoDuration Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje · 2025-06-12T10:24:43Z

based on discussions above, alerts should support negative duration values directly without relying on complicated workarounds. to achieve this cleanly, we will have two functions: now and parseGoDuration.

therefore, a sub-task was created to add negative duration support to model.ParseDuration in prometheus/common#793.

i hope have correctly understood everything mentioned earlier. next, i'll try to involve Loki developers in this ticket for further collaboration.

iamhalje · 2025-06-20T16:03:35Z

@cyriltovena @owen-d @chaudum @slim-bean @trevorwhitney @sandeepsukhani @jeschkies @paul1r @poyzannur FYI adds time templating for Loki alerts, would appreciate a review for anyone

beorn7 · 2025-06-22T23:03:11Z

prometheus/common has a new release tag v0.65.0, which you can update this PR to.

beorn7

See my comments. I think, if we do it this way, it will be very uncontroversial because it sticks to the pattern we already have with toTime.

It would still be good to get feedback from Loki folks that this is even needed.

template/template.go

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje · 2025-06-28T18:57:41Z

as a result, we added the functions now with prometheus type model.Time and toDuration, and replaced ParseDuration with a new function ParseDurationAllowNegative introduced in prometheus/common@v0.65.0 that support negative durations in function parseDuration

this allows us to use expressions like the following in Loki:

groups:
- name: alert
  interval: 1m
  rules:
    expr: |
      16619 == 16619
    for: 0m
    annotations:
      summary: "https://grafana.example.com/dashboard?from={{ (now.Add (toDuration (parseDuration \"-30m\"))).Time.UnixMilli }}&to={{ (now.Add (toDuration (parseDuration \"30m\"))).Time.UnixMilli }}"

beorn7

Thanks. Looks good generally, but still a few details to put straight.

docs/configuration/template_reference.md

beorn7 · 2025-07-01T13:36:43Z

template/template.go

+			//nolint:gocritic // must be a function to avoid template panics (as in Loki).
+			"now": func() model.Time {
+				return model.Now()
+			},


After some consideration, I think we should better return a float representing the number of seconds since epoch to make this useful for toTime and humanizeTime and generally the time and duration arithmetic, which is based on seconds, not milliseconds.

Suggested change

//nolint:gocritic // must be a function to avoid template panics (as in Loki).

"now": func() model.Time {

return model.Now()

},

"now": func() float64 {

now := time.Now()

return float64(now.Unix()) + float64(now.Nanosecond())/float64(time.Second)

},

In different news, it would be good to add tests for now. (For which we need to call a function that we can replace for tests, rather than time.Now()

Drive-by comment only slightly relevant to the rest of the PR:

Go added testing/synctest which mocks time, as an experiment in Go 1.24 and for real in upcoming Go 1.25.
So we could consider using that.

@bboreham thanks for pointer, but Prometheus is still using Go 1.23, we can upgrade?

Prometheus is committed to work with a few Go versions prior to the current one.

You could write a test with a build tag for the higher version, but in this simple case, I have an even better idea.

NewTemplateExpander already accepts an argument timestamp model.Time, which is essentially the evaluation time. So I would simply return that from now. This has two advantages: Easy for test. And most importantly: Consistent between different invocations of now in the same template expansion.

docs/configuration/template_reference.md

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje · 2025-07-01T16:37:54Z

with the latest changes, the working solution for us is to use

{{ ((toTime (now)).Add (toDuration (parseDuration "-30m"))).UnixMilli }}

beorn7 · 2025-07-01T17:18:52Z

with the latest changes, the working solution for us is to use
{{ ((toTime (now)).Add (toDuration (parseDuration "-30m"))).UnixMilli }}

Or maybe with a more pipeline-y approach:

{{ ( "-30m" | parseDuration | toDuration | ( now | toTime ).Add ).UnixMilli }}

beorn7 · 2025-07-01T17:32:11Z

{{ ( "-30m" | parseDuration | toDuration | ( now | toTime ).Add ).UnixMilli }}

Which is probably a good template to add to the tests.

Add parseDurationTime in templating

e7cfd48

Signed-off-by: Dmitry Ponomaryov <me@halje.ru>

beorn7 reviewed Jun 4, 2025

View reviewed changes

template/template.go Outdated Show resolved Hide resolved

template/template.go Outdated Show resolved Hide resolved

iamhalje added 3 commits June 4, 2025 22:00

rename parseDurationTime to parseGoDuration for clarity

ab54019

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

Merge branch 'add-template-parsedurationtime' of github.com:iamhalje/…

03c28d6

…prometheus into add-template-parsedurationtime

iamhalje changed the title ~~Add parseDurationTime in templating~~ Add parseGoDuration in templating Jun 4, 2025

add nowTime function to provide current timestamp in alert templates

b4bd0d6

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

beorn7 reviewed Jun 11, 2025

View reviewed changes

template/template.go Outdated Show resolved Hide resolved

iamhalje added 3 commits June 12, 2025 14:01

use model.ParseDuration instead of time.ParseDuration in function par…

cc36280

…seGoDuration Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

rename function nowTime to now

cc2e0cc

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

add 'parseGoDuration' and 'now' in docs/template_reference

a430b50

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje mentioned this pull request Jun 12, 2025

feat: Support negative duration in new function ParseDurationAllowNegative prometheus/common#793

Merged

iamhalje changed the title ~~Add parseGoDuration in templating~~ Add parseGoDuration to support time handling in Loki alerts. Jun 12, 2025

beorn7 requested changes Jun 22, 2025

View reviewed changes

template/template.go Outdated Show resolved Hide resolved

template/template.go Outdated Show resolved Hide resolved

iamhalje added 2 commits June 28, 2025 23:03

Merge branch 'main' into add-template-parsedurationtime

7b92cab

add toDuration template func and updat enow func to use model.Now

7af5aec

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje added 2 commits June 28, 2025 23:37

change documentation for now and toDuration template functions

28738e9

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

disable gocritic lint for now template func to avoid panics

e4ca9bc

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

iamhalje changed the title ~~Add parseGoDuration to support time handling in Loki alerts.~~ Add template functions to support time handling in Loki alerts. Jun 28, 2025

beorn7 requested changes Jul 1, 2025

View reviewed changes

iamhalje added 2 commits July 1, 2025 21:33

change template "now" function to return float64 timestamp

09ac028

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

update template functions docs

a8079f4

Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>

Add template functions to support time handling in Loki alerts. #16619

Are you sure you want to change the base?

Add template functions to support time handling in Loki alerts. #16619

Conversation

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!