8000 HTTP API for getting alerts · Issue #1570 · prometheus/prometheus · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HTTP API for getting alerts #1570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hryamzik opened this issue Apr 21, 2016 · 36 comments
Closed

HTTP API for getting alerts #1570

hryamzik opened this issue Apr 21, 2016 · 36 comments
Labels
component/ui kind/enhancement priority/P3

Comments

@hryamzik
Copy link

I didn't find anything about API for the /alerts in docs. I can see that /alerts returns html:

router.Get("/alerts", instrf("alerts", h.alerts))

But is it possible to get them with json?
@brian-brazil
Copy link
Contributor
brian-brazil commented Apr 21, 2016 •
edited
Loading

There is no HTTP API, however you can access the ALERTS variable via the usual query API which contains most of the same information.

@hryamzik
Copy link
Author

Looks like only firing alerts are available there and annotations are missing.

@fabxc
Copy link
Contributor
fabxc commented Apr 21, 2016

There was a PR for that a while back. We should reconsider having an API
endpoint.

I raised before that we generally want to have an API that allows to build
a fully custom UI on top. If we ever get to a point where we can build a
more dynamic default API we will need it anyway.

On Thu, Apr 21, 2016, 9:32 PM Roman Belyakovsky notifications@github.com
wrote:

Looks like only firing alerts are available there and annotations are
missing.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1570 (comment)

@hryamzik
Copy link
Author

I'm thinking of a telegram bot to get more details on alerts when internet connection is limited and web UI is difficult to access. By the way it's not mobile friendly for some reason, pretty unusual for a bootstrap based one.

@brian-brazil
Copy link
Contributor

For that sort of thing, a link to a dashboard generally works best.

@hryamzik
Copy link
Author

That's much more to load than just an annotation string. So looking forward for an API. =)

@mtanda
Copy link
Contributor
mtanda commented Apr 27, 2016

Personally I want this to embed Alert status in Grafana.
Grafana scripted dashboard (or panel plugin) can make alert dashboard.
(I make this by parsing HTML, mtanda/grafana@6b357c3 )

If Prometheus provide API, it's very helpful.

@iksaif
Copy link
Contributor
iksaif commented Feb 17, 2017

ALERTS is fine but doesn't seem to show inactive alerts. An API would be great.

Looked quickly at the code, looks like simply converting to json the result of h.ruleManager.AlertingRules() would be a good start.

@fabxc
Copy link
Contributor
fabxc commented Feb 22, 2017

Such an API would get accepted as part of our effort to make all UI features represented in our HTTP API. Contributions are gladly accepted.

@copernico
Copy link

+1

@mg03
Copy link
mg03 commented Apr 1, 2017

@fabxc : i have some code written to expose endpoint /api/v1/alerts which puts out a json like so

{
  "status": "success",
  "data": {
    "alerts": [
      {
        "name": "promup",
        "query": "up{job=\"prometheus\"} != 1",
        "duration": "1m0s",
        "labels": {
          "severity": "critical"
        },
        "annotations": {
          "dashboard": "http://grafana.com/dashboard/somedashbaord",
          "description": "lol hahahahha",
          "summary": "Prom Up"
        },
        "status": "inactive"
      },
      {
        "name": "promx",
        "query": "up{job=\"prom\"} < 1",
        "duration": "1m5s",
        "labels": {
          "severity": "critical"
        },
        "annotations": {
          "dashboard": "http://grafana.com/dashboard/somedashbaord",
          "description": "sad sad",
          "summary": "Promxxxxx Up"
        },
        "status": "pending",
        "activesince": "2017-03-31T19:09:07.808-07:00"
      },
      {
        "name": "promnoann",
        "query": "up{job=\"prom\"} < 1",
        "duration": "10s",
        "labels": {
          "severity": "critical"
        },
        "status": "firing",
        "activesince": "2017-03-31T19:09:07.808-07:00"
      }
    ]
  }
}

Is there any other info that I can expose via the api?
I have written a test case for the above api.
I can make a PR to the repo, would that be acceptible or is there some process (sorry noob to contributing)

This was referenced Apr 4, 2017
Closed
Closed
@moolitayer
Copy link
moolitayer commented May 17, 2017 •
edited
Loading

I don't understand, there already seems to be an API endpoint exposed from alert manager.
That one seems to show also alerts that no longer firing:

GET http://alerts-vm-48-125.myserver.com/api/v1/alerts

{"status":"success","data":[{"labels":{"alertname":"Testing","instance":"vm-48-124.eng.lab.tlv.redhat.com","job":"kubernetes-nodes","monitor":"codelab-monitor"},"annotations":{"description":"a description","summary":"a summary"},"startsAt":"2017-05-17T23:24:37.804Z","endsAt":"2017-05-17T23:27:37.804Z","generatorURL":"http://prometheus-1161204464-e5li2:9090/graph?g0.expr=process_cpu_seconds_total%7Binstance%3D%22vm-48-124.eng.lab.tlv.redhat.com%22%2Cjob%3D%22kubernetes-nodes%22%7D+%3C+4615\u0026g0.tab=0"}]}

prometheus time series collection and processing server

Where does that fit in this discussion?

@jsuchenia
Copy link

@moolitayer In a project name alertmanager != prometheus. However with federation you can scrap alerts like metrics:
http://prometheus.instance/federate?match[]=ALERTS

@moolitayer
Copy link

So there is an API for alerts in alert manager but not in Prometheus?

@juliusv
Copy link
Member
juliusv commented May 29, 2017

@moolitayer Correct.

@moolitayer
Copy link

Thanks @juliusv. I'm interested in collecting alerts from Prometheus and integrating them in an external system. Here is another related question;

For effective collection I would like like to collect only new alerts using pull every X seconds.
Is there a mechanism I can use to get only new alerts? (happened or were resolved since last collection)

(possible solutions: using a time field OR using labels for collected alerts OR a streaming api similar to the kubernetes events endpoint)

@jsuchenia
Copy link

@moolitayer It doesn't make sense. From prometheus point of view it's collecting data every X time and evaluates rules every Y time. If you are scrapping alerts every Z seconds and it's more than Y then you expect that prometheus will keep historical alerts till your next check. Additionally how to handle restarts and your problems (like you lost updates about some resolve)

So it's quite good that at one shot you can get a list of active alarms - you have a clear picture of a status

@moolitayer
Copy link

@jsuchenia It seems alert manager already has that data and I'm wondering if there is a mechanism there to get what I need? If an alert has fired and was immediately resolved for example I would still like to know about it so only resolved alerts are not enough for my use case.

@jsuchenia
Copy link

@moolitayer According to Alert Manager configuration you can specify a webhook that will be triggered whenever an alert state was changed. Is this something that works for you?

@moolitayer
Copy link

@moolitayer According to Alert Manager configuration you can specify a webhook that will be triggered whenever an alert state was changed. Is this something that works for you?

Thanks @jsuchenia. I have to initiate all the connections due to security concerns. Is that possible somehow? (kubernetes watch api is a good example of what I need)

@brian-brazil
Copy link
Contributor

@brancz Did you implement this already?

@brancz
Copy link
Member
brancz commented Jul 17, 2017

@brian-brazil I implemented the alertmanagers endpoint, not alerts. That's still tbd. However as suggested before I would recommend to implement a system as described above as a webhook that is called by the Alertmanager instead of scraping Prometheus alerts. Nonetheless there are usecases for the alerts API.

@moolitayer
Copy link

In case this is of useful to someone, to collect alerts we web hooked alerts into another component (running on the same k8s pod as alert manager) and we are collecting from that component:
https://github.com/openshift/prometheus-alert-buffer

@midnightconman
Copy link

I have a use case where I would like to see triggered and non-triggered alerts, the current alertmanager alerts API currently only exposes triggered alerts. Would you guys welcome a PR to add functionality to be able to glean all alerts? I don't need an API, just the information exposed... I would be happy with adding a label to the ALERTS metric and exposing all alerts there. Thoughts?

@brian-brazil
Copy link
Contributor

It's not possible to list non-firing alerts, as we don't know their labels.

@midnightconman
Copy link

That's a good point... I really don't need an alerts api, but a rules api. I don't mind if it is read-only, just something better than html. In Prometheus 2.0 using /api/v1/rules I am getting: HTTP/1.1 405 Method Not Allowed

@iksaif
Copy link
Contributor
iksaif commented Dec 26, 2017

Maybe a good extension for /api/v1/status/config ?

@midnightconman
Copy link

#3656

@omadawn
Copy link
omadawn commented Feb 9, 2018

I also have an existing need to be able to retrieve all rules via an API (mainly non-firing alerts.) Either #3656 or #3391 should address my need. Is there a plan to implement it yet and can I vote for this functionality if not?
< A3E2 /div>

@burnettk
Copy link

In case anyone comes in here looking for an firing alert dashboard in grafana, here's one: https://grafana.com/dashboards/4181

@amaury-d
Copy link

Hello,

For anyone else looking at the answer, you can call /api/v1/query?query=ALERTS on Prometheus

@simonpasquier
Copy link
Member

See #4318

@ealexhaywood
Copy link

@amaury-d yes, but that only returns firing alerts. People here want inactive and pending alerts as well

@mxinden
Copy link
Member
mxinden commented Aug 13, 2018

@ealexhaywood #4318 exposes pending alerts via api/v1/alerts as well. Inactive alerts are not exposed.

@juliusv as the initial issue is about getting alerts in general, should we close here with #4318 being merged?

@juliusv
Copy link
Member
juliusv commented Aug 13, 2018

@mxinden Yes, thank you! Closing.

@juliusv juliusv closed this as completed Aug 13, 2018
@lock
Copy link
lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
component/ui kind/enhancement priority/P3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0