mirror of
https://github.com/prometheus/prometheus
synced 2026-04-21 07:00:27 +08:00
b70a871988
Building off config-specific Prometheus refresh metrics from an earlier PR (https://github.com/prometheus/prometheus/pull/17138), this deletes refresh metrics like `prometheus_sd_refresh_duration_seconds` and `prometheus_sd_refresh_failures_total` when the underlying scrape job configuration is removed on reload. This reduces un-needed cardinality from scrape job specific metrics while still preserving metrics that indicate overall health of a service discovery engine. For example, `prometheus_sd_refresh_failures_total{config="linode-servers",mechanism="linode"} 1` will no longer be exported by Prometheus when the `linode-servers` scrape job for the Linode service provider is removed. The generic, service discovery specific `prometheus_sd_linode_failures_total` metric will persist however. * fix: add targetsMtx lock for targets access * test: validate refresh/discover metrics are gone * ref: combine sdMetrics and refreshMetrics Good idea from @bboreham to combine sdMetrics and refreshMetrics! They're always passed around together and don't have much of a reason not to be combined. mechanismMetrics makes it clear what kind of metrics this is used for (service discovery mechanisms). --------- Signed-off-by: Will Bollock <wbollock@linode.com>
103 lines
3.6 KiB
Markdown
103 lines
3.6 KiB
Markdown
---
|
|
title: Writing HTTP service discovery
|
|
nav_title: HTTP SD
|
|
sort_rank: 7
|
|
---
|
|
|
|
Prometheus provides a generic [HTTP Service Discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config),
|
|
that enables it to discover targets over an HTTP endpoint.
|
|
|
|
The HTTP Service Discovery is complementary to the supported service
|
|
discovery mechanisms, and is an alternative to [File-based Service Discovery](https://prometheus.io/docs/guides/file-sd/#use-file-based-service-discovery-to-discover-scrape-targets).
|
|
|
|
## Comparison between File-Based SD and HTTP SD
|
|
|
|
Here is a table comparing our two generic Service Discovery implementations.
|
|
|
|
| Item | File SD | HTTP SD |
|
|
| ---- | ------- | ------- |
|
|
| Event Based | Yes, via inotify | No |
|
|
| Update frequency | Instant, thanks to inotify | Following refresh_interval |
|
|
| Format | Yaml or JSON | JSON |
|
|
| Transport | Local file | HTTP/HTTPS |
|
|
| Security | File-Based security | TLS, Basic auth, Authorization header, OAuth2 |
|
|
|
|
## Requirements of HTTP SD endpoints
|
|
|
|
If you implement an HTTP SD endpoint, here are a few requirements you should be
|
|
aware of.
|
|
|
|
The response is consumed as is, unmodified. On each refresh interval (default: 1
|
|
minute), Prometheus will perform a GET request to the HTTP SD endpoint. The GET
|
|
request contains a `X-Prometheus-Refresh-Interval-Seconds` HTTP header with the
|
|
refresh interval.
|
|
|
|
The SD endpoint must answer with an HTTP 200 response, with the HTTP Header
|
|
`Content-Type: application/json`. The answer must be UTF-8 formatted.
|
|
If no targets should be transmitted, HTTP 200 must also be emitted, with
|
|
an empty list `[]`. Target lists are unordered.
|
|
|
|
Prometheus caches target lists. If an error occurs while fetching an updated
|
|
targets list, Prometheus keeps using the current targets list. The targets list
|
|
is not saved across restart. The `prometheus_sd_refresh_failures_total` counter
|
|
metric tracks the number of refresh failures and the `prometheus_sd_refresh_duration_seconds`
|
|
bucket can be used to track HTTP SD refresh attempts or performance. These metrics are
|
|
removed when the underlying scrape job disappears on Prometheus configuration reload.
|
|
|
|
The whole list of targets must be returned on every scrape. There is no support
|
|
for incremental updates. A Prometheus instance does not send its hostname and it
|
|
is not possible for a SD endpoint to know if the SD requests is the first one
|
|
after a restart or not.
|
|
|
|
The URL to the HTTP SD is not considered secret. The authentication and any API
|
|
keys should be passed with the appropriate authentication mechanisms. Prometheus
|
|
supports TLS authentication, basic authentication, OAuth2, and authorization
|
|
headers.
|
|
|
|
## HTTP_SD format
|
|
|
|
```json
|
|
[
|
|
{
|
|
"targets": [ "<host>", ... ],
|
|
"labels": {
|
|
"<labelname>": "<labelvalue>", ...
|
|
}
|
|
},
|
|
...
|
|
]
|
|
```
|
|
|
|
|
|
Examples:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"targets": ["10.0.10.2:9100", "10.0.10.3:9100", "10.0.10.4:9100", "10.0.10.5:9100"],
|
|
"labels": {
|
|
"__meta_datacenter": "london",
|
|
"__meta_prometheus_job": "node"
|
|
}
|
|
},
|
|
{
|
|
"targets": ["10.0.40.2:9100", "10.0.40.3:9100"],
|
|
"labels": {
|
|
"__meta_datacenter": "london",
|
|
"__meta_prometheus_job": "alertmanager"
|
|
}
|
|
},
|
|
{
|
|
"targets": ["10.0.40.2:9093", "10.0.40.3:9093"],
|
|
"labels": {
|
|
"__meta_datacenter": "newyork",
|
|
"__meta_prometheus_job": "alertmanager"
|
|
}
|
|
}
|
|
]
|
|
```
|
|
|
|
## HTTP SD integrations
|
|
|
|
A list of existing HTTP SD integrations can be found on the [Integrations page](https://prometheus.io/docs/operating/integrations/#http-service-discovery) in the Prometheus documentation.
|