Prometheus: Up & Running, 2nd Edition

Chapter 8. Service Discovery

Thus far you’ve had Prometheus find what to scrape using static configuration via static_configs. This is fine for simple use cases,¹ but having to manually keep your prometheus.yml up to date as machines are added and removed would get annoying, particularly if you were in a dynamic environment where new instances might be brought up every minute. This chapter will show you how you can let Prometheus know what to scrape.

You already know where all of your machines and services are, and how they are laid out. Service discovery (SD) enables you to provide that information to Prometheus from whichever database you store it in. Prometheus supports many common sources of service information, such as Consul, Amazon EC2, and Kubernetes out of the box. If your particular source isn’t already supported, you can use the file-based and HTTP-based service discovery mechanisms to hook it in. For file-based service discovery, this could be by having your configuration management system, such as Ansible or Chef, write the list of machines and services they know about in the right format, or a script running regularly to pull it from whatever data source you use. For HTTP-based service discovery, third-party tools such as NetBox offer plug-ins that can be installed to offer Prometheus-compatible HTTP service discovery endpoints. Note that some SD projects² support both HTTP-based service discovery and file-based service discovery.

Knowing what your monitoring targets ...