Primary speaker:
Haroon Rafique, Manager, Development Operations, Student Information Systems, ITS
Description:
Building comprehensive infrastructure observability traditionally requires manual configuration of monitoring targets, brittle configuration management and endless YAML files. Every new server means updating Prometheus configs, managing certificates and hoping nothing breaks. What if your infrastructure could monitor itself automatically from the moment a server boots?
This talk demonstrates a production-ready architecture that combines modern Puppet patterns with automated service discovery to create truly self-configuring observability. Using Puppet's exported resources, nodes automatically register themselves for monitoring without any manual intervention. New web servers are discovered and scraped within minutes of provisioning—no configuration updates required.
You'll learn how to implement:
- Modern Puppet classification using CSR attributes instead of site.pp node definitions. Nodes self-classify by embedding their role directly in their TLS certificate, eliminating centralized configuration bottlenecks.
- Hiera-based roles defined as pure YAML data instead of Puppet manifests, making roles accessible to non-Puppet experts and enabling templated role generation.
- Automated service discovery through Puppet's exported resources. Each node exports its monitoring endpoints to PuppetDB, which a lightweight Python script queries to generate Prometheus file-based service discovery targets. Zero manual configuration required.
- Zero-trust security using Caddy as an mTLS reverse proxy. All metrics traffic is secured with mutual TLS using Puppet's existing CA infrastructure—no additional certificate management needed.
- Universal observability with node_exporter on every server and application-specific exporters (apache_exporter, etc.) automatically configured based on the node's role.
The architecture scales from a handful of servers to thousands, works seamlessly with cloud auto-scaling and provides complete visibility into both system and application metrics. Attendees will observe a demo with working code, container configurations and a deep understanding of how to build self-managing infrastructure that doesn't require operator intervention to stay observable.
