Technologies

Monitoring, observability and SRE

Monitoring tells you something is wrong before a customer does. Observability shows you why. Site reliability engineering (SRE) builds on both, tying alerts to what users experience.

I'm an observability and SRE consultant in London, working with Prometheus, Grafana, Loki, Mimir, Thanos and Grafana Alloy.

What I do

  • Set it up from nothing. Uptime checks, alerts for the things that take systems down, and somewhere to see them.
  • Scale Prometheus. Thanos or Mimir for long-term storage, large Kubernetes estates and queries that stay fast as the data grows.
  • Cut the cost and the noise. Reducing metric cardinality, and trimming a flood of alerts down to the ones that need a person.
  • Replace legacy monitoring such as Nagios and Zenoss with Grafana, Loki, Mimir and Grafana Alloy.
  • Log pipelines with Loki or the ELK stack, including the move from Promtail to Grafana Alloy. I've written up how that migration goes.
  • Dashboards, PromQL and LogQL that answer the questions your team asks.
  • Roll out agents across Linux, Windows and VMware with Ansible.

What I'd look at first

  • Where alerts go, and whether anyone reads them.
  • Whether dashboards show what customers experience, or only that servers are busy.
  • How much the monitoring costs to run.
  • Whether the monitoring would survive the failure it is meant to report.

What it costs

A basic alert setup is a small job, from £500. For anything larger, tell me what you run and I'll quote a fixed price.

I take on monitoring and observability work as an ad hoc job, fractional DevOps days each month, a contract or an audit, on site or remotely across the UK. I write about infrastructure at hamzahkhan.com. Get in touch.

Questions

Can you be involved when an alert fires?

That can be built into regular days each month. Tell me the hours you need covered.

Do we need Prometheus and Grafana, or would something simpler do?

For a single site, an uptime check plus alerts for a full disk and an expiring certificate is a sensible start, and that is a small job. The full stack earns its place once you run several services and need to see why something is slow.

Do you work with the ELK stack?

Yes: Elasticsearch, Logstash and Kibana, alongside Loki.

Tell me what you need

A problem, a project or a role you're filling. Describe it in your own words and I'll reply within one working day.