SMTP relay for operational alerts and system notifications

Why ops email is its own workload

Teams new to infrastructure operations often treat alert email as a side effect — something that gets configured once when a service is deployed and then rarely revisited. This works until it doesn't. The first sign of trouble is usually an incident post-mortem where someone notices that the alert that should have fired either wasn't sent or wasn't delivered. By the time that investigation is complete, the team has discovered that the local sendmail on their monitoring host has been silently bouncing mail for six weeks because of a deliverability issue nobody was watching. The underlying mechanics of how SMTP delivers mail in the first place — and why it's still the universal transport for machine-to-machine notifications — are covered in SMTP fundamentals.

Operational email has a specific profile that distinguishes it from product email:

The "failure cost is hidden" property is what makes operational email uniquely dangerous. Product mail failures surface quickly because recipients notice their missing password resets or receipts. Operational mail failures only surface when something else breaks badly enough to require the ops team, at which point the missing alerts are discovered retroactively. By then the incident has been running for longer than it should have.

The sources that feed the alert stream

Before designing the relay, an accurate inventory of which systems actually need to send email is essential. The list is longer than most teams expect when they first write it down.

1. Monitoring systems. Prometheus Alertmanager, Nagios, Icinga2, Zabbix, Sensu, Grafana alerting, Datadog (email notifications), New Relic, AppDynamics. Every monitoring tool has an email path.
2. Scheduled jobs. cron jobs on individual servers, Jenkins builds, CI/CD pipeline notifications, scheduled reports, database maintenance scripts. Each cron line with MAILTO= is a potential sender.
3. Backup and storage systems. rsync jobs, BorgBackup, restic, AWS Backup notifications, filesystem monitoring tools like AIDE or Tripwire that emit change reports.
4. Security tools. fail2ban ban/unban notifications, ClamAV scan results, OSSEC / Wazuh alerts, intrusion detection events, certificate renewal notifications.
5. Database systems. PostgreSQL log alerts, MySQL slow-query notifications, MongoDB ops manager, Redis persistence alerts, database migration tool output.
6. Build and deploy tools. Jenkins, GitLab CI, CircleCI, TeamCity email notifications on build failure or deployment completion.
7. Infrastructure-as-code. Terraform apply reports, Ansible notification handlers, Chef run notifications.
8. Application logs. Applications themselves that emit email on specific error conditions (usually via log alerting rather than direct email, but not always).

An organization with ten or more servers typically has thirty or forty distinct email-emitting sources, most of which were configured piecemeal over time by different engineers. Auditing this list is the first step of any serious operational-email cleanup. The audit itself often surfaces sources that have been broken for months without anyone noticing — exactly the pattern operational mail is supposed to prevent.

Email vs paging: when to use each

A common mistake in ops email architecture is trying to do everything through email when some signals belong in paging tools instead. The channel matters; the boundary between email and paging is one of the more consequential design decisions for an operations team.

The simplest working heuristic: if the response is "I'll fix that when I get in tomorrow morning," it's email. If the response is "I need to fix that right now," it's paging. Sorting signals into these buckets before writing the alerting rules saves a lot of time later — alert fatigue is a real operational hazard, and it almost always comes from paging on things that should have been email.

The best paging tools in 2017 — PagerDuty, OpsGenie, VictorOps, Statuspage's incident management — all support email as a lower-urgency channel alongside their paging features. Most operations teams now run both: PagerDuty or OpsGenie as the primary paging service for critical alerts, and an email flow for the informational stream. Both need reliable email infrastructure underneath; PagerDuty sends notifications via email as well as via SMS and phone calls, and its own email-in feature relies on your email being reliably delivered to PagerDuty's servers.

The relay that lives next to the monitoring stack

The architecture that works consistently across the organizations we've helped has a specific shape: every source of operational email submits to a shared relay host (or a small cluster of them), and that relay in turn delivers to an external provider or to the recipient mailboxes. The local relay is a thin layer whose job is to aggregate submissions, authenticate them, apply basic policy, and hand them off.

Why the local-relay layer matters

Every operational-email source could theoretically submit directly to an external provider. Some teams do this. The architecture works, but it creates problems:

• Credentials for the external provider end up stored on every server, which is a security problem and an auditing problem.
• Rate limiting, deduplication, and filtering have to be implemented per source rather than centrally.
• Logs of what was sent are scattered across every server rather than collected in one place.
• Authentication records (SPF, DKIM) become harder to manage because every source can theoretically send from any address.
• If the external provider has an outage, every source needs to handle the failure independently.

The local-relay pattern — Postfix or Exim running as a submission proxy — addresses each of these. The external provider's credentials live in one place. Policies are applied once. Logs aggregate. Authentication is consistent. Outage handling happens at the relay, where it's observable and fixable.

Configuration patterns for Postfix as the ops relay

Postfix is the most commonly deployed ops-mail relay in Linux environments. The configuration is modest once the pattern is internalized. The key choices: listen only on the internal network, authenticate submissions from known hosts, sign outbound mail with DKIM, relay to an external provider with proper credentials.

postfix main.cf — ops relay (abbreviated)# Listen on the internal network only
inet_interfaces          = 10.0.1.25
# Do not accept mail from the internet
mynetworks               = 10.0.0.0/8, 127.0.0.0/8, [::1]/128
mynetworks_style         = host

# We are a submission proxy, not a final destination
mydestination            =
local_transport          = error:local delivery disabled

# Require sender validation from clients
smtpd_sender_restrictions =
    permit_mynetworks,
    reject_non_fqdn_sender,
    reject_unknown_sender_domain

# Relay restrictions — anti-open-relay
smtpd_relay_restrictions =
    permit_mynetworks,
    reject_unauth_destination

# Relay outbound through an external provider
relayhost                = [smtp.sendgrid.net]:587
smtp_sasl_auth_enable    = yes
smtp_sasl_password_maps  = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level  = encrypt

# DKIM signing (via OpenDKIM milter)
milter_default_action    = accept
milter_protocol          = 6
smtpd_milters            = inet:127.0.0.1:8891
non_smtpd_milters        = $smtpd_milters

# Modest queue behavior for ops mail
maximal_queue_lifetime   = 1d
bounce_queue_lifetime    = 1d
# Aggressive retry for short outages
minimal_backoff_time     = 60s
maximal_backoff_time     = 900s

# Rate limit to prevent runaway alert storms
smtpd_client_message_rate_limit  = 60
smtpd_client_connection_rate_limit = 30

The sasl_passwd file

/etc/postfix/sasl_passwd# External provider credentials — file is chmod 600, root-only
[smtp.sendgrid.net]:587    apikey:SG.your_api_key_here
# After editing:
#   postmap hash:/etc/postfix/sasl_passwd
#   postfix reload

Three operational choices that matter. First, the rate limits (smtpd_client_message_rate_limit) catch runaway alert storms — a misconfigured cron that loops and sends a thousand mails in a minute gets cut off before it burns through your external provider's quota. Second, the short backoff times (minimal_backoff_time = 60s) mean transient external-provider problems recover quickly without hours of delay. Third, DKIM signing via OpenDKIM ensures every alert from this relay is properly authenticated — broken DKIM on alert mail is one of the most common causes of alerts landing in spam folders. The underlying DKIM mechanics that OpenDKIM wraps are detailed in DKIM signing for transactional email.

Alertmanager's SMTP configuration in detail

Prometheus Alertmanager is the canonical modern monitoring tool, and its SMTP configuration reveals the patterns that translate to the rest of the ecosystem. The Alertmanager can route alerts through email, PagerDuty, Slack, OpsGenie, and various webhooks; the email configuration is straightforward once the relay pattern is in place.

alertmanager.yml — email via local relayglobal:
  # Point at the local Postfix relay — no auth needed from internal hosts
  smtp_smarthost:  'ops-relay.internal:25'
  smtp_from:       'alerts@example.com'
  smtp_require_tls: false   # Local relay; TLS handled at the external hop

  # Global timing
  resolve_timeout: 5m

route:
  # Default routing tree
  group_by:       ['alertname', 'cluster', 'service']
  group_wait:     30s      # Wait to aggregate initial alerts in a group
  group_interval: 5m       # Send grouped updates every 5 minutes
  repeat_interval: 12h     # Repeat an active alert every 12 hours

  # Routing by severity
  receiver: 'email-ops'
  routes:
    # Critical goes to paging
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true       # Also send email for the record

    # Warnings go to a separate email list
    - match:
        severity: warning
      receiver: 'email-warnings'

receivers:
  - name: 'email-ops'
    email_configs:
      - to: 'ops-alerts@example.com'
        send_resolved: true
        headers:
          Subject: '[OPS] {{ .CommonLabels.alertname }}'

  - name: 'email-warnings'
    email_configs:
      - to: 'ops-warnings@example.com'
        send_resolved: true

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'YOUR-PAGERDUTY-INTEGRATION-KEY'
        send_resolved: true

inhibit_rules:
  # If the whole cluster is down, suppress lower-severity alerts
  - source_matchers:
      - severity=~"critical"
      - alertname="ClusterDown"
    target_matchers:
      - severity=~"warning|info"
    equal: ['cluster']

The grouping and repeat_interval decisions

Alertmanager's grouping and repeat behaviors are what prevent alert storms from becoming unreadable. group_wait is the delay before sending the first notification for a new alert group; it batches alerts that arrive in quick succession. group_interval is the delay between updates for an already-active group. repeat_interval is how often the same active alert is re-notified.

For most operational email flows, repeat_interval: 12h is a reasonable starting point — an alert that's been firing for 12 hours gets re-sent so the problem doesn't quietly fade from attention. For paging flows, the repeat interval is often shorter (1-4 hours) to ensure someone sees the issue even if the first page was missed.

Nagios, Zabbix, and Icinga email plumbing

The older-generation monitoring tools — Nagios, its fork Icinga2, and Zabbix — have been the workhorses of operations teams for years. Their email configurations differ in the specifics but share the pattern of shelling out to sendmail or equivalent, which means they inherit whatever the local MTA does.

Nagios and Icinga2

Nagios/Icinga2 command definitiondefine command {
    command_name  notify-host-by-email
    command_line  /usr/bin/printf "%b" "***** Nagios *****\n\n\
        Notification Type: $NOTIFICATIONTYPE$\n\
        Host: $HOSTNAME$\n\
        State: $HOSTSTATE$\n\
        Address: $HOSTADDRESS$\n\
        Info: $HOSTOUTPUT$\n\n\
        Date/Time: $LONGDATETIME$\n" | \
        /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" \
        -S smtp=smtp://ops-relay.internal:25 \
        -r "nagios@example.com" $CONTACTEMAIL$
}

The mail command hands off to the local MTA via /usr/sbin/sendmail, which Postfix provides. If Postfix is configured as an ops relay per the earlier pattern, Nagios alerts flow through the same pipeline as everything else — same authentication, same DKIM signing, same rate limits, same logs.

Zabbix

Zabbix Email Media Type (abbreviated)# Configure → Administration → Media types → Email
#
# Name:               Email
# Type:               Email
# SMTP server:        ops-relay.internal
# SMTP server port:   25
# SMTP helo:          monitor01.example.com
# SMTP email:         zabbix@example.com
# Connection security: STARTTLS
# Authentication:     None   (local relay, no auth needed)
#
# Message format:     HTML
# Enabled:            Yes
#
# Then on Actions → Trigger actions → create action with:
#   Send to users: "ops-team" (group)
#   Subject: {TRIGGER.STATUS}: {TRIGGER.NAME}
#   Message: ...

Zabbix speaks SMTP directly rather than shelling out to the local MTA, which is a minor difference in failure modes: if the local relay is down, Zabbix alerts fail at the SMTP level rather than queuing locally. Either behavior is acceptable; just know which you have.

Cron jobs: the forgotten alert source

Every Unix system's cron daemon has the ability to mail cron output to an address specified in the MAILTO environment variable. This feature is powerful and almost universally misconfigured. The default behavior — if MAILTO is set, output goes to that address; if not, to the owning user's local mailbox — means that many cron jobs are silently landing in /var/mail/root on individual servers and nobody is reading them.

/etc/crontab — doing cron mail correctly# Send cron mail to a centralized address that the ops team actually reads
MAILTO=ops-cron@example.com
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Cron jobs — stdout/stderr that produces output triggers email
# Nightly database backup
0 2 * * * postgres /usr/local/bin/backup_database.sh

# Silent when successful; output (errors) triggers email
*/5 * * * * root /usr/local/bin/check_disk_space.sh

# Explicitly opt out of emails for chatty jobs (pipe to /dev/null)
0 * * * * root /usr/local/bin/rotate_logs.sh > /dev/null 2>&1

# Explicitly override MAILTO for specific jobs
# (use a per-job format with `MAILTO=` on its own line just before the job)

The cron-mail audit

Running find /etc/cron* -type f on every server and correlating against your centralized mailbox for cron mail reveals the set of cron jobs that are actually emitting email. Comparing this to the list of cron jobs you expect to exist often reveals:

• Cron jobs that have been silently failing for weeks (job produced output but nobody was reading it).
• Cron jobs that should be mailing but aren't (because MAILTO was never set on that host).
• Cron jobs that are too chatty (emitting routine success messages that drown out real failures).
• Cron jobs that belong to engineers who have since left the organization.

The audit is usually revealing the first time it's done. The cleanup is modest: silence the too-chatty jobs with explicit output redirection, fix the MAILTO for silent hosts, decide what to do with orphaned jobs. The discipline afterward is to treat cron mail as an ongoing hygiene concern rather than a one-time fix.

The incident-outbound failure mode

The most important failure mode in operational email is also the most overlooked: during a serious incident, the system that's supposed to be alerting is often the same system having the problem. A relay running on a host that's having disk issues can't deliver the disk-full alert. A relay that depends on DNS resolution can't send when DNS is broken. A relay that depends on the network can't notify you that the network is down.

Design patterns for resilience

The heartbeat pattern specifically

The single most valuable resilience pattern is the heartbeat: a scheduled "everything is fine" alert that fires every hour. If it stops arriving, you know the alerting pipeline is broken even if no actual incidents are firing. Implementing this takes fifteen minutes and catches a class of silent failures that would otherwise remain invisible for weeks.

hourly heartbeat cron# On the monitoring host
0 * * * * root echo "Alerting pipeline heartbeat from $(hostname) at $(date)" | \
    mail -s "HEARTBEAT: ops alerting alive" ops-heartbeat@example.com

# On the recipient side, configure an inbox rule that expects one of these
# per hour. If two consecutive heartbeats are missing, page on-call via
# an entirely separate channel (PagerDuty, manual SMS, etc.)

Alert hygiene: rate-limiting and deduplication

Operational email stops being useful when there's too much of it. The team that gets five alerts a day reads them carefully; the team that gets five hundred reads none of them. Keeping the volume manageable is a design problem on the alerting rule side and a protection problem at the relay.

Strategies for alert volume control

1. Group at the source. Alertmanager's grouping, Nagios's notification intervals, and similar features collapse bursts of related alerts into single notifications. Use them aggressively.
2. Inhibit redundant alerts. When "the cluster is down" fires, inhibit the "individual service X is unreachable" alerts that are clearly caused by it. Inhibition rules in Alertmanager and equivalents handle this.
3. Severity thresholds. An alert that fires at 85% disk doesn't also need to fire at 86%, 87%, 88%. Use hysteresis: fire once on the transition into the alert state, fire again only on transition out (resolved) or escalation.
4. Rate-limit at the relay. As shown in the Postfix config earlier — smtpd_client_message_rate_limit catches runaway sources before they flood recipients.
5. Consolidate into digests where appropriate. A "nightly summary" email that aggregates many low-priority informational items is often more useful than 50 individual emails, each of which is easy to ignore.

The uncomfortable question about alert hygiene

Most operations teams have alerts that fire regularly and nobody does anything about. These are noise by definition — if the response is "we just ignore that one," the alert isn't adding value and is diluting the attention available for alerts that do matter. Twice a year, every team should audit which alerts fire, which ones generated action, and which ones were ignored. The ignored ones get deleted or rewritten to fire less often.

Authentication for alert mail

Operational email needs SPF, DKIM, and DMARC just as much as product email does. The reason is simple: if alerts@example.com is poorly authenticated, recipients' mail systems treat it as suspicious, and suspicious ops mail ends up in spam folders at the worst possible moments.

The minimum authentication for ops email

1. SPF record on your organizational domain that includes the external relay you use. If ops mail goes out via AWS SES or SendGrid, those vendors' SPF includes need to be in your record.
2. DKIM signing at the local relay. OpenDKIM milter for Postfix, or the vendor's signing if you rely on SES/SendGrid to sign. Either way, DKIM should be present on every alert.
3. DMARC record with at least p=none. Aggregate reports show whether alert mail is passing authentication for real. Most ops teams discover broken DKIM via DMARC reports long before a human notices. The DMARC rollout playbook from the first wave of adopters, which applies directly here, is covered in DMARC in practice.
4. Dedicated subdomain for ops mail. Use ops.example.com or alerts.example.com for operational mail. Keeps the authentication clean and prevents ops volume from affecting the organizational domain's reputation.

Ops mail volume is low enough that the authentication work is also low. Getting it right once pays off indefinitely; neglecting it produces the failure mode where critical alerts start landing in spam at random. Do the setup early; audit quarterly; move on.

Frequently asked questions

Can't we just use AWS SES directly from every source?

You can. The cost is scattered credentials, scattered logs, scattered policy. For small setups (a handful of servers, simple stack), direct-to-SES works. For anything beyond that, a local relay pays for itself quickly in operational simplicity.

What if we don't have a monitoring stack yet?

Start with cron mail and scheduled-job notifications, which exist whether you have monitoring or not. Add Prometheus+Alertmanager or Icinga2 when ready; the relay layer is already in place when they arrive.

Should we use a paging provider for everything and skip email?

No. Paging providers are optimized for wake-up-now signals, not for the informational stream of backup reports, nightly summaries, and low-priority notifications. Using paging for everything produces alert fatigue; using email for everything misses the critical signals. Use both with clear boundaries.

How do we handle email for international teams in different time zones?

Most paging providers handle time-zone-aware rotation automatically. For email, the typical pattern is a shared ops mailing list that follows the sun, with specific individuals on-call for specific time windows. The details belong in operational runbooks, not in alert routing.

What about Slack or Teams instead of email for ops alerts?

Slack or Teams is fine for some informational alerts and team visibility, but as a primary channel it has limitations: doesn't persist well in archived form, depends on a third-party service being available, and can overwhelm channels when alert volume is high. Most mature operations teams run Slack/Teams as an additional channel for visibility, with email remaining the primary archival record and paging remaining the critical-urgency channel.

How do we test the alerting pipeline without causing false alarms?

Heartbeat alerts (as described earlier) continuously verify the pipeline. For end-to-end testing, most teams schedule a monthly "drill" — a synthetic alert that exercises the full alerting path, observed by the team, and confirmed to produce the expected notifications. The exercise is short and surfaces drift before real incidents expose it.

What's a good starting point for a team that currently has nothing?

Install Postfix on one monitoring host, configure it as a relay per the pattern in this article, point everything at it (cron MAILTO, Nagios/Alertmanager smtp_smarthost, etc.), verify that mail is flowing, add SPF+DKIM+DMARC for the sending domain, set up a heartbeat. Two days of work, and you have a baseline that will serve for years.

Closing perspective

Operational email is the background infrastructure that makes operations teams effective. When it works — reliably, auditably, with the right signals reaching the right channels — it's invisible. When it doesn't work, it fails silently, and the cost shows up during the next incident as a post-mortem finding that the alert didn't reach the people who needed to see it.

The engineering effort to get operational email right is modest. A well-configured local relay, sensible integration with monitoring tools, clear separation of email from paging, cron mail hygiene, authentication for the sending domain, and a heartbeat to catch silent failures — none of this is hard. What makes it uncommon is that each piece has to be done deliberately, and teams that haven't had a painful incident recently often deprioritize the work in favor of more visible projects.

The teams we've seen do this well share a specific pattern: someone owns operational email as a named responsibility. It's not "everyone's job"; it's a specific engineer who maintains the relay, reviews the alert catalog, audits cron output, and periodically verifies the pipeline end-to-end. The ownership is what keeps the discipline alive after the initial setup. Without it, operational email drifts back to the scattered-and-broken state that characterizes most mid-stage operations stacks.

For teams considering whether this is worth the effort: look at your last three post-mortems. Count how many had some variant of "alert didn't fire" or "alert fired but wasn't received" as a contributing factor. If any of them did, the work described in this article would have prevented it. That's usually the argument that gets it prioritized. The investment is small; the protection is large; the pattern is well-understood. It's worth the afternoon to set up properly. And because the recipients' mail systems apply the same scrutiny to ops mail as to any other sender, the underlying sender reputation fundamentals that govern whether alerts reach the inbox at all are worth internalizing for anyone maintaining this infrastructure.