Know When You've Been Touched: SSH Alerts + Intrusion Detection

This post is part 3 of a four-part series Locking Down Your VPS.

Automated updates and Fail2ban handle the noise: bots hammering your SSH port, wordlists running through common passwords. But they don't tell you two things you actually need to know: who logged in successfully, and whether anything on the server changed while you weren't looking.

This post covers both. A PAM hook that pushes a notification to your phone the moment anyone logs in over SSH, and a cron job that checks seven indicators of compromise every 15 minutes and alerts you the second one of them changes.

The login alerting is the part that mirrors what my friend used to do without any of this automation; he'd just happen to mention when he'd seen me log in. Turning that habit into a script means you're not relying on someone else to notice on your behalf. The intrusion detection half takes the same idea further: not just who logged in, but what changed on the server while nobody was watching.

Both alerts route through Ntfy, a lightweight self-hosted push notification service. If you don't have Ntfy running yet, the setup is a one-time cost: install it, create a topic, subscribe on your phone. Everything after that is a curl call to a URL.

Why This, After Fail2ban

Fail2ban reacts to failure. It watches for repeated bad logins and bans the source. That's valuable, but it says nothing about a successful login, and it says nothing about what happens to the filesystem after someone (or something) gets in.

Two gaps, two tools:

1. SSH login alerting tells you the moment a session opens, successful or not banned.
2. Intrusion detection tells you when the server itself changes in ways that shouldn't happen without your knowledge.

Neither replaces the other. Together they cover the before, the during, and the after.

Before You Start: Set Up Ntfy

Both scripts authenticate to Ntfy with a username and password, and both push to a topic. Create a dedicated topic for this server so alerts don't blend into your general notifications:

1. In your Ntfy web UI, subscribe to a new topic, for example vps-alerts
2. Subscribe to the same topic in the Ntfy phone app
3. Have your Ntfy username and password ready; both scripts read them from a shared credentials file

Directory and Credentials Setup

Both scripts share a common directory structure and a single credentials file. Already done if you set this up while following the auto-updates post; skip ahead to Part A if so.

sudo mkdir -p /root/detection/logs
sudo chmod 700 /root/detection
sudo chmod 700 /root/detection/logs

Create the credentials file:

sudo nano /root/detection/.auth

Two lines: Ntfy username on line 1, password on line 2.

yourusername
yourpassword

Lock it down:

sudo chmod 600 /root/detection/.auth

Both scripts read this file. Set it up once and forget it.

Part 1: SSH Login Alerting

This one fires a notification every time someone successfully logs in over SSH. It uses PAM's pam_exec module to run a Perl script on session open, no polling, no delay.

How it works:

1. PAM fires the script on every SSH session event
2. The script checks PAM_TYPE=open_session and ignores logouts
3. It reads Ntfy credentials from /root/detection/.auth
4. It reads PAM environment variables: username, source IP, service
5. It sends a high-priority Ntfy push with host, user, and source IP
6. It logs the event locally

The Script

Save as /usr/local/bin/ssh_alert.pl:

#!/usr/bin/perl
# ssh_alert.pl - PAM exec script for SSH login alerting

use strict;
use warnings;

# --- CONFIG ---
my $NTFY_URL  = "https://ntfy.yourdomain.com/vps-alerts";
my $AUTH_FILE = "/root/detection/.auth";
my $LOG_FILE  = "/root/detection/logs/detect.log";

# --- only fire on session open, not close ---
my $pam_type = $ENV{PAM_TYPE} // '';
exit 0 unless $pam_type eq 'open_session';

# --- load ntfy credentials ---
my ($NTFY_USER, $NTFY_PASS) = ('', '');
open(my $afh, '<', $AUTH_FILE) or do {
    log_msg("[ERROR] ssh_alert.pl cannot read $AUTH_FILE: $!");
    exit 0;
};
$NTFY_USER = <$afh>; chomp $NTFY_USER;
$NTFY_PASS = <$afh>; chomp $NTFY_PASS;
close $afh;

unless ($NTFY_USER && $NTFY_PASS) {
    log_msg("[ERROR] ssh_alert.pl .auth file missing user or pass");
    exit 0;
}

# --- read pam environment ---
my $pam_user    = $ENV{PAM_USER}    // 'unknown';
my $pam_rhost   = $ENV{PAM_RHOST}   // 'unknown';
my $pam_service = $ENV{PAM_SERVICE} // 'unknown';

my $hostname = `hostname -s`;
chomp $hostname;

my $msg = "SSH Login Alert\nHost: $hostname\nUser: $pam_user\nFrom: $pam_rhost\nService: $pam_service";

log_msg("[ALERT] SSH login: user=$pam_user from=$pam_rhost");

system(
    '/usr/bin/curl',
    '--silent', '--output', '/dev/null',
    '-u', "$NTFY_USER:$NTFY_PASS",
    '-H', 'Title: SSH Login Detected',
    '-H', 'Priority: high',
    '-H', 'Tags: warning,key',
    '-d', $msg,
    $NTFY_URL,
);

sub log_msg {
    my ($text) = @_;
    my $ts = do {
        my @t = localtime;
        sprintf('%04d-%02d-%02d %02d:%02d:%02d',
            $t[5]+1900, $t[4]+1, $t[3], $t[2], $t[1], $t[0]);
    };
    open(my $fh, '>>', $LOG_FILE) or return;
    print $fh "$ts $text\n";
    close $fh;
}

exit 0;

Update $NTFY_URL with your domain and topic before installing.

Install It

sudo nano /usr/local/bin/ssh_alert.pl

Paste the script above, save, then:

sudo chmod 700 /usr/local/bin/ssh_alert.pl

Wire Into PAM

sudo nano /etc/pam.d/sshd

Add this line at the end of the file:

session optional pam_exec.so /usr/local/bin/ssh_alert.pl

No SSH restart needed: PAM reads its config on each new login.

Test It

Open a new SSH session from another terminal. You should get a push notification within seconds.

Part 2: Intrusion Detection

This one runs on a schedule rather than on an event. Every 15 minutes, a cron job checks seven indicators of compromise against a saved baseline. The first run establishes the baseline silently. Every run after that compares against it and alerts on any difference.

How auto-update works: after an alert fires, the script overwrites the baseline with the new state. A legitimate change (you added a cron job, you rotated your SSH key) won't re-alert on the next run. The tradeoff: a slow, gradual attack spread across multiple 15-minute windows only ever shows you the per-run diff, not the cumulative change. Worth knowing, not a reason to skip this.

The Script

Save as /root/detection/intrusion_check.sh:

#!/bin/bash
# intrusion_check.sh - Basic IOC detection with Ntfy alerting
# Run via cron every 15 minutes
#
# install:
#   cp intrusion_check.sh /root/detection/intrusion_check.sh
#   chmod 700 /root/detection/intrusion_check.sh
#
# add to crontab (sudo crontab -e):
#   */15 * * * * /root/detection/intrusion_check.sh

set -u

# --- CONFIG ---
BASE_DIR="/root/detection"
BASELINE_DIR="$BASE_DIR/baseline"
LOG_FILE="$BASE_DIR/logs/detect.log"
AUTH_FILE="$BASE_DIR/.auth"
NTFY_URL="https://ntfy.yourdomain.com/vps-alerts"

mkdir -p "$BASELINE_DIR"
mkdir -p "$BASE_DIR/logs"

# --- load ntfy credentials ---
if [ ! -f "$AUTH_FILE" ]; then
    echo "$(date '+%Y-%m-%d %H:%M:%S') [ERROR] missing $AUTH_FILE" >> "$LOG_FILE"
    exit 0
fi
NTFY_USER=$(sed -n '1p' "$AUTH_FILE")
NTFY_PASS=$(sed -n '2p' "$AUTH_FILE")
HOSTNAME=$(hostname -s)

# --- helper: send alert ---
send_alert() {
    local title="$1"
    local message="$2"
    echo "$(date '+%Y-%m-%d %H:%M:%S') [ALERT] $title - $message" >> "$LOG_FILE"
    curl --silent --output /dev/null \
        -u "${NTFY_USER}:${NTFY_PASS}" \
        -H "Title: $title" \
        -H "Priority: high" \
        -H "Tags: warning,skull" \
        -d "Host: $HOSTNAME
$message" \
        "$NTFY_URL"
}

# --- helper: compare and update baseline ---
# usage: check_baseline <name> <current_content_via_stdin>
check_baseline() {
    local name="$1"
    local baseline_file="$BASELINE_DIR/${name}.txt"
    local current_file
    current_file=$(mktemp)
    cat > "$current_file"

    if [ ! -f "$baseline_file" ]; then
        # first run - establish baseline, no alert
        mv "$current_file" "$baseline_file"
        echo "$(date '+%Y-%m-%d %H:%M:%S') [INIT] baseline created for $name" >> "$LOG_FILE"
        return
    fi

    if ! diff -q "$baseline_file" "$current_file" > /dev/null 2>&1; then
        local diff_output
        diff_output=$(diff "$baseline_file" "$current_file" | head -20)
        send_alert "Intrusion Check: $name changed" "$diff_output"
        mv "$current_file" "$baseline_file"
    else
        rm -f "$current_file"
    fi
}

# --- 1. SUID/SGID binaries ---
find / -xdev -type f \( -perm -4000 -o -perm -2000 \) 2>/dev/null | sort \
    | check_baseline "suid_files"

# --- 2. /etc/passwd hash ---
sha256sum /etc/passwd 2>/dev/null | awk '{print $1}' \
    | check_baseline "passwd_hash"

# --- 3. /etc/shadow hash ---
sha256sum /etc/shadow 2>/dev/null | awk '{print $1}' \
    | check_baseline "shadow_hash"

# --- 4. cron jobs (all users + system) ---
{
    for user in $(cut -f1 -d: /etc/passwd); do
        crontab -u "$user" -l 2>/dev/null | sed "s/^/$user: /"
    done
    cat /etc/cron.d/* 2>/dev/null
    cat /etc/crontab 2>/dev/null
} | sort | check_baseline "cron_jobs"

# --- 5. authorized_keys hashes (all users) ---
{
    for home in /root /home/*; do
        if [ -f "$home/.ssh/authorized_keys" ]; then
            sha256sum "$home/.ssh/authorized_keys"
        fi
    done
} 2>/dev/null | sort | check_baseline "authorized_keys_hash"

# --- 6. listening ports ---
ss -tulnH 2>/dev/null | awk '{print $1, $5}' | sort \
    | check_baseline "listening_ports"

# --- 7. world-writable files in system dirs ---
find /etc /usr /bin /sbin /lib /lib64 -xdev -type f -perm -0002 2>/dev/null | sort \
    | check_baseline "world_writable"

exit 0

Update NTFY_URL with your domain and topic before installing.

Install It

# update NTFY_URL in the script first with your domain + topic
sudo cp intrusion_check.sh /root/detection/intrusion_check.sh
sudo chmod 700 /root/detection/intrusion_check.sh

# first run establishes the baseline, no alerts
sudo /root/detection/intrusion_check.sh
sudo cat /root/detection/logs/detect.log

You should see seven [INIT] lines, one per check.

Add the Cron Job

sudo crontab -e

*/15 * * * * /root/detection/intrusion_check.sh

Test It

sudo touch /root/.ssh/authorized_keys
echo "ssh-rsa AAAAtest test@test" | sudo tee -a /root/.ssh/authorized_keys
sudo /root/detection/intrusion_check.sh

This should trigger an authorized_keys_hash changed alert on your phone. Remove the test line and run the script again to reset the baseline.

What You'll See

A normal SSH login produces a push notification within seconds of the session opening: host, user, source IP. Nothing more.

An intrusion check alert looks like this:

Intrusion Check: authorized_keys_hash changed
Host: your-vps
< a1b2c3d4...  /root/.ssh/authorized_keys
---
> e5f6g7h8...  /root/.ssh/authorized_keys

The hash changed. Something touched that file. If it wasn't you, you know within 15 minutes, not after the fact.

Useful Commands Reference

What's Next

Fail2ban blocks based on behavior. SSH alerts and intrusion detection tell you what happened. The one piece still missing: blocking traffic from entire regions before it ever reaches your SSH port, regardless of behavior. In the next post, we configure nftables to drop connections by country.