My Debian web server build

17 October 2024

I've kept a text file of the steps I follow to build a web server for most of my working life. The 2017 version was written for Debian 8 on a VPS and assumed Apache, Virtualmin, PHP 5 and a lot of things that have since gone away. The 2024 version is the same skeleton on Debian 12 (or Ubuntu LTS on the big cloud providers), with PHP 7.4 to 8.3 installed side by side and a firewall that drops by default. This is the current one, with notes where the older way still turns up on servers I'm asked to look after.

The examples use web1.example.com on 192.0.2.10, with the office on 203.0.113.10. Run everything as root unless I say otherwise.

Some minimal images, and a few providers' "hybrid" products, come without an SSH server. If you're on the provider's console:

apt-get install openssh-server
systemctl enable --now ssh

Before doing anything else, make the shell history useful. Timestamps on history have saved me more than once when working out what someone did to a server and when:

cat >> ~/.bashrc <<'EOF'
export HISTTIMEFORMAT="%d/%m/%y %T "
export HISTSIZE=10000
export HISTFILESIZE=10000
EOF
source ~/.bashrc

Set the time zone, bring everything up to date, and give the machine its proper name:

timedatectl set-timezone Europe/London
apt-get update && apt-get -y dist-upgrade
hostnamectl set-hostname web1

/etc/hosts needs the fully qualified name against the server's own address. Postfix, Virtualmin and plenty of other things read it, and a server that can't resolve its own name causes odd problems later:

127.0.0.1    localhost
192.0.2.10   web1.example.com web1

Reboot now, while there's nothing on the box to lose, so you know it comes back up on the new kernel with the new name:

shutdown -r now

(2017: dpkg-reconfigure tzdata, nano /etc/hostname and the hostname command. They still work.)

2. A user, SSH and updates

Cloud images usually give you a default user with sudo and no root password. On a VPS you often get root with a password. Either way, end up with a named admin user, key-only SSH and no password logins:

adduser admin
usermod -aG sudo admin
mkdir -p /home/admin/.ssh
cp ~/.ssh/authorized_keys /home/admin/.ssh/
chown -R admin:admin /home/admin/.ssh
chmod 700 /home/admin/.ssh && chmod 600 /home/admin/.ssh/authorized_keys

On a cloud image, that user is also what you'll log in to Webmin with later, since root has no password there. Give it a password with passwd admin for sudo and the panel; SSH will still only accept keys.

Then /etc/ssh/sshd_config.d/local.conf:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

sshd -t && systemctl reload ssh

Open a second session as admin before you close the first. If the second one can't get in, you still have the first.

Security updates should install themselves:

apt-get install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

Time sync: in 2017 I installed ntpdate and pointed it at the UK pool servers (0.uk.pool.ntp.org to 3.uk.pool.ntp.org). Debian now ships systemd-timesyncd, which is enough for a web server; check it's on with timedatectl and look for System clock synchronized: yes.

Some general tools I always want:

apt-get -y install zip unzip screen curl plocate git net-tools

3. Virtualmin, or plain Nginx

Most of my servers host several sites for different people, and for that I use Virtualmin GPL: it creates a Unix user, home directory, PHP-FPM pool, database and web server config per site, and handles certificates and backups. When a server runs one application, I skip it and configure Nginx by hand. Both are below.

With Virtualmin

Run the installer inside screen, so a dropped SSH session doesn't kill it halfway:

screen
curl -fsSLo virtualmin-install.sh https://software.virtualmin.com/gpl/scripts/virtualmin-install.sh
sh virtualmin-install.sh --bundle LEMP --minimal

--bundle LEMP installs Nginx rather than Apache, and --minimal leaves out the mail filtering, antivirus and other services a web-only server doesn't need. In 2017 there were no such options: you installed the full Apache stack, stopped Apache, installed webmin-virtualmin-nginx and webmin-virtualmin-nginx-ssl, and then turned features off one by one.

Log in at https://192.0.2.10:10000 and run the post-install wizard. Then, under System Settings, Features and Plugins, turn off everything you won't use. On a web-only box that's typically the DNS domain, mail, spam and virus filtering, Mailman, AWstats, DAV and protected directories. Do that before removing packages, or Virtualmin's configuration check fails because it expects them.

Remove the installer and its log afterwards; the log contains more about the box than you want lying around:

rm -f ~/virtualmin-install.sh ~/virtualmin-install.log

Without it

apt-get install nginx mariadb-server

Then one PHP-FPM pool and one Nginx server block per site, set up as in the next section.

Clearing out

Whichever route, remove services that aren't needed rather than just firewalling them. On a web-only server:

apt-get purge --auto-remove bind9 dovecot-core spamassassin clamav clamav-daemon clamav-freshclam mailman postgresql rpcbind

rpcbind is easy to miss; it only matters for NFS and listens on port 111 on every interface by default. Check what's left afterwards with ss (section 6).

4. Several PHP versions side by side

Sites move to new PHP versions at their own pace, so a shared server needs several installed at once. Ondřej Surý's packages make that easy: each version installs under its own name (php8.2-fpm, php7.4-fpm and so on) with its own config in /etc/php/<version>/.

On Ubuntu it's a PPA:

add-apt-repository ppa:ondrej/php
apt-get update

On Debian, add the repository and its signing key:

apt-get install -y lsb-release ca-certificates curl
curl -sSLo /usr/share/keyrings/deb.sury.org-php.gpg https://packages.sury.org/php/apt.gpg
echo "deb [signed-by=/usr/share/keyrings/deb.sury.org-php.gpg] https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
apt-get update

Then the same set of extensions for each version you need:

for v in 7.4 8.1 8.2 8.3; do
  apt-get -y install php$v-fpm php$v-cli php$v-mysql php$v-gd php$v-curl php$v-intl \
    php$v-mbstring php$v-zip php$v-xml php$v-soap php$v-bcmath
done

php-mcrypt is in all my older lists and is gone from PHP 7.2 onwards; don't go looking for it.

Choose which version the plain php command runs, for cron jobs and Composer:

update-alternatives --set php /usr/bin/php8.2

Pools

Each site gets its own FPM pool running as the site's own user, so one compromised site can't read another's files. Virtualmin creates these for you and lets you pick the version per site. By hand, /etc/php/8.2/fpm/pool.d/example.com.conf:

[example.com]
user = example
group = example
listen = /run/php/php8.2-fpm-example.sock
listen.owner = www-data
listen.group = www-data
pm = ondemand
pm.max_children = 10
php_admin_value[error_log] = /home/example/logs/php-error.log

And in the site's Nginx server block:

location ~ \.php$ {
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php8.2-fpm-example.sock;
}

Moving a site to another PHP version is a matter of moving the pool file to the other version's pool.d and changing the socket path.

Settings

My old lists edited php.ini for each version and then deleted the CLI and CGI copies and symlinked them to the Apache one, so every SAPI used the same settings. It works, but upgrades fight you over it. A cleaner way is one small file per version that phpenmod links into every SAPI. /etc/php/8.2/mods-available/local.ini:

; priority=99
expose_php = Off
max_execution_time = 300
max_input_time = 600
memory_limit = 512M
session.gc_probability = 1
opcache.enable = 1
opcache.memory_consumption = 256

phpenmod -v 8.2 local
systemctl reload php8.2-fpm

expose_php = Off stops PHP announcing its version in an X-Powered-By header. The time and memory limits are what the CMSs I host need; set your own.

session.gc_probability = 1 needs explaining. Debian turns PHP's own session clean-up off and runs a cron job instead, which only knows about the default session directory. Virtualmin gives each site its own session directory, which that cron job never sees, so old sessions pile up until the disk fills. Turning PHP's garbage collection back on fixes it.

If a site uses code encoded with ionCube, the loader comes from ionCube's website as a zip containing one .so per PHP version. Unpack it to /usr/local/ioncube and add zend_extension = /usr/local/ioncube/ioncube_loader_lin_8.2.so (matching the version) to that version's local.ini, before any other Zend extensions. If nothing needs it, don't install it.

5. Send-only Postfix

Web applications need to send mail: password resets, order confirmations, cron output. The server should never receive it. Postfix as a null client does that. In /etc/postfix/main.cf, replace the existing inet_interfaces line and add the rest:

inet_interfaces = loopback-only
inet_protocols = ipv4
mydestination =
local_transport = error:local delivery is disabled

loopback-only means Postfix only accepts mail from programs on the server itself. local_transport stops it delivering anything locally, so a mistake can't leave mail piling up in /var/mail. inet_protocols = ipv4 is there because many hosts have no reverse DNS on their IPv6 addresses, and big mail providers reject mail from addresses without it.

In /etc/postfix/master.cf, comment out the submission service if it's enabled; nobody should be authenticating to this server to send mail:

#submission inet n       -       y       -       -       smtpd

Cron and system mail goes to root@ the server's domain. Send it somewhere someone reads with /etc/postfix/virtual:

root    you@example.com

postconf -e 'virtual_alias_maps = hash:/etc/postfix/virtual'
postmap /etc/postfix/virtual

Sending straight to the internet from a VPS address often lands in spam, and some providers block port 25 outright. Then it's better to relay through your provider's relay or a transactional mail service:

relayhost = [smtp.example.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt

/etc/postfix/sasl_passwd:

[smtp.example.com]:587 your-username:your-password

chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
systemctl restart postfix

Whichever way it leaves, the sending domain needs SPF and DKIM records covering it, or receivers will treat it as forged.

6. Check what's listening

This is the step I'd keep if I could only keep one. After all of the above:

ss -tulpn

(In 2017 it was netstat -tulpn; that's in net-tools and still works.) On a Virtualmin web server, what you want to see is roughly:

Netid State  Local Address:Port  Process
tcp   LISTEN 127.0.0.1:3306      mariadbd
tcp   LISTEN 127.0.0.1:25        master
tcp   LISTEN 0.0.0.0:22          sshd
tcp   LISTEN 0.0.0.0:80          nginx
tcp   LISTEN 0.0.0.0:443         nginx
tcp   LISTEN 0.0.0.0:10000       miniserv.pl
tcp   LISTEN 0.0.0.0:20000       miniserv.pl

MySQL and Postfix on loopback only, the web server and SSH on everything, and Webmin (10000) and Usermin (20000). Anything else listening on 0.0.0.0 needs a reason.

7. Saying less about yourself

Version numbers in banners and headers help attackers pick an exploit. Turn them off.

Nginx, in /etc/nginx/nginx.conf:

server_tokens off;

Apache, if you're on it, in /etc/apache2/conf-available/security.conf:

ServerTokens Prod
ServerSignature Off
TraceEnable Off

On Apache I also turn off the modules nothing uses and make sure the ones every CMS expects are on:

a2dismod -f autoindex dav dav_fs
a2enmod rewrite headers expires
systemctl restart apache2

If ProFTPD is installed (older Virtualmin installs always had it), add ServerIdent off under ServerName in /etc/proftpd/proftpd.conf. Better still, don't run FTP at all; SFTP comes with SSH.

Admin tools under a site, such as phpMyAdmin, should only answer the office. Nginx:

location /dbadmin {
    allow 203.0.113.10;
    deny all;
}

Apache 2.4:

<Location /dbadmin>
    Require ip 203.0.113.10
</Location>

The Order deny,allow / Allow from syntax in older configs is Apache 2.2. It only works on 2.4 through mod_access_compat, and mixing the two styles gives results you won't expect.

8. Firewall: drop by default

Every version of my list has had an iptables ruleset, but the early ones had the INPUT policy set to ACCEPT and sent a list of admin ports to a chain that only allowed the office. That protects the ports on the list and nothing else: install Elasticsearch or Redis on a public interface next month and it's open to the world. The current version drops everything not explicitly allowed.

On Debian 12 I write it in nftables directly. /etc/nftables.conf:

#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    set admin_v4 {
        type ipv4_addr
        flags interval
        elements = { 203.0.113.10, 198.51.100.0/28 }
    }

    chain input {
        type filter hook input priority 0; policy drop;
        iif lo accept
        ct state established,related accept
        ct state invalid drop
        ip protocol icmp icmp type { echo-request, destination-unreachable, time-exceeded } accept
        meta l4proto ipv6-icmp accept
        tcp dport { 80, 443 } accept
        ip saddr @admin_v4 tcp dport { 22, 10000, 20000 } accept
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

nft -c -f /etc/nftables.conf
systemctl enable --now nftables
nft list ruleset

nft -c checks the file without loading it. The inet table covers IPv4 and IPv6 together, which iptables never did; plenty of servers with a careful iptables ruleset are wide open over IPv6. The admin set holds the office and a VPN range; add a v6 set the same way if you connect over IPv6. Keep the provider's console login working, because one day you'll lock yourself out.

The same thing in iptables, for older servers, goes in /etc/iptables/rules.v4 with iptables-persistent installed:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:ADMIN - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22,10000,20000 -j ADMIN
-A ADMIN -s 203.0.113.10/32 -j ACCEPT
-A ADMIN -s 198.51.100.0/28 -j ACCEPT
-A ADMIN -j DROP
COMMIT

Load it with iptables-apply /etc/iptables/rules.v4, which rolls back if you don't confirm in time. If you have an iptables file already, iptables-restore-translate -f rules.v4 produces nftables syntax as a starting point.

Two notes. Newer Virtualmin installers may set up firewalld for you; if so, use that (zones and sources do the same job) or remove it, but don't run two firewall managers. And Virtualmin installs fail2ban, which adds its own rules on top; that's fine with either.

9. Certificates

With Virtualmin, request a Let's Encrypt certificate for each site from its SSL settings page. It renews them itself, and the same page can copy a site's certificate to Webmin, Usermin and Postfix so the admin panel stops showing a browser warning.

Without it:

apt-get install certbot python3-certbot-nginx
certbot --nginx -d example.com -d www.example.com
certbot renew --dry-run

Debian's package installs a systemd timer for renewal; systemctl list-timers | grep certbot shows it. The dry run proves renewal will work before the first one is due in two months.

Back in 2017 I terminated TLS in Pound in front of Varnish, and Pound wanted the key, certificate and chain in a single file (cat ssl.key ssl.cert ssl.ca > ssl.pem). If you meet that setup, that file has the private key in it, so it needs the same permissions as the key.

10. The database

On current Debian and Ubuntu, MariaDB's and MySQL's root account authenticates by Unix socket: mysql as the system root user gets straight in with no password, and nobody else can log in as root at all. Leave it that way. Older lists, including mine, had a step to switch root back to password authentication with an UPDATE mysql.user statement; that's both less secure and, on MySQL 8, broken, because PASSWORD() no longer exists.

For your own access, create a named admin user:

CREATE USER 'dbadmin'@'localhost' IDENTIFIED BY 'CHANGE_ME';
GRANT ALL PRIVILEGES ON *.* TO 'dbadmin'@'localhost' WITH GRANT OPTION;

And put its credentials in ~/.my.cnf with chmod 600, rather than typing them on the command line where they land in the history:

[client]
user=dbadmin
password=CHANGE_ME

mytop and the MySQL clients read that file. MySQLTuner is still worth running after a week of real traffic:

apt-get install mysqltuner mytop
mysqltuner

For Magento sites, Redis goes on the same box (apt-get install redis-server). Debian's package binds it to localhost only. Check that it does, because Redis on a public interface with no password is one of the most common ways servers get taken over.

11. Backups

A backup you haven't restored isn't a backup. What I do:

Virtualmin's scheduled backups (Backup and Restore, Scheduled Backups) of every site: files, databases and Virtualmin's own settings. Daily, to a local directory and to an S3 bucket.
Without Virtualmin, a nightly cron job that runs mysqldump --single-transaction for each database and tar for each site, then copies the results off the server.
The off-site copy goes to a bucket used for nothing else, with versioning on and a lifecycle rule to expire old versions. The credentials on the server can only write to that bucket. On AWS, use an instance role instead of keys. Elsewhere, an access key limited to that bucket, stored with chmod 600.
Every few months, restore one site onto a scratch server and check it loads.

In 2017 the off-site copy went up with s3cmd from the s3tools APT repository. These days s3cmd is in Debian's own archive, and the AWS CLI does the same job.

12. Finishing

updatedb
shutdown -r now

updatedb builds the locate index so locate works straight away. The final reboot proves everything starts on its own: the firewall loads, PHP-FPM pools come up, the sites answer. Better to find out now than at 3am after a kernel update.

My 2017 list ended with history -c. I put it there because I'd typed passwords on the command line during the build. The better fix is not to: use ~/.my.cnf, read -s, or a password prompt, and leave the history alone. It's the record of what was done to the server.

1. First login