NFS and Samba mounts between Linux servers

2 November 2016

The job: a CMS ran on two Debian web servers, and the uploaded files directory had to be the same on both. Anything an editor uploaded on one box had to be visible on the other straight away. The simplest answer on a private network is to keep the files on one server and mount them on the other over NFS.

In the examples the file server is 192.0.2.10, the second web server is 192.0.2.20, and the site lives in /home/example/public_html owned by the user example. On a real setup these should be addresses on the private interface between the two machines, not the public ones.

The file server

Install the server side:

apt-get install nfs-kernel-server

I don't export the site directory directly. NFSv4 likes a single export root, so I make /export the root and bind-mount the real directory into it. That keeps the export paths short and means nothing outside /export is ever reachable.

mkdir -p /export/uploads
mount --bind /home/example/public_html/application/files /export/uploads

Make the bind mount survive a reboot in /etc/fstab:

/home/example/public_html/application/files  /export/uploads  none  bind  0  0

Then the exports, in /etc/exports:

/export          192.0.2.20(rw,sync,fsid=0,no_subtree_check,crossmnt)
/export/uploads  192.0.2.20(rw,sync,nohide,no_subtree_check)

What the options mean:

fsid=0 marks /export as the NFSv4 pseudo-root. Clients then mount paths relative to it, so /export/uploads is mounted as 192.0.2.10:/uploads.
nohide and crossmnt let the client see the bind-mounted directory under the root instead of an empty folder.
sync makes the server commit writes to disk before replying. async is faster, and I tried it, but a crash can lose writes the client thinks are saved. For files people upload once and expect to stay, sync is the right default.
no_subtree_check turns off a check that causes more problems than it solves when files get renamed.
Root squashing is on by default, which maps root on the client to nobody on the server. Leave it on. no_root_squash lets root on the web server do anything to the files, which you don't want on a box that runs PHP.

Load the exports and check them:

exportfs -rav
service nfs-kernel-server restart
showmount -e localhost

Ownership: the part that wastes the afternoon

NFS passes numeric user and group ids around. If example is uid 1005 on the file server and uid 1003 on the web server, the web server's PHP processes can't write to the share, and ls -l shows the wrong owner or nobody.

Check both ends:

id example

If the numbers differ, fix one side so they match (usermod -u and groupmod -g, then chown the user's files), or create the user with an explicit id on the second server in the first place.

NFSv4 also maps names through idmapd. Both machines need the same domain in /etc/idmapd.conf:

[General]
Domain = example.com

Restart nfs-common on the client and nfs-kernel-server on the server after changing it. If files still show as nobody:nogroup on the client, the domains don't match.

I went round the loop of chmod 777 on the export directory before getting this right. Don't. Once the ids line up, normal ownership and 755/644 permissions work, and if you use Virtualmin you can put a site's permissions back with:

virtualmin fix-domain-permissions --domain example.com

Locking it down

NFSv4 only needs TCP 2049. Allow that from the web server's address and nothing else. NFSv3 and showmount also use rpcbind on 111 and mountd on a random port, which is one more reason to stick to v4.

The TCP wrappers files still apply to rpcbind and mountd on this era of Debian, so as a second layer:

# /etc/hosts.deny
rpcbind mountd : ALL

# /etc/hosts.allow
rpcbind mountd : 192.0.2.20 127.0.0.1

The web server

apt-get install nfs-common
mkdir -p /home/example/public_html/application/files
mount -t nfs4 192.0.2.10:/uploads /home/example/public_html/application/files

Try creating and deleting a file as the site user, then make it permanent in /etc/fstab:

192.0.2.10:/uploads  /home/example/public_html/application/files  nfs4  _netdev,noatime,bg,proto=tcp  0  0

_netdev stops the system trying to mount it before the network is up, and bg means a missing file server makes the boot carry on and retry in the background instead of hanging. I first had a long attribute cache (actimeo=1800) to cut down chatter, but that means a new upload can take up to half an hour to appear on the other server. For a shared uploads directory the defaults are the right trade.

To unmount:

umount /home/example/public_html/application/files

When it doesn't work

These are the commands I ended up running over and over:

# is anything listening on 111 and 2049?
netstat -tulpn | grep -E ':(111|2049) '

# which RPC services are registered?
rpcinfo -p

# does the NFS service answer over TCP?
rpcinfo -t 192.0.2.10 nfs

# what does the server export? (needs rpcbind and mountd)
showmount -e 192.0.2.10

If rpcinfo -p fails on the server, rpcbind isn't running (service rpcbind restart). If it works locally but not from the client, it's the firewall or hosts.allow. If the mount works but writes fail, go back to the ownership section.

When the other end is Samba

If the share lives on a Windows box or a Samba server instead, the client side is CIFS:

apt-get install cifs-utils

Keep the credentials out of fstab in a file only root can read, /root/.smb-example:

username=example
password=your-password

chmod 600 /root/.smb-example

And the fstab line:

//192.0.2.30/uploads  /mnt/uploads  cifs  credentials=/root/.smb-example,uid=example,gid=example,file_mode=0644,dir_mode=0755,vers=3.0,_netdev  0  0

CIFS doesn't carry Unix ownership, so uid and gid decide who owns everything on the mounted side. Set them to the site user and the web server can write to it.