OpenSearch in Docker on its own data disk

30 April 2025

I wanted a single OpenSearch node on a small cloud VM for testing, with the index data on a separate block storage volume. That way the VM can be rebuilt or resized without touching the data, and the volume can be grown on its own when it fills up. This is OpenSearch 2.19 on an Ubuntu VM with Docker installed.

Format and mount the volume

Attach the volume in the provider's control panel first. Then find it by its stable name under /dev/disk/by-id/, not as /dev/sdc or similar, which can change between boots:

ls -l /dev/disk/by-id/

On Linode a volume called storage shows up as scsi-0Linode_Volume_storage; other providers use their own naming. Format it (only the first time: this wipes it), make a mount point and mount it:

mkfs.ext4 /dev/disk/by-id/scsi-0Linode_Volume_storage
mkdir /mnt/storage
mount /dev/disk/by-id/scsi-0Linode_Volume_storage /mnt/storage

Add it to /etc/fstab so it comes back after a reboot:

/dev/disk/by-id/scsi-0Linode_Volume_storage /mnt/storage ext4 defaults,noatime,nofail 0 2

noatime stops the kernel writing an access time every time a file is read, which on a search index is constant pointless writing.
nofail lets the VM finish booting if the volume is missing or detached. Without it, a detached volume leaves you at an emergency shell with no network.

Test the fstab line before you rely on it:

umount /mnt/storage
mount -a
df -h /mnt/storage

Prepare the host

OpenSearch maps a lot of files into memory and will not start with the kernel default limit:

sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' > /etc/sysctl.d/99-opensearch.conf

The process inside the container runs as UID 1000, so the data directory on the volume has to belong to that UID:

mkdir -p /mnt/storage/opensearch
chown -R 1000:1000 /mnt/storage/opensearch

Run it

docker run -d --name opensearch \
  --restart unless-stopped \
  -p 127.0.0.1:9200:9200 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=CHANGE_ME" \
  -v /mnt/storage/opensearch:/usr/share/opensearch/data \
  opensearchproject/opensearch:2.19.1

What matters here:

-p 127.0.0.1:9200:9200 publishes the port on localhost only. Docker writes its own iptables rules, so a port published on all interfaces is reachable from the internet even if ufw says it is blocked. Put a reverse proxy in front, or use an SSH tunnel, if you need to reach it from elsewhere.
discovery.type=single-node skips cluster discovery and the bootstrap checks that assume several nodes.
Since 2.12 the image will not start without OPENSEARCH_INITIAL_ADMIN_PASSWORD, and it must be a strong password (it is checked). It only takes effect the first time, when the security index is created.
The heap should be about half the VM's memory, with the same minimum and maximum.
Pin the image version. latest will move you to a new major version the next time the container is recreated.

Check it is up, from the VM:

curl -sk -u admin:CHANGE_ME https://localhost:9200
curl -sk -u admin:CHANGE_ME "https://localhost:9200/_cat/health?v"

-k is needed because the image comes with demo TLS certificates. They are fine for testing; replace them with your own before anything real goes in.

Changing the admin password later

Changing the environment variable after the first start does nothing: the users now live in the security index on the data volume. To change the admin password, make a hash and load it into that index.

Generate the hash inside the container:

docker exec -it opensearch \
  /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p 'new-password'

Edit config/opensearch-security/internal_users.yml inside the container and replace the hash: value on the admin user with the new one. Change only that line. The file must keep its _meta block and the other built-in users; a file with just an admin user in it will be rejected or will remove the others.

_meta:
  type: "internalusers"
  config_version: 2

admin:
  hash: "<bcrypt-hash-from-hash.sh>"
  reserved: true
  backend_roles:
  - "admin"
  description: "Admin user"

Then push the security config into the index with securityadmin.sh, authenticating with the admin certificate that came with the demo setup:

docker exec -it opensearch bash -c '
  cd /usr/share/opensearch &&
  plugins/opensearch-security/tools/securityadmin.sh \
    -cd config/opensearch-security/ -icl -nhnv \
    -cacert config/root-ca.pem \
    -cert config/kirk.pem -key config/kirk-key.pem'

Changes made inside the container are lost if it is recreated, but the security index on the volume keeps the new password. If you would rather the YAML files survived as well, bind-mount your own copy of internal_users.yml from the host.

Growing the volume

When the data disk fills, resize the volume in the provider's panel (some providers want it detached or the VM restarted for this; check their documentation), then grow the filesystem. ext4 can be grown while mounted:

resize2fs /dev/disk/by-id/scsi-0Linode_Volume_storage

If the volume did have to be detached, stop the container first so OpenSearch is not writing to a disk that disappears.