---
title: "OpenSearch in Docker on its own data disk"
date: "2025-04-30"
path: "/227-opensearch-in-docker-on-its-own-data-disk.html"
section: "/ramblings"
description: "Running a single OpenSearch node in Docker with its data on a separate block storage volume, and changing the admin password once it is up."
---

I wanted a single OpenSearch node on a small cloud VM for testing, with the index data on a separate block storage volume. That way the VM can be rebuilt or resized without touching the data, and the volume can be grown on its own when it fills up. This is OpenSearch 2.19 on an Ubuntu VM with Docker installed.

## Format and mount the volume

Attach the volume in the provider's control panel first. Then find it by its stable name under `/dev/disk/by-id/`, not as `/dev/sdc` or similar, which can change between boots:

```bash
ls -l /dev/disk/by-id/
```

On Linode a volume called `storage` shows up as `scsi-0Linode_Volume_storage`; other providers use their own naming. Format it (only the first time: this wipes it), make a mount point and mount it:

```bash
mkfs.ext4 /dev/disk/by-id/scsi-0Linode_Volume_storage
mkdir /mnt/storage
mount /dev/disk/by-id/scsi-0Linode_Volume_storage /mnt/storage
```

Add it to `/etc/fstab` so it comes back after a reboot:

```text
/dev/disk/by-id/scsi-0Linode_Volume_storage /mnt/storage ext4 defaults,noatime,nofail 0 2
```

- `noatime` stops the kernel writing an access time every time a file is read, which on a search index is constant pointless writing.
- `nofail` lets the VM finish booting if the volume is missing or detached. Without it, a detached volume leaves you at an emergency shell with no network.

Test the fstab line before you rely on it:

```bash
umount /mnt/storage
mount -a
df -h /mnt/storage
```

## Prepare the host

OpenSearch maps a lot of files into memory and will not start with the kernel default limit:

```bash
sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' > /etc/sysctl.d/99-opensearch.conf
```

The process inside the container runs as UID 1000, so the data directory on the volume has to belong to that UID:

```bash
mkdir -p /mnt/storage/opensearch
chown -R 1000:1000 /mnt/storage/opensearch
```

## Run it

```bash
docker run -d --name opensearch \
  --restart unless-stopped \
  -p 127.0.0.1:9200:9200 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=CHANGE_ME" \
  -v /mnt/storage/opensearch:/usr/share/opensearch/data \
  opensearchproject/opensearch:2.19.1
```

What matters here:

- `-p 127.0.0.1:9200:9200` publishes the port on localhost only. Docker writes its own iptables rules, so a port published on all interfaces is reachable from the internet even if ufw says it is blocked. Put a reverse proxy in front, or use an SSH tunnel, if you need to reach it from elsewhere.
- `discovery.type=single-node` skips cluster discovery and the bootstrap checks that assume several nodes.
- Since 2.12 the image will not start without `OPENSEARCH_INITIAL_ADMIN_PASSWORD`, and it must be a strong password (it is checked). It only takes effect the first time, when the security index is created.
- The heap should be about half the VM's memory, with the same minimum and maximum.
- Pin the image version. `latest` will move you to a new major version the next time the container is recreated.

Check it is up, from the VM:

```bash
curl -sk -u admin:CHANGE_ME https://localhost:9200
curl -sk -u admin:CHANGE_ME "https://localhost:9200/_cat/health?v"
```

`-k` is needed because the image comes with demo TLS certificates. They are fine for testing; replace them with your own before anything real goes in.

## Changing the admin password later

Changing the environment variable after the first start does nothing: the users now live in the security index on the data volume. To change the admin password, make a hash and load it into that index.

Generate the hash inside the container:

```bash
docker exec -it opensearch \
  /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p 'new-password'
```

Edit `config/opensearch-security/internal_users.yml` inside the container and replace the `hash:` value on the `admin` user with the new one. Change only that line. The file must keep its `_meta` block and the other built-in users; a file with just an admin user in it will be rejected or will remove the others.

```yaml
_meta:
  type: "internalusers"
  config_version: 2

admin:
  hash: "<bcrypt-hash-from-hash.sh>"
  reserved: true
  backend_roles:
  - "admin"
  description: "Admin user"
```

Then push the security config into the index with `securityadmin.sh`, authenticating with the admin certificate that came with the demo setup:

```bash
docker exec -it opensearch bash -c '
  cd /usr/share/opensearch &&
  plugins/opensearch-security/tools/securityadmin.sh \
    -cd config/opensearch-security/ -icl -nhnv \
    -cacert config/root-ca.pem \
    -cert config/kirk.pem -key config/kirk-key.pem'
```

Changes made inside the container are lost if it is recreated, but the security index on the volume keeps the new password. If you would rather the YAML files survived as well, bind-mount your own copy of `internal_users.yml` from the host.

## Growing the volume

When the data disk fills, resize the volume in the provider's panel (some providers want it detached or the VM restarted for this; check their documentation), then grow the filesystem. ext4 can be grown while mounted:

```bash
resize2fs /dev/disk/by-id/scsi-0Linode_Volume_storage
```

If the volume did have to be detached, stop the container first so OpenSearch is not writing to a disk that disappears.