Recording IQ with SigMF metadata

After a few years of SDR work I had a disk full of files called things like capture_433M_2Msps.bin. The file name was the only record of the centre frequency and sample rate, and nothing at all recorded the sample format, the gain, the hardware or what was being transmitted at the time. Opening an old one meant guessing whether it was 8-bit unsigned from an RTL-SDR, 16-bit integers from a USRP or 32-bit floats from GNU Radio.

SigMF, the Signal Metadata Format, fixes that with a JSON sidecar file. It's an open specification, GNU Radio and a growing number of tools understand it, and it's simple enough to write by hand.

The format

A SigMF recording is two files with the same base name:

• name.sigmf-data: the raw samples, exactly as they came off the radio. No header, no conversion.
• name.sigmf-meta: JSON describing them.

The two can also be bundled into a single name.sigmf tar archive for sharing.

The metadata has three sections:

• global: facts about the whole file. The required ones are core:datatype and core:version; core:sample_rate is optional in the spec but you always want it. Hardware, author, description and licence go here too.
• captures: a list of segments, each starting at a sample index. Each says what the radio was tuned to (core:frequency) and when it started (core:datetime). A recording made in one go has one capture; if you retuned mid-recording you'd add another at the sample where it changed.
• annotations: optional notes on ranges of samples, each with a start, a length and, if you like, frequency edges and a comment. Useful for marking where a test transmission is in a long file.

Keys are namespaced. Everything in the standard is core:, and extensions add their own namespaces, so tools can ignore anything they don't understand.

Datatypes

core:datatype is the bit that saves the guessing. It's built from:

• c for complex (interleaved I and Q) or r for real,
• f, i or u for float, signed or unsigned integer,
• the bit width,
• _le or _be for byte order (not needed for 8-bit types).

So the common ones are:

Because the data file is untouched, you can describe an existing raw capture without converting it, just by writing the right metadata next to it.

Writing a recording in Python

The sigmf package on PyPI is the reference library. It computes a SHA-512 of the data file, validates the metadata against the schema and writes the JSON:

python3 -m pip install sigmf numpy

This example generates a test tone in place of real radio samples, so it runs anywhere. In practice samples comes from your SDR's API, or the data file already exists from a command-line capture.

import numpy as np
import sigmf
from sigmf.utils import get_data_type_str, get_sigmf_iso8601_datetime_now

sample_rate = 2_000_000
centre_freq = 433_920_000
n = sample_rate // 10

t = np.arange(n) / sample_rate
samples = (0.5 * np.exp(2j * np.pi * 100_000 * t)).astype(np.complex64)
samples.tofile("tone.sigmf-data")

meta = sigmf.SigMFFile(
    data_file="tone.sigmf-data",
    global_info={
        sigmf.DATATYPE_KEY: get_data_type_str(samples),
        sigmf.SAMPLE_RATE_KEY: sample_rate,
        sigmf.HW_KEY: "RTL-SDR v3",
        sigmf.DESCRIPTION_KEY: "100 kHz test tone from a signal generator",
    },
)
meta.add_capture(0, metadata={
    sigmf.FREQUENCY_KEY: centre_freq,
    sigmf.DATETIME_KEY: get_sigmf_iso8601_datetime_now(),
})
meta.add_annotation(0, n, metadata={
    sigmf.FREQ_LOWER_EDGE_KEY: centre_freq + 95_000,
    sigmf.FREQ_UPPER_EDGE_KEY: centre_freq + 105_000,
    sigmf.COMMENT_KEY: "test tone, -40 dBm at the generator",
})
meta.tofile("tone.sigmf-meta")

get_data_type_str() works the datatype out from the NumPy array (cf32_le here), which avoids getting it wrong by hand. tofile() validates before writing, so a missing required field fails here and not months later when someone tries to open the file.

The resulting tone.sigmf-meta:

{
    "global": {
        "core:datatype": "cf32_le",
        "core:description": "100 kHz test tone from a signal generator",
        "core:hw": "RTL-SDR v3",
        "core:num_channels": 1,
        "core:offset": 0,
        "core:sample_rate": 2000000,
        "core:sha512": "0d26...2c03",
        "core:version": "1.2.6"
    },
    "captures": [
        {
            "core:datetime": "2022-12-30T14:02:11.418263Z",
            "core:frequency": 433920000,
            "core:sample_start": 0
        }
    ],
    "annotations": [
        {
            "core:comment": "test tone, -40 dBm at the generator",
            "core:freq_lower_edge": 434015000,
            "core:freq_upper_edge": 434025000,
            "core:sample_count": 200000,
            "core:sample_start": 0
        }
    ]
}

I've shortened the hash. The core:version is whichever version of the spec your installed library writes.

Reading it back

import sigmf

rec = sigmf.fromfile("tone")
print(rec.sample_rate)
print(rec.get_captures()[0][sigmf.FREQUENCY_KEY])

samples = rec.read_samples()
first = rec[0:1024]

fromfile finds the .sigmf-meta and .sigmf-data from the base name, or opens a .sigmf archive. read_samples() returns a complex NumPy array whatever the stored datatype, so code that reads cu8 from an RTL-SDR and ci16_le from a USRP looks the same. Slicing reads just that range from disk, which matters when the file is several gigabytes.

To produce a single file for sharing, write the archive instead:

meta.tofile("tone.sigmf")

What to watch for

• Write the metadata at the start of a capture, not afterwards from memory. The datetime and frequency should come from the same code that tuned the radio.
• Record the gain and the antenna in the description or an extension field. They're the first things you'll want when comparing two recordings.
• Interleaved 8-bit RTL-SDR samples are unsigned and centred around 128, not zero. Declare them as cu8 and let the library convert them, rather than converting yourself and forgetting the offset.
• The library hashes the whole data file when it writes the metadata. On a multi-gigabyte capture that takes a noticeable time, so do it after the capture has finished, not in the receive loop.

Today: older releases of the library kept the key constants on the class (SigMFFile.DATATYPE_KEY and so on). They still work, with deprecation warnings, but new code should use the module-level names as above.