/docs · General · Practical

Audit a dataset of conversations: where the audio really comes from

Measure a folder of call, voice-agent or text-to-speech recordings for a lossy source, upsampling, splices, mono in disguise, clipping and bit depth, 20 files at a time, and know when the audio is uploaded and when it is not.

Who this is for

You train, fine-tune or evaluate on conversational audio: call recordings, voice-agent logs, synthetic speech, a corpus you scraped or bought. Before you trust the labels, the scores or a model built on it, you want to know what the files really are. A file labelled 16 kHz can be 8 kHz telephone audio stretched wider. A WAV can be a decoded MP3. A stereo file can be one channel twice. A take can be clipped, and two sources can be joined inside one file. None of that is in the header.

It is for teams that decide for themselves what to do with a finding. It does not judge labels or transcripts, it does not read the words, identify speakers or tell a synthetic voice from a real one. See what AudioLab does not do.

What you get

A script, audit_dataset.py, that measures every audio file in a folder, 20 at a time, and writes one row per file to a CSV.
A summary of what is worth a look, with the file names, and the bandwidth for each sample rate in the folder.
Every finding comes from one analysis, the signal analysis (POST /v1/signallab), and every number it reads there is validated: the codec wall (lossySource), the original sample rate behind a resampler (upsampledFromHz), joins between sources (bandwidthSplices), the channels and how alike they are (channels, monophonicProbability), clipping (hasClipping, clippingRegions), the bits in use (effectiveBitDepth) and the bandwidth (bandwidthHz).

Before you start

Know when the audio is uploaded. The script uploads it: every file goes to the AudioLab API, hosted in the EU, and only numbers come back. The workspace is the other way: it runs the engine in your browser and uploads nothing. Its batch gives you the ten numbers of the comparison guide over many files, by hand, with the bandwidth and the clipped samples; it does not name a codec wall, a resampling, a splice or a channel layout. If your recordings must not leave your own systems, use the workspace and accept that it covers less.
Know what the recordings contain. Recordings of calls usually contain personal data. Sending them to the API makes us your processor: read the data processing addendum and /privacy before you run the script on real recordings, and start with files you are free to share.
Get a key. It is free and needs no card: sign in on /account and generate one. The script needs it, because it asks first what your plan has left (GET /v1/usage, which is free and does not count as a call).
Know what it costs. Each file is one call, and its size counts against your data for the month, so on the free tier (1,000 calls a month, 2 GB a month and 60 requests a minute) you audit a sample, not a corpus of ten thousand. A local file takes two requests: one to upload it and one to measure it. The script paces itself to the requests a minute of your plan, so 300 files at 60 a minute take at least ten minutes. Start with --sample 300: a random sample, the same one every time, is enough to see whether there is a problem.
Python 3.9 or newer, and the package:

pip install audiolabtools

The script

"""Audit a folder of audio files for where the audio really comes from.

One row per file in a CSV, and a summary of what is worth a look.

    python audit_dataset.py audio/ audit.csv               every file in audio/, 20 at a time
    python audit_dataset.py audio/ audit.csv --sample 300  a random sample of 300, the same
                                                           300 every time: the cheap first look

https://audiolab.tools/docs/start/audit-a-conversation-dataset
Every file is uploaded to the AudioLab API (https://audiolab.tools/privacy) and counts as one
call, so this needs a key (AUDIOLAB_KEY). A file that is already in the CSV is skipped, so a
run that stopped can be started again.
Exit code: 0 when every file was measured, 2 when something could not be (the CSV says what).
"""

from __future__ import annotations

import argparse
import csv
import random
import statistics
import sys
import time
from pathlib import Path

from audiolabtools import AudioLab, AudioLabError

AUDIO = {".wav", ".mp3", ".flac", ".m4a", ".ogg", ".opus", ".aac", ".aif", ".aiff"}
FIELDS = [
    "sampleRate", "channels", "bandwidthHz", "upsampledFromHz", "lossySource",
    "bandwidthSplices", "effectiveBitDepth", "monophonicProbability",
    "hasClipping", "clippingRegions",
]
COLUMNS = ["file", "findings", *FIELDS, "error"]
BATCH = 20  # the most the API takes in one call
# On a stereo file a monophonicProbability at or above this counts as the same signal on both
# sides. That is our choice: the number says only that near 1 means mono in disguise.
SAME_BOTH_SIDES = 0.99
KINDS = [
    "upsampled", "lossy source", "source unclear", "bandwidth splices",
    "stereo, same both sides", "clipping", "fewer than 16 bits",
]


def findings(d):
    """What is worth a look in one file, as 'kind (detail)'.

    Only what the engine itself says. The limits that are ours: SAME_BOTH_SIDES and 16 bits.
    """
    found = []
    if d.get("upsampledFromHz") is not None:
        found.append(f"upsampled (from {d['upsampledFromHz']:g} Hz)")
    if str(d.get("lossySource")).startswith("lossy"):
        found.append(f"lossy source ({d['lossySource'][6:]})")
    elif d.get("lossySource") == "unknown":
        found.append("source unclear")
    if d.get("bandwidthSplices"):
        found.append(f"bandwidth splices ({d['bandwidthSplices']})")
    if d.get("channels") == 2 and (d.get("monophonicProbability") or 0) >= SAME_BOTH_SIDES:
        found.append("stereo, same both sides")
    if d.get("hasClipping"):
        regions = d.get("clippingRegions") or 0
        found.append(
            f"clipping ({regions} region{'' if regions == 1 else 's'})"
            if regions else "clipping (a short run at the peak)"
        )
    if d.get("effectiveBitDepth") is not None and d["effectiveBitDepth"] < 16:
        found.append(f"fewer than 16 bits ({d['effectiveBitDepth']})")
    return found


def write_csv(path, rows):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=COLUMNS)
        writer.writeheader()
        writer.writerows(rows.values())


def summary(rows, out):
    measured = [r for r in rows.values() if not r["error"]]
    plural = "" if len(measured) == 1 else "s"
    out(f"\n{len(measured)} file{plural} measured, {len(rows) - len(measured)} failed\n")
    out(f"{'what':25}{'files':>5}")
    for kind in KINDS:
        hit = [
            r["file"] for r in measured
            if any(x == kind or x.startswith(kind + " (") for x in r["findings"].split("; "))
        ]
        names = ", ".join(hit[:5]) + (f" and {len(hit) - 5} more" if len(hit) > 5 else "")
        out(f"{kind:25}{len(hit):>5}   {names}".rstrip())
    out("\nbandwidth by sample rate")
    for rate in sorted({int(float(r["sampleRate"])) for r in measured}):
        widths = sorted(float(r["bandwidthHz"]) for r in measured
                        if int(float(r["sampleRate"])) == rate)
        if len(widths) == 1:
            out(f"  {rate} Hz    1 file   {widths[0]:.0f} Hz")
        else:
            median = statistics.median(widths)
            out(f"  {rate} Hz  {len(widths):>3} files  {widths[0]:.0f} to {widths[-1]:.0f} Hz, "
                f"median {median:.0f}")


def main(argv, client=None, out=print, wait=time.sleep):
    ask = argparse.ArgumentParser(description="Audit a folder of audio files.")
    ask.add_argument("folder")
    ask.add_argument("csv")
    ask.add_argument("--sample", type=int, help="measure a random sample of this many files")
    args = ask.parse_args(argv[1:])
    folder, target = Path(args.folder), Path(args.csv)

    files = sorted(p for p in folder.rglob("*") if p.suffix.lower() in AUDIO)
    if args.sample and args.sample < len(files):
        files = sorted(random.Random(1).sample(files, args.sample))
    rows = {}
    if target.exists():
        with open(target, newline="", encoding="utf-8") as f:
            rows = {r["file"]: r for r in csv.DictReader(f)}

    def name(p):
        return p.relative_to(folder).as_posix()

    todo = [p for p in files if name(p) not in rows or rows[name(p)]["error"]]

    client = client or AudioLab()  # reads AUDIOLAB_KEY
    try:
        usage = client.usage()  # free, and it says what this run may cost before it costs it
        size = sum(p.stat().st_size for p in todo)
        calls_left, bytes_left = usage["calls"]["remaining"], usage["data"]["remainingBytes"]
        left = "no limit" if calls_left is None else f"{calls_left} calls"
        out(f"{len(todo)} files to measure ({size / 1e6:.0f} MB). Plan {usage['plan']}: "
            f"{left} left this month, {usage['requestsPerMinute']} requests a minute.")
        if (calls_left is not None and len(todo) > calls_left) or (
            bytes_left is not None and size > bytes_left
        ):
            out("That is more than your plan has left this month. "
                "Start with --sample, or wait for the new month.")
            return 2

        result = 0
        for i in range(0, len(todo), BATCH):
            chunk = todo[i : i + BATCH]
            started = time.monotonic()
            answer = client.batch([str(p) for p in chunk], endpoint="signallab", round=True)
            for p, r in zip(chunk, answer["results"]):
                d = r.get("data") or {}
                row = {
                    "file": name(p),
                    "findings": "; ".join(findings(d)) if r["ok"] else "",
                    "error": "" if r["ok"] else r.get("error", "failed"),
                }
                row.update({k: d.get(k, "") for k in FIELDS})
                rows[row["file"]] = row
                if not r["ok"]:
                    result = 2
            write_csv(target, rows)
            if len(answer["results"]) < len(chunk):
                # the API stopped inside the batch: a full minute, or a limit of your plan
                why = (answer["results"] or [{}])[-1].get("error", "no reason given")
                done = len(answer["results"])
                out(f"The API stopped after {done} of {len(chunk)} files: {why}")
                out("Run it again: the files already in the CSV are skipped.")
                return 2
            # A local file takes two requests: one to mint its signed upload, one for the
            # analysis. Stay under the requests a minute of the plan.
            pace = 2 * len(chunk) * 60 / usage["requestsPerMinute"]
            wait(max(0, pace - (time.monotonic() - started)))
    except AudioLabError as e:
        out(f"Stopped: {e}")
        return 2
    write_csv(target, rows)
    summary(rows, out)
    return result


if __name__ == "__main__":
    sys.exit(main(sys.argv))

It lists the audio files in the folder, takes the sample if you ask for one, asks what your plan has left and refuses a run that does not fit, then sends the files to POST /v1/batch with endpoint="signallab", 20 at a time, each uploaded first. After every batch it writes the CSV. If the API stops inside a batch (a full minute, or a limit of your plan), the script says so and exits; run it again and the files that are already in the CSV are skipped.

A run on seven recordings

To show what the findings look like, we built seven recordings from one of our own voice samples, each with one known defect, and ran the script on them. The recordings were measured by the engine, and the script was run against a stand-in for the API with those answers, so what follows is what the script prints.

clean.wav: the voice and a quiet noise floor, as a 16-bit WAV at 44.1 kHz.
telephone.wav: brought down to 8 kHz and back up to 16 kHz.
from-mp3-64k.wav: encoded as a 64 kbps MP3 and decoded to WAV.
clipped.wav: 24 dB louder than the format allows.
stereo-twice.wav: the same voice in both channels of a stereo file.
spliced.wav: eight seconds as it is, then four seconds of the same voice at telephone bandwidth.
eight-bit.wav: stored as 8-bit samples.

7 files to measure (14 MB). Plan plus: 9988 calls left this month, 120 requests a minute.

7 files measured, 0 failed

what                     files
upsampled                    1   from-mp3-64k.wav
lossy source                 0
source unclear               2   from-mp3-64k.wav, spliced.wav
bandwidth splices            1   spliced.wav
stereo, same both sides      1   stereo-twice.wav
clipping                     2   clipped.wav, eight-bit.wav
fewer than 16 bits           1   eight-bit.wav

bandwidth by sample rate
  16000 Hz    1 file   4297 Hz
  44100 Hz    6 files  16516 to 22028 Hz, median 22028

The CSV has one row per file with the findings and the numbers behind them. The findings column of these seven:

clean.wav         -
clipped.wav       clipping (22 regions)
eight-bit.wav     clipping (a short run at the peak); fewer than 16 bits (8)
from-mp3-64k.wav  upsampled (from 32000 Hz); source unclear
spliced.wav       source unclear; bandwidth splices (1)
stereo-twice.wav  stereo, same both sides
telephone.wav     -

How to read it

A finding is a reason to listen, not a verdict. Open the CSV, sort by the column you care about, and listen to the files at the top.
Read the bandwidth next to the sample rate. telephone.wav has no finding at all, and it is the one that matters most: its bandwidth is 4,297 Hz in a file at 16,000 Hz, so nothing above about 4 kHz was ever in it. The engine names an original rate (upsampledFromHz) only when the roll-off starts where a standard rate would put it, and otherwise says nothing, so here the summary of bandwidth by sample rate is what shows it. Look for a sample rate with a wide spread, or one file far below the others.
Two explanations for one wall are one finding. from-mp3-64k.wav says “upsampled from 32,000 Hz” and “source unclear”. Its bandwidth is 16,516 Hz in a file at 44.1 kHz. A low-bitrate MP3 and a resampled 32 kHz source leave a similar wall: lossySource stays unknown rather than guess, and upsampledFromHz names a 32 kHz source. The cause here was an MP3, so that second line is a wrong explanation of a right observation, and the page for the measure says a very low bitrate codec can mimic a resampler. For your purpose both lines say the same thing: nothing above about 16.5 kHz, so this file does not carry what its sample rate promises.
A join shows as a bandwidth splice. spliced.wav has one: the bandwidth jumped by more than 3 kHz and stayed there for at least two seconds, where the telephone-band part starts. In music a count can also be an arrangement change. In a conversation it is more often a different source, a different codec, or a clip pasted in.
Stereo twice is two numbers. stereo-twice.wav has two channels and a monophonicProbability of 1. The name is misleading: it is the correlation between left and right, never below 0, and near 1 means mono in disguise. The script calls a stereo file with 0.99 or more “same both sides”; that limit is ours. If your recordings put each speaker on their own channel, a value near 1 means they were mixed down.
Clipping has two columns. hasClipping is true for any short run of samples at the file’s own peak. clippingRegions counts only the stretches long enough to matter. eight-bit.wav has the first and no regions, clipped.wav has 22. Look at the regions when you want to hear it.
Bits in use. effectiveBitDepth is the number of bits the samples really use. A file stored as 16 bits that reads 8 had its resolution cut before it was stored. A lossy file decoded on the fly reads 24, which is why this number does not find lossy sources: that is what lossySource is for.

What it cannot do

It cannot see every lossy source. The engine looks for the wall an encoder leaves. Our own voice samples are MP3 files and the engine reads them as lossless: it found no wall in them. lossySource: lossless means no wall was found, not that the file was never lossy.
It does not always name the original rate of a resampled file. telephone.wav is the example. The page for the measure lists the cases it deliberately leaves alone.
It does not prove that two files share a source: there is no fingerprint. A join between two recordings of the same kind leaves no trace in the band.
It does not say whether a voice is synthetic, identify a speaker, check a label or read a transcript.
A batch is one call per file in sequence, so a large corpus is slow and costs calls. Sample first.

Privacy and limits

Your audio is not kept. It is analysed in memory in Frankfurt. A compressed file (MP3, FLAC, M4A) passes through a temporary file that is deleted right after decoding. A file you upload through the signed-upload route is removed within the hour. We keep the numbers and, if you let us, the file name. See /privacy.
Limits. Free: 1,000 calls and 2 GB a month, 60 requests a minute. Plus: 10,000 calls, 20 GB, 120 a minute. A batch takes up to 20 files, a file can be 150 MB, and each file in a batch is one call. Every accepted request counts, including one that then fails. A file that cannot be measured is a row with its error in the CSV, and the other files are still measured.
Where you stand. AudioLab().usage() returns the plan and the calls and data used this month against the limits. The script asks it before it sends anything. Reading it is free and does not count as a call.