/docs · General · Practical

Check audio quality in CI: fail a build when a file breaks your rules

A short Python script and a GitHub workflow that measure each audio file with the AudioLab API, compare the numbers with limits you set, and fail the build with a message that says why.

Who this is for

You ship audio out of a pipeline: text-to-speech prompts, voice-agent recordings, podcast episodes, game or app sound. A build should stop when a file is too quiet, clips, is band-limited or is noisy, before a listener hears it.

It is for teams that set their own limits. AudioLab measures the file; whether a number passes is your rule. It does not judge how natural a voice sounds and it does not read the words. See what AudioLab does not do.

What you get

A script, audio_gate.py, that measures each file you give it, checks your limits and exits with a code your CI understands. Its only dependency is the audiolabtools package.
A limits file, gate.json, with one entry per number.
A workflow, audio-gate.yml, that runs the script on pull requests and measures only the audio files the pull request adds or changes.
Every failing line names the number, its value and the limit, and says how far to trust the number: its status and its margin. The person who reads a red build knows what they are looking at. See how to read a response.

Before you start

Get a key. It is free and needs no card: sign in on /account and generate one. In GitHub, store it as a repository secret named AUDIOLAB_KEY. Never commit it.
Know that the files leave the runner. Every file the gate measures is sent to the AudioLab API, hosted in the EU, and only numbers come back. See “Privacy and limits” below. If your audio is not allowed to leave your own systems, this is not the tool for the build. The workspace measures in your browser and uploads nothing, but by hand, not in a build.
Know what it costs. The ten numbers come from three analyses: the loudness numbers from one call, the voice numbers from a second, the bandwidth from a third. The script asks only for the calls your limits need. The limits file below uses all three, so a file costs three calls; a limits file with loudness numbers only costs one call a file. Each call counts against your calls and your data for the month. The free tier is 1,000 calls a month, 2 GB a month and 60 requests a minute.
Python 3.9 or newer, and the package:

pip install audiolabtools

Your limits

{
  "integratedLufs": { "min": -26, "max": -16 },
  "truePeakDb": { "max": -1 },
  "clipping": { "oneOf": ["None"] },
  "signalToNoiseDb": { "min": 30 },
  "bandwidthHz": { "min": 7000 }
}

Each entry names one of the ten numbers and gives a min, a max or both. clipping is a word, so it gets a list of the words you allow in oneOf. These limits are examples: set your own from your own good recordings. Take twenty you consider good and use their 5th and 95th percentile, as step 7 of “How to read them” in the comparison guide says.

The script

"""Fail a build when an audio file breaks your rules.

    python audio_gate.py gate.json audio/intro.wav audio/outro.wav

gate.json holds the limits, one entry per number:
https://audiolab.tools/docs/start/audio-quality-gate-in-ci
Every file is sent to the AudioLab API; only numbers come back.

Exit code: 0 every file passes, 1 a file breaks a rule, 2 a file could not be checked
(a key, a limit of your plan, no network). A build must not pass on 2.
"""

from __future__ import annotations

import json
import sys
import time

from audiolabtools import (
    ApiTimeout,
    ApiUnreachable,
    AudioLab,
    AudioLabError,
    AuthenticationError,
    MissingApiKey,
    PlanLimitError,
    RateLimitError,
)

PASSED, BROKEN, COULD_NOT_CHECK = 0, 1, 2

# The same for every file, so there is no point in going on: a key that is refused,
# a limit of your plan, a minute that is full, no network.
STOPS = (
    AuthenticationError, PlanLimitError, RateLimitError,
    MissingApiKey, ApiUnreachable, ApiTimeout,
)

NEAR = "  within the margin: the number cannot say if the file is past it"


def broken_rules(numbers, rules):
    """The rules one file breaks, one line each.

    A line carries the status and the margin of the number: how far to trust it.
    """
    lines = []
    for field, rule in rules.items():
        m = numbers[field]
        trust = (m.status or "no status") + (f", margin {m.margin}" if m.margin else "")
        if m.value is None:
            lines.append(f"{field} could not be measured")
        elif "oneOf" in rule:
            if m.value not in rule["oneOf"]:
                allowed = ", ".join(rule["oneOf"])
                lines.append(f"{field} is {m.value}, allowed: {allowed} ({trust})")
        else:
            mg = m.margin
            slack = max(mg.plus_minus or 0, mg.plus or 0, mg.minus or 0) if mg else 0
            low, high = rule.get("min"), rule.get("max")
            past = []
            if low is not None and m.value < low:
                past.append(("below the minimum", low))
            if high is not None and m.value > high:
                past.append(("above the maximum", high))
            for word, limit in past:
                lines.append(f"{field} is {m.value} {m.unit}, {word} {limit} ({trust})")
                if abs(m.value - limit) <= slack + 1e-9:
                    lines.append(NEAR)
    return lines


def measure(client, path, fields, wait=time.sleep):
    """The numbers of one file. On a full minute the API says how long to wait: do that once."""
    try:
        return client.ten_numbers(path, only=fields)
    except RateLimitError as e:
        wait(e.retry_after or 60)
        return client.ten_numbers(path, only=fields)


def main(argv, client=None, out=print, wait=time.sleep):
    if len(argv) < 3:
        out("usage: python audio_gate.py gate.json file [file ...]")
        return COULD_NOT_CHECK
    with open(argv[1], encoding="utf-8") as f:
        rules = json.load(f)
    client = client or AudioLab()  # reads AUDIOLAB_KEY
    result, broken, unchecked = PASSED, 0, 0
    for path in argv[2:]:
        try:
            lines = broken_rules(measure(client, path, list(rules), wait), rules)
        except STOPS as e:
            out(f"STOP   {path}\n         {e}")
            return COULD_NOT_CHECK
        except AudioLabError as e:  # a problem with this file only: say so, check the others
            out(f"ERROR  {path}\n         {e}")
            unchecked += 1
            result = COULD_NOT_CHECK
            continue
        out(f"{'FAIL' if lines else 'ok  '}   {path}")
        for line in lines:
            out(f"         {line}")
        if lines:
            broken += 1
            result = max(result, BROKEN)
    checked = len(argv) - 2
    out(f"{checked} file{'' if checked == 1 else 's'} checked, {broken} break a rule, "
        f"{unchecked} could not be checked")
    return result


if __name__ == "__main__":
    sys.exit(main(sys.argv))

It reads the limits, asks ten_numbers for the numbers the limits name (only=, so nothing else is called), and prints one line per broken rule. Two things it decides for you. A full minute is waited out once, as long as the API asks in its Retry-After, because that is what the API says to do. A key that is refused, a limit of your plan or no network stops the run at the first file, because the same would happen to every file; a problem with one file is reported and the others are still checked.

The workflow

name: audio quality gate

on:
  pull_request:
    paths: ["audio/**"]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install audiolabtools
      - name: Measure the audio files this pull request adds or changes
        env:
          AUDIOLAB_KEY: ${{ secrets.AUDIOLAB_KEY }}
          BASE: ${{ github.base_ref }}
        run: |
          mapfile -d '' files < <(git diff -z --name-only --diff-filter=AM \
            "origin/$BASE...HEAD" -- 'audio/*.wav' 'audio/*.mp3' 'audio/*.flac')
          if [ "${#files[@]}" -eq 0 ]; then echo "No audio files changed."; exit 0; fi
          python tools/audio_gate.py tools/gate.json "${files[@]}"

Put audio_gate.py and gate.json in tools/, and the audio in audio/. The last step passes the script only the files that this pull request adds or changes, so an old file is not measured again on every push. A pull request from a fork gets none of your secrets. Without a key the script runs as the keyless trial, which allows only 20 calls a month, so the gate soon stops with a limit error and exit code 2. Decide whether you skip the job for forks.

What a failed check looks like

FAIL   audio/prompt-welcome.wav
         integratedLufs is -31.5 LUFS, below the minimum -26 (validated, margin ±0.1 LU)
1 file checked, 1 break a rule, 0 could not be checked

This is the output of the script on the example limits, for a file that is far too quiet. The exit code is 1, so the build fails, and the log says which file, which number, which limit and how far to trust the number.

When a number is closer to the limit than its margin, the line says so. The same file against a minimum of -31.4 instead of -26:

FAIL   audio/prompt-welcome.wav
         integratedLufs is -31.5 LUFS, below the minimum -31.4 (validated, margin ±0.1 LU)
           within the margin: the number cannot say if the file is past it
1 file checked, 1 break a rule, 0 could not be checked

How to read it

Exit 1 means the file broke a rule. Exit 2 means the gate could not tell. The line carries the error code and the page that explains it, for example https://audiolab.tools/docs/errors#quota_exceeded. A build must not pass on exit 2: a gate that cannot reach the API has not checked anything.
Read the margin before you argue with a failure. A line followed by “within the margin” is a number that cannot tell you whether the file is past the limit, because the number itself can be off by more than the distance. When the margin is not symmetric, the script uses the larger side. Either loosen the limit by the margin or fix the file. The script does not decide for you.
A limit on a proxy compares, it does not state a fact. speechRatio and longSilences are proxies, and the line says so. They are good at catching a big change and poor at policing a decimal.
Start loose, then tighten. Run the gate with wide limits for a while and read what it prints, then narrow the limits to what your good recordings show. A gate that fails every build gets switched off.
A run measures what it is given. Changing gate.json does not measure the files that are already in the repository. After you change a limit, run the script once over the whole folder.

What it cannot do

It measures the file, not what is said. A file inside every limit can still be the wrong sentence, a poor voice or the wrong language.
It has no memory. It does not compare a file with an earlier version or with a reference. To compare vendors or versions over many sentences, use the comparison guide.
It does not run offline. The engine is not published as a package, so every file goes to the API.
Its numbers have the status and the margin they have. See what AudioLab does not do.

Privacy and limits

Your audio is not kept. It is analysed in memory in Frankfurt. A compressed file (MP3, FLAC, M4A) passes through a temporary file that is deleted right after decoding. A file you upload through the signed-upload route is removed within the hour. We keep the numbers and, if you let us, the file name. See /privacy.
Limits. Free: 1,000 calls and 2 GB a month, 60 requests a minute. Plus: 10,000 calls, 20 GB, 120 a minute. A local file over 4 MB is uploaded once through the signed-upload route before it is measured: that counts as a request but not as a call. Every accepted request counts, including one that then fails. An error tells you what to do: /docs/errors.
Where you stand. AudioLab().usage() returns the plan and the calls and data used this month against the limits. Reading it is free and does not count as a call.