Troubleshooting

This page is organised by what you see, because that is what you have when something breaks. Find the symptom, read what it means, run the commands, apply the fix. Each section is self-contained; you should not have to understand how ETTIX is built to use it.

Three things worth knowing before you start

ETTIX serves two separate listeners and it matters which one you are talking to. The playback listener (http.listen, :8080 by default) carries HLS playlists, segments, content keys, DVR playback and the two probes /health and /ready. The administration listener (http.admin_listen, 127.0.0.1:8081 by default) carries every /api/v1/ route and the administration interface. The administration API is never served on the playback listener, so a request to :8080/api/v1/status is a 404 rather than a permissions problem.

Almost every diagnosis on this page begins with one of three commands: the service status, the journal, or config check. Run all three before forming a theory.

systemctl status ettix-stream
journalctl -u ettix-stream -n 200 --no-pager
ettix-stream config check --config /etc/ettix-stream/config.json

The examples below use curl against http://127.0.0.1:8081 with no credential, which works when http.admin_auth is empty and the listener is on loopback. If you have configured a credential, add the header — an API key is presented as its id and secret joined by a dot:

curl -H "Authorization: Bearer <id>.<secret>" http://127.0.0.1:8081/api/v1/status

Symptom index

The service will not start

What you see

systemctl status ettix-stream shows the unit as failed, or as activating (auto-restart) in a loop every couple of seconds. The shipped unit sets Restart=always with RestartSec=2, so a configuration the server refuses looks like a restart loop rather than a single clean failure.

What it means

Nearly always the configuration file. ETTIX validates the whole document before it binds a single port, and unknown fields are an error rather than something quietly ignored — a setting you misspelled must not look as though it took effect. Because validation happens first, a failure here means nothing was started and nothing on the machine was changed.

What to run

systemctl status ettix-stream
journalctl -u ettix-stream -n 200 --no-pager
ettix-stream config check --config /etc/ettix-stream/config.json

config check is the first thing to run, always. Its exit code tells you which kind of problem you have:

Exit 3 is the interesting one. It lists all the problems rather than stopping at the first, so one editing pass can fix them all:

$ ettix-stream config check --config /etc/ettix-stream/config.json
error: /etc/ettix-stream/config.json: config: 2 problems:
  - http.admin_listen: "0.0.0.0:8081" is not loopback; bind it to 127.0.0.1, or set
    http.admin_allow_remote and configure http.admin_auth once you have put a firewall,
    VPN or authenticating proxy in front of it too
  - streams[0] (channel1).source.rtmp.stream_key: at least 8 characters
$ echo $?
3

How to fix it

Edit the field the message names and run config check again until it exits 0, then systemctl restart ettix-stream. The messages that come up most often:

Everything answers 503, and the administration interface still works

What you see

Every playlist, segment, content key and DVR request answers HTTP 503 with a Retry-After: 60 header and a body naming a licence code:

{
  "error": {
    "code": "LICENSE_EXPIRED",
    "message": "this server is not licensed to serve this request",
    "request_id": "…"
  }
}

Publishers are refused. Recording has stopped. /ready answers 503, so a load balancer drains this server. But /health still answers 200 and the administration interface is perfectly responsive.

What it means

This is the licence, and the asymmetry is deliberate. The streaming engine stops; the administration interface stays up so that the licence can be installed without physical access to the machine. /health is deliberately unaffected — the process is alive and an operator must be able to reach it. That is also why you should health-check on /health but route on /ready: routing on /health sends viewers to a server that answers 503 to every playlist.

What to run

curl -s http://127.0.0.1:8081/api/v1/license | jq .

The state field says which situation you are in. There are six, and the cure is different for each:

How to fix it

curl -sX POST http://127.0.0.1:8081/api/v1/license/renew | jq .

curl -sX POST http://127.0.0.1:8081/api/v1/license/activate \
  -H 'Content-Type: application/json' \
  -d '{"token": "<activation token>"}' | jq .

curl -sX POST http://127.0.0.1:8081/api/v1/license/activate \
  -H 'Content-Type: application/json' \
  -d '{"lease": "<lease document>"}' | jq .

The matching log lines, if you would rather grep than poll, are LICENSE_OFFLINE, LICENSE_EXPIRED, LICENSE_REVOKED, LICENSE_RESTRICTED, LICENSE_GRACE_ENDED and CLOCK_ROLLBACK.

"no trusted licence authority keys are available"

What you see

Licensing reports unlicensed and every activation attempt fails. The message is:

license: no trusted licence authority keys are available: this build has no
compiled-in licence authority key and none is configured, so no lease can be
verified; install a release build, or set license.trusted_keys for a
self-hosted authority

What it means

This is a build problem, not a configuration one. Nothing you can put in config.json will change it, and no amount of re-activating will help. A lease is an Ed25519-signed document; verifying it requires the licence authority's public key, which is compiled into a release build. A build made without one installs, starts, ingests, records and serves perfectly well — and then refuses every activation, because it has nothing to verify a lease against.

The message is written this way on purpose. The alternative failure — a later, vaguer complaint about an unknown key id — has a completely different cure, and telling the two apart afterwards is much harder than saying so now.

How to fix it

Install an official release build. The release script refuses to package a binary with no authority key unless it is explicitly told to, so a build in this state came from a local go build or from a deliberately key-less release.

The one legitimate case for seeing this on a build you meant to make is running your own licence authority, which is configured by adding to license.trusted_keys. Configuration can only add keys to the compiled-in set; it can never remove one, and a configured key whose id collides with a compiled-in id is refused rather than preferred. Otherwise editing a JSON file could substitute a different key for a built-in id, and the built-in key would be decoration.

A publisher is refused

What you see

The encoder connects and is then dropped. OBS reports a failure to publish; ffmpeg typically prints Operation not permitted. On the wire the connection is closed with NetStream.Publish.BadName and the description Publish rejected. The stream stays in state idle.

What it means

The client is told nothing useful on purpose. Every refusal — wrong key, unknown stream, wrong RTMP application, stream disabled, licence lapsed — looks identical to the publisher, and all of them are padded to the same fixed delay so they cannot be told apart by timing either. A publisher that could distinguish "wrong key" from "no such stream" could enumerate your streams.

The real reason is in your log and nowhere else.

What to run

journalctl -u ettix-stream --since -10m | grep "publish rejected"

Each rejection logs one line carrying a stable code. These codes are part of the operational contract — alerts get written against them — and they do not change between a stream that uses a plain stream key and one that grows a full publisher block later:

How to fix it

For a wrong or missing key, the key is in the configuration file, in the stream's source.rtmp.stream_key:

grep -n stream_key /etc/ettix-stream/config.json

The installer generates a real key when it writes a starter configuration, so on a fresh installation this is a value you have never seen rather than a default you can guess. If the file still says change-this-stream-key, the server will refuse to start; replace it. A stream key must be at least 8 characters.

The publisher may present it in either of two places, and ETTIX accepts both because encoders differ:

allow_anonymous turns the credential check off for a stream entirely: any publisher that can reach the RTMP listener is accepted. It exists for a listener that is already protected by something else — a private network, a firewall rule — and it is refused in combination with configured publisher credentials, so that a configuration cannot half-say two things at once. When it is set, ETTIX logs INGEST_UNAUTHENTICATED at startup so the state is never silent. Turn it on only if you can say what is protecting the listener instead.

Every stream must resolve to one of the three: a stream_key, allow_anonymous, or configured publisher credentials. A stream with none of them fails validation with stream_key is required unless allow_anonymous is true or the stream configures publisher credentials.

An SRT source will not connect

What you see

The stream never leaves idle. connect_failures climbs on the source. The journal carries source connect failed lines, rate-limited so a source retrying every second does not flood the log.

What it means

First, the direction. ETTIX dials out for an srt_pull source: it is the caller and the far end must be a listener. If the far end is also configured to call out, nothing will ever connect no matter what else is right.

Second, SRT is UDP. A firewall rule, a security group or a NAT mapping written for TCP on that port does nothing at all. This is the single most common cause and it is invisible from the ETTIX side, because an unreachable listener and a refused handshake look the same from here.

What to run

The connection error is kept per source, with the passphrase and stream id stripped out of it, and it is easiest to read from the API:

curl -s http://127.0.0.1:8081/api/v1/streams/channel1/sources | jq '.'
journalctl -u ettix-stream --since -10m | grep "source connect failed"

Look at last_error on each source. There are two messages that matter and they mean quite different things:

A third case connects and then produces nothing useful: the source shows as connected, but srt.packets_undecrypted climbs. That is a passphrase that matches on one side only.

How to fix it

ettix-stream config check --config /etc/ettix-stream/config.json

The stream is live but viewers stutter

What you see

The stream reports live, the playlist updates, and playback still breaks up, freezes or drops out.

What it means

There are three separate places this can come from and they have three different cures. Two counters tell them apart, and reading them the wrong way round wastes a lot of time.

What to run

curl -s http://127.0.0.1:8081/api/v1/streams/channel1 | jq '.sources[] | {id, bitrate_bps, srt, demux}'

DVR is not recording

What you see

Nothing is appearing on disk, or a destination that was working has stopped.

What to run

curl -s http://127.0.0.1:8081/api/v1/disks | jq '.disks[].destinations'
curl -s http://127.0.0.1:8081/api/v1/streams/channel1/dvr | jq .
journalctl -u ettix-stream | grep "dvr destination"

What it means

Each destination reports a state, and every transition is logged. Read the state and the reason together:

The reason field carries the specific cause, and these strings are exact:

How to fix it

curl -sX POST http://127.0.0.1:8081/api/v1/streams/channel1/dvr/recording \
  -H 'Content-Type: application/json' -d '{"action": "start"}' | jq .

systemctl cat ettix-stream | grep ReadWritePaths
systemctl edit --full ettix-stream     # add the new root to ReadWritePaths=
systemctl daemon-reload
systemctl restart ettix-stream

ettix-stream config check --config /etc/ettix-stream/config.json --dvr-roots

If the index rather than the media is damaged — a destination reported as degraded after an unclean shutdown, say — the index is rebuilt from the media at startup, so most damage needs no action at all. When it does:

ettix-stream dvr verify --path /mnt/dvr1 --stream channel1          # fast
ettix-stream dvr verify --path /mnt/dvr1 --stream channel1 --deep   # reads every byte
ettix-stream dvr repair --path /mnt/dvr1 --stream channel1          # rebuild from media

repair reads the media and writes a new index. It does not modify or delete a segment. verify --deep compares each segment against the SHA-256 recorded when it was written, which detects corruption, not tampering — the digest lives in the journal on the same volume as the media, so anyone who could alter one could alter the other. The tool says so in its own output.

The administration interface shows a sign-in form that never works, or I cannot reach it at all

Three distinct problems present almost identically. Work through them in this order.

1. Nothing responds on the administration port

The administration listener defaults to 127.0.0.1:8081 — loopback only. Not "loopback unless you set something", and not an empty default: an omitted admin_listen binds loopback rather than every interface, and the server refuses at the point of binding as well as during validation, so no code path can produce a remotely reachable administration API by omission.

This is because the API is privileged. It can stop a broadcast, read every stream key, and delete recorded media. Reachable and anonymous is a combination the server refuses rather than warns about.

So from another machine, there is nothing to reach, and that is correct. Test from the server itself first:

curl -s http://127.0.0.1:8081/api/v1/status | head

If that works, the service is fine and you need a tunnel. An SSH local forward is the recommended way in, because it needs no configuration change on the server and no port opened anywhere:

ssh -N -L 8081:127.0.0.1:8081 you@stream.example.com

Then open http://127.0.0.1:8081/ on your own machine. Browsers treat 127.0.0.1 as a trustworthy origin, so this works with no certificate and no exceptions to click through.

If you genuinely need the listener bound to a real address — for a management VLAN, say — it takes two settings, and validation will not accept one without the other: http.admin_allow_remote to acknowledge the exposure, and at least one credential in http.admin_auth. Put a firewall, VPN or authenticating proxy in front of it as well. The server logs ADMIN_API_REMOTE whenever it is in this state.

2. The sign-in form appears, and every attempt fails

You reach the interface, it shows a sign-in form, and signing in reports "Sign-in failed" no matter what you type — including the credentials you are certain of.

This means no credential is configured. The sign-in routes exist only when http.admin_auth configures at least one user or API key; with none configured they are not registered at all, the interface's first request for a session gets a 404, and it falls back to showing the form. The form is real; it is talking to endpoints that are not there.

Note the corollary: on a loopback listener with no credential configured, the API itself is open. So while you cannot sign in through the browser, you can use curl from the machine and everything will work. That is the supported single-operator setup, and it is why validation permits an empty admin_auth only on loopback.

To create a credential:

ettix-stream admin hash --user ops

It prompts twice on the terminal with echo off and prints one line to standard output — the JSON object to paste into http.admin_auth.users. The password is never a flag, because a password on a command line ends up in the shell history, in /proc/<pid>/cmdline for every user on the machine, and in the audit log. The minimum length is 12 characters.

$ ettix-stream admin hash --user ops

Add to http.admin_auth.users:

{"username": "ops", "password_hash": "$argon2id$v=19$m=19456,t=2,p=1$…", "read_only": false}

For automation, generate an API key instead. The secret half is generated rather than chosen, and it is printed once — ETTIX stores only the hash and cannot print it again:

ettix-stream admin hash --api-key monitoring --read-only

Paste the object into the configuration, then:

ettix-stream config check --config /etc/ettix-stream/config.json
systemctl restart ettix-stream

3. Sign-in appears to succeed, then bounces straight back to the form

You enter the right password, the page moves, and you are immediately back at the sign-in form. Repeatedly.

This is the session cookie being refused by your browser. The cookie is named with the __Host- prefix, which is a browser-enforced promise that the cookie is Secure, has Path=/ and has no Domain — together, that stops a subdomain from setting a cookie this server would then trust. The Secure attribute is therefore always present and is not conditional on anything.

Browsers accept a Secure cookie from http://127.0.0.1 and http://localhost, so the loopback default works with no TLS. The one case a browser refuses is a remote administration listener reached over plain HTTP with no TLS-terminating proxy in front of it. Refusing is the right behaviour there: the alternative is a session cookie travelling in the clear across a network.

Fix it by reaching the interface over loopback (the SSH forward above) or by putting TLS in front of the administration listener. Note that curl does not enforce cookie prefixes, so an API-key credential will keep working from the command line while a browser cannot sign in at all — which is exactly how this becomes confusing.

Locked out

After 10 consecutive failures the account is refused outright for 15 minutes, and the client address is counted separately with the same limit. The response is HTTP 429 and the interface says "Too many failed attempts. Try again later."; the log carries ADMIN_LOGIN_LOCKED_OUT. Both counters are in memory, so restarting the service clears them — at the cost of ending every active session.

A change in the administration interface will not save

What you see

You edit something in the interface, press save, and it reports an error. The change is not applied and is not written to disk.

What it means

A configuration change is validated in full, written to the file in full, and then applied — in that order, so a bad document changes nothing and a crash between the write and the reload leaves the file and the next start agreeing with each other. The error tells you which of those steps failed.

How to fix a permission error

The administration API writes the configuration file back — saving from the interface, enabling or disabling a stream, editing a destination. That write is an atomic replace: a temporary file is created in the same directory, flushed, and renamed over the original. A crash at any point leaves either the old file or the new one, never a truncated one. But it means the service needs write permission on the directory, not merely on the file.

Two things must both be true:

# 1. The directory is listed in the unit
systemctl cat ettix-stream | grep ReadWritePaths
#    Expect /etc/ettix-stream among the paths

# 2. The directory is group-writable by the service account
ls -ld /etc/ettix-stream
#    Expect: drwxrws--- root ettix-stream   (mode 2770)
ls -l /etc/ettix-stream/config.json
#    Expect: -rw-r----- root ettix-stream   (mode 0640)

Both are set correctly by install.sh. The setgid bit keeps the group on files the service creates; the file stays 0640, so nothing outside the service account can read the stream keys. If you have tightened the directory to 0750 by hand, every save will fail with "permission denied" — which reads like a bug in the API rather than a permission that was never granted.

The trade-off is real and is worth stating: a compromised streaming process can rewrite its own configuration. It is accepted because the alternative is shipping a configuration API that cannot save. root still owns the directory.

Saved, but nothing happened

If the response says "restart_required": true, the change was written and accepted but part of it cannot take effect in a running process. Listener addresses, the RTMP block, the HLS cache size, the data directory, and the licence and telemetry blocks are all read once when the process starts:

systemctl restart ettix-stream

This is reported rather than hidden for a specific reason. An operator who corrects a licence authority URL, sends a reload, is told it succeeded, and then watches nothing change has no way to tell a configuration mistake from a broken server. Stream additions, removals and changes do apply on a reload (systemctl reload ettix-stream, which sends SIGHUP).

Certificates

What you see

HTTPS does not work, or a browser refuses the certificate, or the service will not start with tls is configured but the certificate manager could not start.

What it means

There is no silent fallback. A configuration that asks for TLS and cannot have it is a startup failure, not a server that quietly serves plain HTTP, and no substitute or self-signed certificate is ever presented — a certificate a client will not trust is worse than a refusal it can read.

What to run

The certificate status carries the last error, already safe to display:

curl -s http://127.0.0.1:8081/api/v1/certificates | jq .

And to see what a client actually receives — ask openssl, not a browser, because a browser may be showing you a cached intermediate:

echo | openssl s_client -connect stream.example.com:443 -servername stream.example.com 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates -ext subjectAltName

How to fix it

For a certificate you manage yourself, correct the file and let the reload timer pick it up — reload_interval re-reads both files on a timer so an external renewer can replace them without restarting ETTIX. A file that has changed but does not parse is an error that keeps the working certificate in place, so a half-written file during a renewal is a transient log line rather than a broken server.

For ACME, the "renew now" button in the administration interface, or the endpoint behind it, clears the backoff so that a corrected configuration takes effect at once. It deliberately does not lift the daily attempt cap:

curl -sX POST http://127.0.0.1:8081/api/v1/certificates/renew | jq .

If you switched from staging to production and browsers still see the untrusted certificate, the server is serving the one it already has. The staging state has to go:

systemctl stop ettix-stream
rm -rf /var/lib/ettix-stream/acme
systemctl start ettix-stream

The server, or a build of it, is inexplicably slow

What you see

Everything works and everything is slow: segments take a long time to write, the DVR falls behind, a test run that used to take minutes takes an hour, or the machine feels loaded while showing very little CPU.

What it means

Measure fsync before believing anything else. Durable writes dominate this kind of workload, and the cost of one flush varies by two orders of magnitude between volumes on the same machine. During this project's own validation, fsync on one volume measured 83 ms median on an idle machine — an ageing consumer SSD, nearly full — while a RAID array on the same host answered in 0.93 ms. Ninety times. Every theory about contention, about the code, and about load was wrong until that was measured, and load only doubled the bad figure: the disk was pathological at rest.

What to run

ETTIX does not ship a benchmark, so use whatever the machine has. This gives a rough per-flush cost on a specific volume, which is what matters — run it on each volume separately, including the DVR roots and whatever holds TMPDIR:

# 1000 flushed 4 KiB writes. Divide the elapsed time by 1000.
dd if=/dev/zero of=/var/lib/ettix-stream/fsynctest bs=4k count=1000 oflag=dsync
rm -f /var/lib/ettix-stream/fsynctest

If fio is installed it gives a proper distribution rather than a mean, which is more useful because the tail is what hurts. It is not part of any ETTIX dependency and may well not be there.

Then read what ETTIX itself measures. The disk figures are per DVR destination and are the closest thing to a direct answer:

curl -s http://127.0.0.1:8081/api/v1/disks | jq '.disks[] | {device, busy_percent, destinations}'
curl -s http://127.0.0.1:8081/api/v1/system | jq '{cpu: .cpu, memory: .memory}'

How to fix it

Move the work to a faster volume. That is usually the whole answer, and it is cheaper than any amount of tuning. For a DVR destination, give it a dedicated filesystem on storage that can take the write rate.

If you cannot, the durability setting per destination is the lever, and it is a genuine trade rather than a free win:

If you are building and testing ETTIX rather than running it, the same rule applies to the test suite: put TMPDIR and any test database on the fastest volume the machine has before concluding anything about the code. On this project's validation host, moving them accounted for every timeout that had previously been blamed on contention.

Collecting evidence for support

Run these and send the output. Between them they identify the build, prove whether the configuration is valid, and describe the state of the licence, the streams and the recent past.

ettix-stream version --json
ettix-stream config check --config /etc/ettix-stream/config.json
curl -s http://127.0.0.1:8081/api/v1/status | jq .
curl -s http://127.0.0.1:8081/api/v1/streams | jq .
curl -s http://127.0.0.1:8081/api/v1/license | jq .
curl -s http://127.0.0.1:8081/api/v1/disks | jq .
curl -s http://127.0.0.1:8081/api/v1/certificates | jq .
curl -s http://127.0.0.1:8081/api/v1/system | jq .
curl -s 'http://127.0.0.1:8081/api/v1/events?limit=500' | jq .
journalctl -u ettix-stream --since -1h --no-pager

Add the Authorization: Bearer <id>.<secret> header to each curl if the administration API has a credential configured.

The installation_id from /api/v1/license is the identifier support will ask for first. It is a public identifier and is safe to quote; so is the public_key in the same response, which is registered with Ettix.com by design. The private half is not in that response, is not reachable from it, and a test asserts that its bytes never appear in the marshalled form.

Everything in the list above is redacted. The log formatter and the event ring share the same redaction function rather than each having a copy, so a secret cannot reach one sink by escaping the other, and release validation confirmed on a live system that the stream key, the SRT passphrase and the administration password appeared zero times in the journal.

If the problem involved a specific request, include its request_id. Every error body carries one and it appears in the access log, which makes finding the corresponding server-side line exact rather than a search by timestamp.

What is not on this page

Two procedures are deliberately kept separate because they are destructive if run in the wrong situation, and both need their own reading before you touch anything: recovering a lost encryption master key, and rebuilding an installation after the machine is gone. There is no recovery for a master key with no backup — encrypted recordings are unreadable, permanently, and Ettix holds no copy, which is precisely what makes the recordings yours. If you have not taken a backup of it, take one now:

ettix-stream keys export --config /etc/ettix-stream/config.json --out ettix-master-key.backup

Move it off this machine. A backup that lives on the machine it protects is not a backup.