Monitoring

This page covers what to watch on a running server: the numbers the System page shows and what each of them actually means, the HTTP endpoints an external monitoring system should poll, and a table of what is worth waking someone for.

ETTIX measures the machine itself by reading /proc, /sys and the filesystems directly. There is no Prometheus exporter, no agent to install and no third-party metrics library in the binary. Everything below is available from the administration interface and from two JSON endpoints, which is enough to drive a dashboard or an alerting system of your own.

The System page

Open the administration interface and choose System in the navigation. On a default installation that is http://127.0.0.1:8081/#/system. The page is built from two endpoints, GET /api/v1/system for the machine and this process, and GET /api/v1/disks for the volumes ETTIX writes to.

The collector samples every two seconds. Percentages and per-second figures are the difference between two samples, so for the first interval after start-up there is nothing to divide and the page says so rather than drawing a row of zeroes. In the JSON that state is rates_available: false, and interval_seconds is the period the rates were averaged over.

CPU

The first meter is total CPU across all cores over the last interval, and underneath it is the split that explains where the time went.

Both of the last two are routinely misread, and in opposite directions. High iowait is not CPU load — the processor is idle. It is a storage symptom, and on a recording server it usually means the DVR volume, not the encoder. Steal is not your fault and you cannot tune it away — the machine is slow because a neighbour is busy or the instance is oversubscribed, and the fix is a different instance or a different host, not a configuration change here. Sustained steal above a few per cent on a VM is the first thing to look at when the server feels slow while every other number says idle.

The per-core panel below the CPU meter shows each logical CPU separately. It is worth a glance, because a single thread pinned at 100 % on one core of sixteen is a 6 % average — a bottleneck that is invisible in the total.

Load average is the 1, 5 and 15 minute figures from the kernel, and the bar beside them is the 1-minute load as a percentage of the core count. That comparison is the whole point: a load average of 8 is comfortable on a 16-core machine and severe on a 4-core one, so the number is meaningless until it is divided by cores. One unit of load per core means the machine is fully committed with nothing queued; above that, work is waiting.

Memory

The Used meter is deliberately not "total minus free". It is MemTotal minus MemAvailable, which means reclaimable page cache is excluded from used. Linux fills otherwise idle memory with cached file contents; on a server writing DVR segments that cache can be most of the machine's RAM. Counting it as used would show 95 % memory pressure on a machine under no pressure at all, and would train you to ignore the number that matters. The cached_bytes figure is shown separately, as information rather than as consumption.

Swap is shown only when the machine has any. On a live media server, swap in use is a warning rather than a statistic: a page fault that has to reach the disk in the middle of a segment write is a delay measured in milliseconds, which viewers see. The interface starts colouring swap at 10 % used for that reason, far earlier than it colours anything else.

In a container, the totals are the cgroup's, not the host's, and source reads cgroup rather than meminfo. Reporting the host's 256 GB inside a 2 GB container would look healthy right up to the moment the kernel kills the process, so the limit wins and the host's figures are shown beside it as a separate line. Used inside a cgroup is memory.current minus the cgroup's own file cache, for the same reclaimability reason.

Storage and disk I/O

This panel has one row per volume ETTIX writes to — every configured DVR destination path, merged so that two streams recording to the same path are one volume with two destinations rather than two volumes each claiming the whole disk. Each row shows free space against capacity, a fill bar, the block device behind the path, what that device is reading and writing per second, its I/O busy percentage, and the rate this server's own recorders are writing there.

Two details are worth knowing:

• Capacity for a DVR destination comes from that destination's own storage backend — the same measurement the retention rule uses when it decides whether to delete old recordings. It is not a second, independent check. A disagreement between "the API says 40 GB free" and "retention thinks it is out of space" is not a thing that can happen here.
• recorded_bytes is what this server's recordings occupy, as the DVR index counts them. It is not capacity minus free space: other things live on the disk too, and the gap between the two is a useful sanity check when a volume fills up unexpectedly.

For an extra path that is not a DVR destination, free space is the unprivileged figure — the space an ordinary process can actually use. The blocks a filesystem holds back as a root-only reserve are excluded, because a recording cannot be written into them.

Block devices

The Block devices table is the raw per-device view from /proc/diskstats: bytes read and written per second, completed reads and writes per second, requests currently in flight, and I/O busy. Partitions are listed and marked as such; the totals count whole disks only, since adding a partition to the disk that contains it would count every byte twice. RAM disks and loop devices are excluded because they are noise on every dashboard.

busy_percent is the fraction of wall-clock time the device had at least one request in flight. It is exactly the figure iostat prints as %util, and it is the number that tells you a DVR disk is saturated before writes start failing. A single spinning disk sitting near 100 % is doing all it can, and any further recording load will queue.

Network

One row per interface, in and out, in bits per second rather than bytes, because that is the unit a link is sold in and the unit an encoder is configured in. Errors and dropped counters are the interface's own lifetime totals; what matters is whether they move, not what they are. The totals at the top exclude loopback: traffic from this process to itself would otherwise be counted twice and would make a quiet server look busy.

The ETTIX process

The last panel is this process rather than the machine.

Below these, when anything could not be measured, the page lists the source, the file and the reason. The same condition is written to the log once, as SYSSTAT_SOURCE_UNAVAILABLE, when it first appears and once again when it clears — not every two seconds for the life of the process.

Endpoints for external monitoring

/health and /ready, on the playback listener

These two live on the playback listener — port 8080 by default — and neither needs a credential, so a load balancer can reach them without being given one. The difference between them is the part that matters.

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/ready

Both endpoints are also present on the administration listener, which is convenient for a local check but is not where a load balancer should look.

The administration API

Everything else is on the administration listener, which defaults to 127.0.0.1:8081 and is loopback-only unless http.admin_allow_remote is set. All of these require a credential from http.admin_auth once one is configured — on the loopback default, where a fresh installation has none, they are open — presented either as an API key in an Authorization header or as a browser session cookie. All are read-only and all answer GET.

curl -s -H "Authorization: Bearer <id>.<secret>" \
  http://127.0.0.1:8081/api/v1/system

curl -s -H "Authorization: Bearer <id>.<secret>" \
  "http://127.0.0.1:8081/api/v1/events?level=warn&since=-15m&limit=100"

Wiring up an external check

What to alert on

The thresholds below are the ones the administration interface itself colours at, which makes an alert and the screen an operator opens next agree with each other. Where a sensible universal threshold does not exist, the column says what to look for instead.

The machine

Streams and storage

Licence and certificates

Events worth paging on

The event ring carries a code on the entries that matter, which makes /api/v1/events?code=… a precise alert source. These mean a customer is already affected or is about to be:

Logs

The event ring is bounded and lives in memory, so it is the right place to look during an incident and the wrong place to look for last Tuesday. For that, read the service log.

journalctl -u ettix-stream -f
journalctl -u ettix-stream -p warning --since -1h

Setting log.format to "json" produces one object per line, which is what a log shipper wants. The log formatter and the event ring share the same redaction function rather than a copy of it, so stream keys, SRT passphrases and administration passwords cannot reach one sink by escaping the other.

What is not here

There is no Prometheus /metrics endpoint, no OpenTelemetry exporter and no StatsD output in this build. An external monitoring system consumes the JSON endpoints above, or subscribes to /api/v1/events/stream. Historical retention is likewise outside the server: the System page shows the present, and the event ring holds a bounded recent window. If you need trends over weeks, poll /api/v1/system into your own time-series store.

When something is wrong rather than merely worth watching, continue to Troubleshooting.