Fix web UI hangs, stalls and lost config edits #1

Merged

aluminum-ice merged 2 commits from fix/web-ui-hangs into main 2026-10-26 07:14:41 +00:00

Conversation 0 Commits 2 Files changed 18 +732 -68

aluminum-ice commented 2026-10-26 07:08:13 +00:00

Owner

Copy link

What this PR does

Fixes the ways the device web UI could hang, stall or wedge under real-world timing: daemon restarts that outlast the page's wait, apply calls timing out on changes that still land, threads pinned by stalled clients or hung plugins, and a lost-update race on config.toml. It also points CLAUDE.md at this PR template.

Checklist

• Lint passes — make lint (ansible --syntax-check, yamllint, shellcheck --severity=error). The Lint workflow runs this on the PR.
• Tests pass — make test (the kestrel_pi pytest suite). The Test workflow runs this on the PR.
• No secrets, no personal information. This is a public repository. The diff contains no real account passwords or hashes, Wi-Fi passphrases, private keys, API tokens, .env files, real names, personal emails, MAC addresses, device serials, or GPS coordinates — placeholders only.
• Docs updated in the same PR, if this changes something an operator sees or configures: the README package table, the relevant page under docs/, and the on-device operator guide (GUIDE_MD in kestrel_pi/userguide.py) where it applies.
• Commit messages are short and imperative.

Release impact

• App-layer-safe — changes only files the in-place update engine carries: the kestrel_pi package, plugins/*.py, systemd units, the journald drop-in, a lib script already in the engine's LIB_SCRIPTS, or the CLI.
• Reflash-class — touches a build input the engine does not carry onto a running device: apt or pip packages, boot options / config.txt / cmdline, kernel modules, the base image or partition size, builder/config.json, builder/config.yml, the Makefile, .forgejo/workflows/, a shipped config default (config.toml / default.conf), or a brand-new lib script.

Changed install targets: the kestrel_pi package, config-set and plugins-ctl (both already in LIB_SCRIPTS), the CLI, and two units (kestrel-daemon.service, kestrel-plugins-apply@.service; the engine runs daemon-reload). CLAUDE.md and docs/ are not installed on the device. update_kind=auto picks app-layer correctly.

Notes for review

Changes

• Daemon stop is bounded. All plugins' on_unload calls run concurrently within one 25 s budget (Loader.unload_all), and a hung one is logged and abandoned. main() exits with os._exit after the clean stop, so hook-pool threads stuck in a hook are no longer joined. TimeoutStopSec=40, and the restart-wait page polls for about 2 minutes (was about 40 s).
• Apply timeouts nest: CLI 40 s < helper RuntimeMaxSec 50 s < client 55 s. The client has one overall deadline and returns on the one-line reply, instead of waiting for the helper to exit.
• capture disable / agent disable write config first, then stop with --no-block. Before, a helper killed mid-stop could leave capture stopped but still enabled. BT re-pair restarts onboarding with --no-block.
• Web server:
  • 15 s socket timeout per connection.
  • While a plugin's overrun on_webhook call is still running, further calls to that plugin get a 503 instead of a new thread.
  • 5 s interface-address cache (failures not cached), and concurrent systemctl is-active probes share one call.
  • config.toml is re-parsed only when its stat changes (callers get a copy), and /networks is cached until a store file changes.
  • A listener is released only after two consecutive polls without its address.
  • Restart-wait probes are aborted after 5 s.
• Config writes are serialized with an exclusive flock on .config.toml.lock beside the config, in config-set.py, which every config writer goes through.

Behaviour changes

• kestrel capture disable / agent disable return before the service has finished stopping. The update engine is unaffected, because it uses plain systemctl stop.
• In a badly stalled case, a capture/agent toggle can wait up to 55 s for its reply, instead of failing at 20 s while the change still landed.
• The dashboard address list can be up to 5 s stale.
• A new /etc/kestrel/.config.toml.lock file is created on the first config write.
• Plugins unload concurrently, and services are withdrawn after all unloads finish. No shipped plugin's on_unload uses services or context.apply.
• Exit skips atexit hooks (e.g. gpiozero/lgpio cleanup). The kernel releases the GPIO handles, and systemd removes /run/kestrel on stop.

Testing

• 739 passed, 7 skipped. New tests cover a hung on_unload, the apply client's line return and overall deadline, the stuck-webhook 503, the stalled-connection timeout against real werkzeug, the caches, the reconcile debounce and the config lock.
• I confirmed the connection-timeout test fails without the fix.
• Without the lock, 12 parallel whitelist adds kept 1 entry; with it, all 12.
• The restart-wait JS, run under node with hung fetches: the old script stalls, the new one recovers.
• Not yet run on a device.

🤖 Generated with Claude Code

## What this PR does Fixes the ways the device web UI could hang, stall or wedge under real-world timing: daemon restarts that outlast the page's wait, apply calls timing out on changes that still land, threads pinned by stalled clients or hung plugins, and a lost-update race on `config.toml`. It also points CLAUDE.md at this PR template. ## Checklist - [x] **Lint passes** — `make lint` (ansible `--syntax-check`, `yamllint`, `shellcheck --severity=error`). The **Lint** workflow runs this on the PR. - [x] **Tests pass** — `make test` (the `kestrel_pi` pytest suite). The **Test** workflow runs this on the PR. - [x] **No secrets, no personal information.** This is a public repository. The diff contains no real account passwords or hashes, Wi-Fi passphrases, private keys, API tokens, `.env` files, real names, personal emails, MAC addresses, device serials, or GPS coordinates — placeholders only. - [x] **Docs updated in the same PR**, if this changes something an operator sees or configures: the README package table, the relevant page under `docs/`, and the on-device operator guide (`GUIDE_MD` in `kestrel_pi/userguide.py`) where it applies. - [x] Commit messages are short and imperative. ## Release impact - [x] **App-layer-safe** — changes only files the in-place update engine carries: the `kestrel_pi` package, `plugins/*.py`, systemd units, the journald drop-in, a lib script already in the engine's `LIB_SCRIPTS`, or the CLI. - [ ] **Reflash-class** — touches a build input the engine does not carry onto a running device: apt or pip packages, boot options / `config.txt` / `cmdline`, kernel modules, the base image or partition size, `builder/config.json`, `builder/config.yml`, the `Makefile`, `.forgejo/workflows/`, a shipped config default (`config.toml` / `default.conf`), or a brand-new lib script. Changed install targets: the `kestrel_pi` package, `config-set` and `plugins-ctl` (both already in `LIB_SCRIPTS`), the CLI, and two units (`kestrel-daemon.service`, `kestrel-plugins-apply@.service`; the engine runs `daemon-reload`). `CLAUDE.md` and `docs/` are not installed on the device. `update_kind=auto` picks app-layer correctly. ## Notes for review **Changes** - **Daemon stop is bounded.** All plugins' `on_unload` calls run concurrently within one 25 s budget (`Loader.unload_all`), and a hung one is logged and abandoned. `main()` exits with `os._exit` after the clean stop, so hook-pool threads stuck in a hook are no longer joined. `TimeoutStopSec=40`, and the restart-wait page polls for about 2 minutes (was about 40 s). - **Apply timeouts nest:** CLI 40 s < helper `RuntimeMaxSec` 50 s < client 55 s. The client has one overall deadline and returns on the one-line reply, instead of waiting for the helper to exit. - **`capture disable` / `agent disable` write config first, then stop with `--no-block`.** Before, a helper killed mid-stop could leave capture stopped but still enabled. BT re-pair restarts onboarding with `--no-block`. - **Web server:** - 15 s socket timeout per connection. - While a plugin's overrun `on_webhook` call is still running, further calls to that plugin get a 503 instead of a new thread. - 5 s interface-address cache (failures not cached), and concurrent `systemctl is-active` probes share one call. - `config.toml` is re-parsed only when its stat changes (callers get a copy), and `/networks` is cached until a store file changes. - A listener is released only after two consecutive polls without its address. - Restart-wait probes are aborted after 5 s. - **Config writes are serialized** with an exclusive `flock` on `.config.toml.lock` beside the config, in `config-set.py`, which every config writer goes through. **Behaviour changes** - `kestrel capture disable` / `agent disable` return before the service has finished stopping. The update engine is unaffected, because it uses plain `systemctl stop`. - In a badly stalled case, a capture/agent toggle can wait up to 55 s for its reply, instead of failing at 20 s while the change still landed. - The dashboard address list can be up to 5 s stale. - A new `/etc/kestrel/.config.toml.lock` file is created on the first config write. - Plugins unload concurrently, and services are withdrawn after all unloads finish. No shipped plugin's `on_unload` uses services or `context.apply`. - Exit skips atexit hooks (e.g. gpiozero/lgpio cleanup). The kernel releases the GPIO handles, and systemd removes `/run/kestrel` on stop. **Testing** - 739 passed, 7 skipped. New tests cover a hung `on_unload`, the apply client's line return and overall deadline, the stuck-webhook 503, the stalled-connection timeout against real werkzeug, the caches, the reconcile debounce and the config lock. - I confirmed the connection-timeout test fails without the fix. - Without the lock, 12 parallel whitelist adds kept 1 entry; with it, all 12. - The restart-wait JS, run under node with hung fetches: the old script stalls, the new one recovers. - **Not yet run on a device.** 🤖 Generated with [Claude Code](https://claude.com/claude-code)

aluminum-ice added 1 commit 2026-10-26 07:08:13 +00:00

Fix web UI hangs, stalls and lost config edits ... 40a3e89bd2

- Bound daemon stop: plugins unload concurrently within 25s, exit skips
  joining stuck hook threads, TimeoutStopSec=40; restart wait ~2 min
- Nest apply timeouts (CLI 40s < helper 50s < client 55s); client returns
  on the reply line with one overall deadline
- capture/agent disable write config first and stop with --no-block;
  bt re-pair restarts onboarding with --no-block
- 15s socket timeout on web connections
- Cap a stuck plugin on_webhook at one leaked thread (503 while busy)
- Cache interface addresses, config.toml parse and captured networks;
  share concurrent systemctl probes
- Abort a stuck restart-wait probe after 5s
- Keep a web listener through a single failed address lookup
- Serialize config.toml writes with a lock file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aluminum-ice added 1 commit 2026-10-26 07:12:37 +00:00

Point CLAUDE.md at the PR template ... 96072e381e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aluminum-ice merged commit 7f5a10c86d into main 2026-10-26 07:14:41 +00:00

aluminum-ice deleted branch fix/web-ui-hangs 2026-10-26 07:14:41 +00:00

aluminum-ice referenced this pull request from a commit 2026-10-26 07:14:42 +00:00

Merge pull request 'Fix web UI hangs, stalls and lost config edits' (#1) from fix/web-ui-hangs into main

aluminum-ice referenced this pull request 2026-10-26 08:09:17 +00:00

Add beta update channel and harden in-place updates #2

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

aluminum-ice/kestrel!1

No description provided.