spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

Ops console for host incidents & security alerts (API + Angular page), dev-ops/admin only #297

New Issue
Open
opened 2026-06-30 16:42:04 +00:00 by spikerj · 5 comments
spikerj commented 2026-06-30 16:42:04 +00:00
Owner
Copy Link
Copy Source

Problem

The host-level remediation + security monitoring capability (#287) is live, but its output is not browsable in-app. Today the only ways to see what the swarm is reporting are:

  • Email to team@spikersoft.com (alert-on-threshold only — MinSeverity=high, with cooldowns, so most signal never produces a message).
  • Seq (the security console) for SecurityAlert log queries.
  • MongoDB system_incidents collection for the SystemIncident read-model — but nothing renders it.

There is no SpikerSoft.Api endpoint and no Angular page for any of this (confirmed: none of SystemIncident / SecurityAlert / HostJournalEvent are referenced in SpikerSoft.Api). Operators can't answer "what's unhealthy right now / what fired overnight" without shelling into Seq/Mongo.

Proposal

A read-only Ops console built in the existing idiom:

Backend (spikersoft-backend)

  • New read service + controller (e.g. api/ops/incidents, api/ops/incidents/{id}, summary/count endpoints) over the existing system_incidents Mongo read-model (MongoIncidentStore). Filter by node, kind, status (Detected/Remediating/Resolved/AlertedOnly), severity, time range.
  • Security alerts — DECIDED: persist to Mongo. Add a lightweight security_alerts read-model written by SpikerSoft.EventHandlers.SecurityMonitor (it currently only logs SecurityAlerts to Seq + keeps detector state in Redis; it has no Mongo dependency today). Persist alongside the existing Seq log + security.alert.{node} republish — do not remove those. Expose it via api/ops/security-alerts with node/category/severity/time filters. Mirror the MongoIncidentStore conventions (deterministic IDs / upsert + occurrence counter where it makes sense, configurable collection name, fail-open so a Mongo blip never blocks ingestion). This adds a MongoDB dependency to the SecurityMonitor stack — wire it into its docker-stack.yml (it already runs on SERVER, where Mongo is reachable) and health checks.
  • Read-only for v1: no remediation-command dispatch from the UI (the write/actuation path stays server-side, behind its existing toggles).

Frontend (spikersoft-angular)

  • New page (under /devops or /admin/ops) listing incidents + security alerts with the filters above and a detail view (occurrence count, signals, status timeline, last remediation result).
  • Menu entry surfaced only to authorized roles.

Authorization (hard requirement)

This surface must require the dev-ops OR admin role. staff is NOT sufficient — unlike the existing /admin/* tools which use the IsAdminOrStaff() / roles: ["Admin","Staff"] pattern.

  • Backend: add a stricter check/policy (e.g. IsDevOpsOrAdmin() or a named "DevOpsOrAdmin" authorization policy) and apply it to the new controller — do not reuse IsAdminOrStaff().
  • Frontend: guard the route with RoleGuard + data: { roles: ["dev-ops", "admin", "Admin"] } and gate the menu entry with hasAnyRole(...).

Dependency: the dev-ops realm role does not exist yet, and the role taxonomy is currently ad-hoc. This depends on the staff-role specialization work (#298). Until dev-ops exists, gate to admin only and wire dev-ops in once the role is defined.

Acceptance criteria

  • Read-only API over system_incidents with node/kind/status/severity/time filters.
  • SecurityMonitor persists a security_alerts read-model to Mongo (in addition to Seq), exposed via a read-only api/ops/security-alerts with node/category/severity/time filters.
  • Mongo persistence in SecurityMonitor is fail-open and its stack/health checks updated for the new MongoDB dependency.
  • Angular ops page (list + detail) consuming both feeds.
  • Access denied (403 / redirect) for staff-only and unauthenticated users; allowed for dev-ops and admin.
  • No remediation actions are dispatchable from the UI in v1.
  • Backend authorization does not fall back to IsAdminOrStaff().

Out of scope (v1)

  • Triggering/approving remediation commands from the UI.
  • Live streaming/websocket updates (poll/refresh is fine to start).
  • Backfilling historical alerts that only ever hit Seq (the Mongo read-model starts accumulating from deploy).

Related: #287 (capability), #298 (staff-role specialization / dev-ops role).

## Problem The host-level remediation + security monitoring capability (#287) is live, but its output is **not browsable in-app**. Today the only ways to see what the swarm is reporting are: - **Email** to `team@spikersoft.com` (alert-on-threshold only — `MinSeverity=high`, with cooldowns, so most signal never produces a message). - **Seq** (the security console) for `SecurityAlert` log queries. - **MongoDB** `system_incidents` collection for the `SystemIncident` read-model — but nothing renders it. There is **no `SpikerSoft.Api` endpoint and no Angular page** for any of this (confirmed: none of `SystemIncident` / `SecurityAlert` / `HostJournalEvent` are referenced in `SpikerSoft.Api`). Operators can't answer "what's unhealthy right now / what fired overnight" without shelling into Seq/Mongo. ## Proposal A read-only **Ops console** built in the existing idiom: **Backend (`spikersoft-backend`)** - New read service + controller (e.g. `api/ops/incidents`, `api/ops/incidents/{id}`, summary/count endpoints) over the existing `system_incidents` Mongo read-model (`MongoIncidentStore`). Filter by node, kind, status (`Detected/Remediating/Resolved/AlertedOnly`), severity, time range. - **Security alerts — DECIDED: persist to Mongo.** Add a lightweight `security_alerts` read-model written by `SpikerSoft.EventHandlers.SecurityMonitor` (it currently only logs `SecurityAlert`s to Seq + keeps detector state in Redis; it has **no** Mongo dependency today). Persist alongside the existing Seq log + `security.alert.{node}` republish — do not remove those. Expose it via `api/ops/security-alerts` with node/category/severity/time filters. Mirror the `MongoIncidentStore` conventions (deterministic IDs / upsert + occurrence counter where it makes sense, configurable collection name, fail-open so a Mongo blip never blocks ingestion). This adds a MongoDB dependency to the SecurityMonitor stack — wire it into its `docker-stack.yml` (it already runs on `SERVER`, where Mongo is reachable) and health checks. - Read-only for v1: **no** remediation-command dispatch from the UI (the write/actuation path stays server-side, behind its existing toggles). **Frontend (`spikersoft-angular`)** - New page (under `/devops` or `/admin/ops`) listing incidents + security alerts with the filters above and a detail view (occurrence count, signals, status timeline, last remediation result). - Menu entry surfaced only to authorized roles. ## Authorization (hard requirement) This surface must require the **`dev-ops`** OR **`admin`** role. **`staff` is NOT sufficient** — unlike the existing `/admin/*` tools which use the `IsAdminOrStaff()` / `roles: ["Admin","Staff"]` pattern. - Backend: add a stricter check/policy (e.g. `IsDevOpsOrAdmin()` or a named `"DevOpsOrAdmin"` authorization policy) and apply it to the new controller — do **not** reuse `IsAdminOrStaff()`. - Frontend: guard the route with `RoleGuard` + `data: { roles: ["dev-ops", "admin", "Admin"] }` and gate the menu entry with `hasAnyRole(...)`. **Dependency:** the `dev-ops` realm role does not exist yet, and the role taxonomy is currently ad-hoc. This depends on the staff-role specialization work (#298). Until `dev-ops` exists, gate to `admin` only and wire `dev-ops` in once the role is defined. ## Acceptance criteria - [ ] Read-only API over `system_incidents` with node/kind/status/severity/time filters. - [ ] `SecurityMonitor` persists a `security_alerts` read-model to Mongo (in addition to Seq), exposed via a read-only `api/ops/security-alerts` with node/category/severity/time filters. - [ ] Mongo persistence in `SecurityMonitor` is fail-open and its stack/health checks updated for the new MongoDB dependency. - [ ] Angular ops page (list + detail) consuming both feeds. - [ ] Access denied (403 / redirect) for `staff`-only and unauthenticated users; allowed for `dev-ops` and `admin`. - [ ] No remediation actions are dispatchable from the UI in v1. - [ ] Backend authorization does not fall back to `IsAdminOrStaff()`. ## Out of scope (v1) - Triggering/approving remediation commands from the UI. - Live streaming/websocket updates (poll/refresh is fine to start). - Backfilling historical alerts that only ever hit Seq (the Mongo read-model starts accumulating from deploy). Related: #287 (capability), #298 (staff-role specialization / `dev-ops` role).
spikerj added the enhancement label 2026-06-30 16:42:04 +00:00
spikerj commented 2026-06-30 16:42:42 +00:00
Author
Owner
Copy Link
Copy Source

Companion / dependency: #298 (specialize the keycloak-admin library toward SpikerSoft staff roles and define the dev-ops role). Until dev-ops exists, gate this console to admin only and add dev-ops once #298 lands.

Companion / dependency: #298 (specialize the `keycloak-admin` library toward SpikerSoft staff roles and define the `dev-ops` role). Until `dev-ops` exists, gate this console to `admin` only and add `dev-ops` once #298 lands.
spikerj commented 2026-06-30 17:24:43 +00:00
Author
Owner
Copy Link
Copy Source

Decision locked: persist security alerts to Mongo (option a). SecurityMonitor will write a fail-open security_alerts read-model (mirroring MongoIncidentStore conventions) in addition to its existing Seq log + RabbitMQ republish, and the ops API will read from it via api/ops/security-alerts. This adds a MongoDB dependency to the SecurityMonitor stack. Updated the body and acceptance criteria accordingly; the Seq-proxy alternative is dropped.

**Decision locked: persist security alerts to Mongo** (option a). `SecurityMonitor` will write a fail-open `security_alerts` read-model (mirroring `MongoIncidentStore` conventions) in addition to its existing Seq log + RabbitMQ republish, and the ops API will read from it via `api/ops/security-alerts`. This adds a MongoDB dependency to the SecurityMonitor stack. Updated the body and acceptance criteria accordingly; the Seq-proxy alternative is dropped.
spikerj commented 2026-07-05 08:53:14 +00:00
Author
Owner
Copy Link
Copy Source

Deliverable 1 (read-only incidents API) merged in spikersoft-backend PR #93. api/ops/incidents (paged list, filters node/kind/status/severity/time), .../{id} (detail), .../summary (counts by status/severity/node) over the existing system_incidents read-model — so the auto-remediation incidents (#376) are now browsable via API instead of Seq/Mongo-only.

  • No heavy-dep leak: the API doesn't reference the worker Infrastructure assembly where SystemIncident lives; added a Bson read DTO + raw-IMongoClient read service (fail-open), reusing the writer's Remediation:IncidentsCollection.
  • Authz per the hard requirement: OpsAuthorization.IsDevOpsOrAdmin — admin-only, does NOT reuse IsAdminOrStaff() (staff denied, proven by tests). One-line seam to add the dev-ops role once #298 defines it — marked // #298.
  • 31 new tests (staff/anon/plain-user denied across all three endpoints; admin allowed; filters/paging/sort/summary).

Still open for #297:

  • Deliverable 2 — SecurityMonitor persists security_alerts to Mongo + api/ops/security-alerts. Deferred because it adds a MongoDB dependency to that worker + its docker-stack.yml + health checks (has an infra-repo change).
  • Angular ops page (list + detail, dev-ops/admin route guard) — separate angular PR.
  • Wire the real dev-ops role in once #298 lands.

Ticket stays open.

**Deliverable 1 (read-only incidents API) merged** in spikersoft-backend PR #93. `api/ops/incidents` (paged list, filters node/kind/status/severity/time), `.../{id}` (detail), `.../summary` (counts by status/severity/node) over the existing `system_incidents` read-model — so the auto-remediation incidents (#376) are now browsable via API instead of Seq/Mongo-only. - **No heavy-dep leak:** the API doesn't reference the worker `Infrastructure` assembly where `SystemIncident` lives; added a Bson read DTO + raw-`IMongoClient` read service (fail-open), reusing the writer's `Remediation:IncidentsCollection`. - **Authz per the hard requirement:** `OpsAuthorization.IsDevOpsOrAdmin` — **admin-only**, does NOT reuse `IsAdminOrStaff()` (staff denied, proven by tests). One-line seam to add the `dev-ops` role once #298 defines it — marked `// #298`. - 31 new tests (staff/anon/plain-user denied across all three endpoints; admin allowed; filters/paging/sort/summary). **Still open for #297:** - **Deliverable 2** — `SecurityMonitor` persists `security_alerts` to Mongo + `api/ops/security-alerts`. Deferred because it adds a MongoDB dependency to that worker + its `docker-stack.yml` + health checks (has an infra-repo change). - **Angular ops page** (list + detail, `dev-ops`/admin route guard) — separate angular PR. - Wire the real `dev-ops` role in once #298 lands. Ticket stays open.
spikerj commented 2026-07-05 09:32:54 +00:00
Author
Owner
Copy Link
Copy Source

Deliverable 2 staged as spikersoft-backend PR #95 — NOT merged, needs your call (flagging a latent prod-safety issue it uncovered).

⚠️ Discovery: the write-side already landed on master as commit bfbff89 (Cursor-authored, Jul 1) ungated / always-on — it unconditionally does .WithMongoDB(), adds a Mongo health check to /healthz, and always persists. But spikersoft-security-monitor/docker-stack.yml has no mongo network (only rabbitmq/redis/seq-attachable/jaeger). So a deploy from that commit creates an IMongoClient that can't reach mongo-router:27017 (connection-failure noise) and its Mongo health check fails → /healthz unhealthy → swarm restart risk. Worth checking whether SecurityMonitor has been flapping since Jul 1 (its healthz is VPN-internal on SERVER:8094, so I can't reach it from here).

PR #95 does two things:

  1. Fix: flag-gates the write-side OFF by default (SecurityMonitor:PersistAlertsToMongo) — OFF ⇒ no Mongo client / health check / persistence, byte-identical to pre-bfbff89 (matches the actual infra). Merging + deploying this returns SecurityMonitor to a safe state.
  2. Feature: adds the read-only api/ops/security-alerts endpoint (admin-gated, mirrors the #93 incidents API), fail-open to empty until data exists.

Your decision (two valid paths):

  • (a) Accept default-off now (fixes the risk immediately) → later add the mongo network to the SecurityMonitor stack + flip PersistAlertsToMongo=true to enable persistence.
  • (b) Fix-forward → add the mongo network first, then this can default on.

I left it unmerged because it reverses bfbff89's always-on behavior, and that's your architectural call (plus you may want eyes on how an always-on Mongo dep reached a worker with no mongo network). 113/113 filtered tests green; off-path proven to create no Mongo client. Ticket stays open.

**Deliverable 2 staged as spikersoft-backend PR #95 — NOT merged, needs your call** (flagging a latent prod-safety issue it uncovered). ⚠️ **Discovery:** the write-side already landed on master as commit `bfbff89` (Cursor-authored, Jul 1) **ungated / always-on** — it unconditionally does `.WithMongoDB()`, adds a Mongo health check to `/healthz`, and always persists. But `spikersoft-security-monitor/docker-stack.yml` has **no `mongo` network** (only rabbitmq/redis/seq-attachable/jaeger). So a deploy from that commit creates an `IMongoClient` that can't reach `mongo-router:27017` (connection-failure noise) **and its Mongo health check fails → `/healthz` unhealthy → swarm restart risk.** Worth checking whether SecurityMonitor has been flapping since Jul 1 (its healthz is VPN-internal on SERVER:8094, so I can't reach it from here). **PR #95 does two things:** 1. **Fix:** flag-gates the write-side OFF by default (`SecurityMonitor:PersistAlertsToMongo`) — OFF ⇒ no Mongo client / health check / persistence, byte-identical to pre-`bfbff89` (matches the actual infra). Merging + deploying this returns SecurityMonitor to a safe state. 2. **Feature:** adds the read-only `api/ops/security-alerts` endpoint (admin-gated, mirrors the #93 incidents API), fail-open to empty until data exists. **Your decision (two valid paths):** - **(a) Accept default-off now** (fixes the risk immediately) → later add the `mongo` network to the SecurityMonitor stack + flip `PersistAlertsToMongo=true` to enable persistence. - **(b) Fix-forward** → add the `mongo` network first, then this can default on. I left it unmerged because it reverses `bfbff89`'s always-on behavior, and that's your architectural call (plus you may want eyes on how an always-on Mongo dep reached a worker with no mongo network). 113/113 filtered tests green; off-path proven to create no Mongo client. Ticket stays open.
spikerj commented 2026-07-22 02:55:31 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): backend fully merged (PR #48 write side, #93 incidents read API, #95 flag-gated persistence + security-alerts API). REMAINING: the Angular ops-console page — no trace on angular master.

Board-sweep status (2026-07-22): backend fully merged (PR #48 write side, #93 incidents read API, #95 flag-gated persistence + security-alerts API). REMAINING: the Angular ops-console page — no trace on angular master.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#297