[Infra][SonarQube] CE fails extracting LARGE uploaded reports: PGbytea ArrayIndexOutOfBoundsException reading ce_task_input (root cause of the old #344 crash) #642

New Issue

2026-07-17T06:32:46Z

spikerj commented

2026-07-17 06:32:46 +00:00

Split out of #632, where the full diagnosis lives. Summary of evidence:

Backend master scan (CI container, SonarScanner 10.4.1, WITH test coverage → large multi-part report) uploaded successfully at 06:05 on 2026-07-17; the CE task FAILED in 257ms: java.lang.ArrayIndexOutOfBoundsException: Index 58 out of bounds for length 55 at org.postgresql.util.PGbytea.getHex → ExtractReportStep.appendPart (task f21b34f6-9518-4874-a0a0-c9cbb2d9c3a1, stacktrace on #632).
A locally-run scan (SonarScanner 11.2.1, no coverage → smaller report) uploaded minutes later processed in ~45s: analysis 2026-07-17T06:29 landed and re-baselined the project. So the CLIENT chain is fine; the failure is server-side and correlates with report size/parts (learn.spikersoft.com's smaller reports have never hit it).
The exception means the stored bytea reads back malformed from ce_task_input — pgjdbc's escape-format decoder chokes mid-buffer. Suspects: pgjdbc/postgres 15 interaction on multi-hundred-MB bytea rows, or blob corruption at insert (disk pressure on SERVER?).
This is the concrete root cause of the historical #344 'CE Task finished abnormally' mystery that motivated the (since removed) continue-on-error.

Next steps: (1) the already-queued backend CI scan (run 8645, dispatch) gives the decisive second data point for container/coverage uploads — if it fails identically, it's deterministic for large reports; (2) SERVER-side check via one-shot swarm job on the sonarqube stack's postgres: SHOW bytea_output;, SELECT uuid, octet_length(input_data) FROM ce_task_input;, disk state of /home/joey/docker/sonarqube; (3) consider bumping the SonarQube image (the bundled pgjdbc) — the stack pins sonar-scanner-datacenter:2026.2.0.121184.

Until resolved, backend scans WITH coverage may fail CE-side; scans keep failing LOUDLY (post-#632 behavior), so no silent rot.

Split out of #632, where the full diagnosis lives. Summary of evidence: - Backend master scan (CI container, SonarScanner 10.4.1, WITH test coverage → large multi-part report) uploaded successfully at 06:05 on 2026-07-17; the CE task FAILED in 257ms: `java.lang.ArrayIndexOutOfBoundsException: Index 58 out of bounds for length 55` at `org.postgresql.util.PGbytea.getHex` → `ExtractReportStep.appendPart` (task f21b34f6-9518-4874-a0a0-c9cbb2d9c3a1, stacktrace on #632). - A locally-run scan (SonarScanner 11.2.1, no coverage → smaller report) uploaded minutes later processed in ~45s: analysis 2026-07-17T06:29 landed and re-baselined the project. So the CLIENT chain is fine; the failure is server-side and correlates with report size/parts (learn.spikersoft.com's smaller reports have never hit it). - The exception means the stored bytea reads back malformed from `ce_task_input` — pgjdbc's escape-format decoder chokes mid-buffer. Suspects: pgjdbc/postgres 15 interaction on multi-hundred-MB bytea rows, or blob corruption at insert (disk pressure on SERVER?). - This is the concrete root cause of the historical #344 'CE Task finished abnormally' mystery that motivated the (since removed) continue-on-error. **Next steps:** (1) the already-queued backend CI scan (run 8645, dispatch) gives the decisive second data point for container/coverage uploads — if it fails identically, it's deterministic for large reports; (2) SERVER-side check via one-shot swarm job on the sonarqube stack's postgres: `SHOW bytea_output;`, `SELECT uuid, octet_length(input_data) FROM ce_task_input;`, disk state of /home/joey/docker/sonarqube; (3) consider bumping the SonarQube image (the bundled pgjdbc) — the stack pins sonar-scanner-datacenter:2026.2.0.121184. Until resolved, backend scans WITH coverage may fail CE-side; scans keep failing LOUDLY (post-#632 behavior), so no silent rot.

spikerj referenced this issue

2026-07-17 06:32:46 +00:00

[CI][Backend] sonar-scan has uploaded nothing since 2026-07-14 — scanner gets a wrong SonarQube host and continue-on-error hides the failure #632

spikerj commented

2026-07-17 06:48:34 +00:00

Reproducibility verdict: NOT deterministic. The dispatched CI run 8645 — same container, same scanner 10.4.1, same large coverage-bearing report shape as the 06:05 failure — completed end-to-end and its CE task processed in 23s (analysis + 55.2% coverage now live on the dashboard). Three uploads total today: container 06:05 FAILED (PGbytea), local 06:30 SUCCESS, container 06:45 SUCCESS.

Downgrading this from 'blocking' to 'intermittent, monitor + forensics': the 06:05 blob either landed corrupt (transient disk/IO condition on SERVER during insert?) or hit a rare pgjdbc edge. Suggested when convenient, not urgent: the one-shot swarm job forensics from the ticket body (SHOW bytea_output; octet_length over ce_task_input; dmesg/df around 06:05 on SERVER), and consider the SonarQube image/pgjdbc bump. Failures stay loud post-#632, so a recurrence will be visible on the workflow immediately — if it recurs, escalate this back to active investigation.

Reproducibility verdict: NOT deterministic. The dispatched CI run 8645 — same container, same scanner 10.4.1, same large coverage-bearing report shape as the 06:05 failure — completed end-to-end and its CE task processed in 23s (analysis + 55.2% coverage now live on the dashboard). Three uploads total today: container 06:05 FAILED (PGbytea), local 06:30 SUCCESS, container 06:45 SUCCESS. Downgrading this from 'blocking' to 'intermittent, monitor + forensics': the 06:05 blob either landed corrupt (transient disk/IO condition on SERVER during insert?) or hit a rare pgjdbc edge. Suggested when convenient, not urgent: the one-shot swarm job forensics from the ticket body (SHOW bytea_output; octet_length over ce_task_input; dmesg/df around 06:05 on SERVER), and consider the SonarQube image/pgjdbc bump. Failures stay loud post-#632, so a recurrence will be visible on the workflow immediately — if it recurs, escalate this back to active investigation.

spikerj commented

2026-07-17 07:37:02 +00:00

Monitoring update: five consecutive CE SUCCESSes since the single 06:05 failure — 06:30 (local), 06:45 (CI container + coverage), 06:58 x2 and 07:31 (post-merge master scans, all container path). The one-off-corruption theory holds; no recurrence across the exact report shape that failed. Keeping this open for the low-priority SERVER-side forensics only.

spikerj commented

2026-07-29 17:08:14 +00:00

Audited against origin/master — NOT DONE in the one part git can settle; the rest is live-state.

The image pin has not moved. sonarqube/docker-stack.yml:31 still reads image: git.spikersoft.com/spikerj/sonar-scanner-datacenter:2026.2.0.121184 — the exact version this ticket's step 3 proposed moving off. Both Postgres services are postgres:15 (:9, :72), also unchanged.

Everything decisive here is live-state and leaves no repo trace, so I'm explicitly not inferring anything from git silence:

CI run 8645's outcome
SHOW bytea_output; and the ce_task_input row sizes on the SERVER Postgres
disk state of /home/joey/docker/sonarqube

One thing worth weighing before doing the image bump. This ticket's root cause — a PG bytea ArrayIndexOutOfBoundsException reading ce_task_input on large uploaded reports — is a size-driven failure on the SERVER Postgres. Two other open tickets touch the same machine's storage and memory:

#503 — confirmed bad RAM on SERVER, mitigated with a GRUB_BADRAM hole; the DIMM was never replaced and memtest was never run to map the full extent.
#482 / #561 — Keycloak Postgres pg_toast_2619 corruption on that same host, with the repair never run.

Three Postgres/memory faults on one machine with known-bad memory is a pattern worth checking before treating this as a SonarQube version bug. The SHOW data_checksums; step already listed on #482 would help establish whether these share a cause — if they do, the image bump won't fix anything and the DIMM replacement becomes the actual blocker for all three.

Remaining: the live diagnostics above, then either the image bump or the hardware path depending on what they show.

Audited against `origin/master` — **NOT DONE in the one part git can settle; the rest is live-state.** **The image pin has not moved.** `sonarqube/docker-stack.yml:31` still reads `image: git.spikersoft.com/spikerj/sonar-scanner-datacenter:2026.2.0.121184` — the exact version this ticket's step 3 proposed moving off. Both Postgres services are `postgres:15` (`:9`, `:72`), also unchanged. **Everything decisive here is live-state and leaves no repo trace**, so I'm explicitly not inferring anything from git silence: - CI run 8645's outcome - `SHOW bytea_output;` and the `ce_task_input` row sizes on the SERVER Postgres - disk state of `/home/joey/docker/sonarqube` **One thing worth weighing before doing the image bump.** This ticket's root cause — a `PG bytea ArrayIndexOutOfBoundsException` reading `ce_task_input` on **large** uploaded reports — is a size-driven failure on the SERVER Postgres. Two other open tickets touch the same machine's storage and memory: - **#503** — confirmed bad RAM on SERVER, mitigated with a `GRUB_BADRAM` hole; the DIMM was never replaced and memtest was never run to map the full extent. - **#482 / #561** — Keycloak Postgres `pg_toast_2619` corruption on that same host, with the repair never run. Three Postgres/memory faults on one machine with known-bad memory is a pattern worth checking before treating this as a SonarQube version bug. The `SHOW data_checksums;` step already listed on #482 would help establish whether these share a cause — if they do, the image bump won't fix anything and the DIMM replacement becomes the actual blocker for all three. **Remaining:** the live diagnostics above, then either the image bump or the hardware path depending on what they show.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#642