Lantern

Appearance

Discord Notify Pipe Buffer ​

What is this? ​

A fix for a CI infrastructure bug: the Discord notification workflow dies on any push whose commit-message payload is too large for a Unix pipe buffer. ​

  • .github/workflows/discord-notify.yml's "Check if notification should be skipped" step piped large, dynamically sized values (a commit message, or a JSON array of commits) into head. head does not wait for a large writer to finish once it has what it wants: its early exit sends the writer SIGPIPE, and the step runs under set -Eeuo pipefail, so that SIGPIPE takes the whole step down with no output at all.
  • The trigger was the squash-merge commit for 130153ca (run 32734315417): a 94,631-byte, 1,978-line commit message, 22 branch commits concatenated by the squash, well past the 64KB pipe buffer.

What is the current state? ​

Fixed on branch feat/admin-and-merchant-portals, verified against the real oversized commit and a synthetic oversized-commit-count push. Not yet merged. ​

  • Three lines in the "Check if notification should be skipped" step had the identical shape: pipe a large value into head, which can exit before the writer is done. All three are fixed, not just the one the crash report pointed at.

The fix removes the pipe rather than raising the limit. ​

  • MSGS="$(printf '%s\n' "$HEAD_COMMIT_MESSAGE" | head -n 1)" became MSGS="${HEAD_COMMIT_MESSAGE%%$'\n'*}". Bash parameter expansion takes everything before the first newline with no subshell, no pipe, and no second process that could exit early.
  • MSGS="$(jq -r '.[].message | split("\n")[0]' <<<"$COMMITS_JSON" | head -n 50)" became MSGS="$(jq -r 'limit(50; .[].message | split("\n")[0])' <<<"$COMMITS_JSON")". jq's own limit() caps the generator's output to 50 values before anything leaves the process, instead of asking head to discard the rest on the far side of a pipe.
  • COMMITTERS="$(jq -r '...' <<<"$COMMITS_JSON" | head -n 50)" got the identical limit(50; ...) treatment.
  • The <<<"$COMMITS_JSON" herestring feeding jq was never the bug: bash materializes a herestring to a temp file before the reader starts, so there is no live writer process for it to starve. The bug was always on the output side: piping one process's stdout into another process that might stop reading early.

Proven against the real 94,631-byte commit message, not just asserted. ​

  • The pre-fix step body, run verbatim against git log -1 --format=%B 130153ca (94,631 bytes, 1,978 lines): exit 141 (128 + SIGPIPE), zero bytes of output, matching the CI run's own "died in 0.2s, no error output" symptom exactly.
  • The fixed step body, run against the same 94,631-byte message: exit 0, skip=false written correctly, MSGS correctly holds just the real subject line (docs(agents): agent asset hierarchy, plus a drift gate for the workflows layer (#952)).
  • The two jq | head -n 50 lines needed a different fixture to fail: a synthetic 3,000-commit COMMITS_JSON array whose unbounded jq output is about 273KB, against the roughly 4.5KB that head -n 50 actually keeps. Run against the pre-fix step body: exit 141, zero output, reproduced 5 of 5 runs. Against the fixed step body: exit 0, skip=false, MSGS and COMMITTERS each correctly capped at 50 lines, 5 of 5 runs.
  • Regression checks against the fixed step body: a small 3-commit push with [skip ci] in one subject line still sets skip=true with the right reason; a tag present only in a commit body, not the subject, still does not trigger skip (the subject-only rule from #348 still holds); a bot actor (dependabot[bot]) still sets skip=true.
  • Every reproduction and proof command ran against the exact bytes now committed to .github/workflows/discord-notify.yml: the step body was extracted programmatically from the live file, not retyped by hand.

Gates run clean. ​

  • npm run lint:actionlint, using a checksum-verified pinned actionlint v1.7.12 binary (Go is not installed on this machine, so the script's own local-dev fallback would otherwise have skipped rather than run): all workflows clean.
  • bash -n against every multi-line run: block in the file (5 blocks): no syntax errors.
  • Em dash count on the touched file: 2 before, 2 after, delta 0. Both predate this change and sit outside the lines it touched (one in the top-of-file permissions: comment, one two lines above the first fixed line); neither was added by this fix.

What did the #966 follow-up fix? ​

The two instances #966 named: ai-changelog.yml line 54, and discord-notify.yml's "Prepare commit info" fallback branch (line 127). Both fixed, same session, same branch. ​

  • ai-changelog.yml's "Check if run should be skipped" step: if printf '%s\n' "$HEAD_COMMIT_MESSAGE" | head -n 1 | grep -qiE '...'; then became a parameter-expansion extraction of the first line (FIRST_LINE="${HEAD_COMMIT_MESSAGE%%$'\n'*}") feeding the same grep -qiE check, the identical pattern #956 used.
  • discord-notify.yml's "Prepare commit info" step, jq-unavailable/empty-commits fallback branch: FIRST_LINE="$(printf '%s' "${HEAD_COMMIT_MESSAGE:-"(no message)"}" | head -n1 | cut -c1-100)" became the same first-line parameter expansion followed by a ${FIRST_LINE:0:100} substring, removing both the head and the cut from the pipeline.
  • Neither file overrides shell: for the touched steps (confirmed by grep, not assumed from the issue text), so GitHub Actions' default bash --noprofile --norc -eo pipefail {0} applies throughout ai-changelog.yml, and discord-notify.yml's "Prepare commit info" step additionally declares shell: bash plus its own explicit set -Eeuo pipefail. Both steps genuinely run under pipefail.

The two instances turned out to fail in different ways, and only one matches what #966 predicted. Proven, not assumed, by extracting each step's real body and running it against the real 94,631-byte 130153ca message. ​

  • discord-notify.yml's fallback branch matches #966's prediction exactly: it is a plain assignment (FIRST_LINE="$(...)"), not an if condition, so a SIGPIPEd printf fails the whole step. Pre-fix, run against the real oversized message: exit 141, zero output, 5 of 5 runs. Post-fix: exit 0, 5 of 5, and the extracted commit line matches the real subject exactly (docs(agents): agent asset hierarchy, plus a drift gate for the workflows layer (#952)).
  • ai-changelog.yml's instance sits inside an if condition, and bash's errexit explicitly does not fire on a command whose status is being tested as an if condition, regardless of what pipefail computes for it. Pre-fix, run against the same real oversized message: exit 0, 5 of 5, step survives. This is a real, mechanical difference, not scheduling luck: confirmed by inspecting PIPESTATUS after the pipeline, which reads 141 0 1 every time (printf SIGPIPEd, head fine, grep legitimately found no match on this particular message), so printf is dying every run, the if-exemption just keeps that from killing the step.
  • That same PIPESTATUS mechanic uncovers a worse, silent bug the "does it crash" framing would have missed: pipefail reports the pipeline's status as the rightmost non-zero exit among all stages. When printf is the only stage that fails (141) and grep legitimately matches (0, -q exits 0 on a hit), pipefail still reports 141, a non-zero status, so the if reads false even though the skip tag really was on line 1. Proven with a second fixture (a synthetic message with [skip ci] on line 1 and the real oversized body appended): pre-fix, PIPESTATUS reads 141 0 0 and the if takes the false branch, SKIP_REASON never gets set, 5 of 5 runs, a false negative with no error and no visible symptom. Post-fix, the same fixture: skip=true, 5 of 5.
  • So ai-changelog.yml's real pre-fix failure mode was never "dead step, no output" (the discord-notify.yml shape); it was a silent false negative on the skip check for any oversized commit message, which the same fix (remove the pipe) also eliminates, since there is no longer a second process for printf to race against.

Regression checks against both fixed step bodies. ​

  • ai-changelog.yml: a bot actor (dependabot[bot]) still sets skip=true on the unrelated, untouched check earlier in the same step; a normal small human commit with no tags still reads skip=false.
  • discord-notify.yml: a small message through the same fallback branch still renders the correct commit line; the old cut -c1-100 and the new ${FIRST_LINE:0:100} were compared directly on a 169-character first line and produced byte-identical 100-character output, confirming the truncation behavior did not change silently.
  • Every reproduction and proof command ran against step bodies extracted programmatically (by YAML block-scalar boundaries, not fixed line numbers) from the live files, both before and after the fix, not retyped by hand.

Gates run clean, same as the #956 fix. ​

  • npm run lint:actionlint, same checksum-verified pinned actionlint v1.7.12 binary, independently re-verified against the published checksum list before use (Go is still not installed on this machine, so this was a real run, not the script's own skip path): all workflows clean.
  • bash -n against every multi-line run: block in both files (5 in ai-changelog.yml, 5 in discord-notify.yml): no syntax errors.
  • Em dash count: ai-changelog.yml 0 before, 0 after. discord-notify.yml 2 before, 2 after (the same two pre-existing, untouched occurrences the #956 fix already documented). Zero added by this fix on either file. The repo-wide npm run lint:emdash separately flags one unrelated file (docs/projects/merchant-and-venue-dashboards/harness/admin-report.json, a different in-flight project, not touched here) with 7 new occurrences against a baseline of 0; that finding is unrelated to this fix and out of scope for it.

Surveyed the rest of .github/workflows/ for the same shape. Two more genuinely exposed instances found, filed as a new issue rather than fixed here (out of this session's scoped door). ​

  • #968: bug(ci): the #956/#966 pipe-buffer SIGPIPE shape also lives in ai-changelog.yml's PR-body head -c 500 lines: ai-changelog.yml lines 219 and 262 (post-fix numbering), in the "Generate changelog with AI" step, each pipe a PR body through jq | tr -d '\r' | head -c 500 inside a plain assignment. GitHub allows PR bodies up to 65,536 characters, which can exceed the 64KB pipe buffer in bytes once multi-byte UTF-8 is involved, so this is the same "reads a prefix and exits early" shape, this time a byte-count prefix (-c) rather than a line-count one (-n).
  • A third candidate at line 154 (MERGE_PR=$(printf ... | grep -oE '^Merge pull request #[0-9]+' | head -1 | grep -oE '[0-9]+' || true)) was deliberately left out of #968: the grep -oE stage narrows its output to at most a handful of short matching lines before head -1 ever sees it, regardless of how large the upstream data is, so it is not meaningfully exposed.
  • Several structurally identical grep|head -1 / jq|head -1 pairs also exist in issue-triage.yml and assign-milestone.yml, all narrowing to a single bounded ID/value the same way; not filed, different file, same low-risk reasoning.

Built with VitePress

Copyright © 2026 Lantern