Session Report — omnictx harvest-freshness → live CDP → WA-network research

2026-07-15 (spans 2026-07-10 → 07-15) · repo valter-silva-au/omnictx · branch feat/harvest-freshness · + workspace Code/valter

Summary

Built the entire omnictx harvest-freshness PRD (F1–F5) end-to-end: per-source freshness in the store, an MCP freshness tool + GET /stats + extension readout, an on-demand "refresh now" gesture, and the active-refresh lane — capture_url and refresh_source — that drives the user's own browser over a loopback CDP attach. Reviewed it (two-axis), then live-tested it against a real debug Chrome + the real store, which surfaced and fixed two genuine bugs (Chrome 136+ rejects any DevTools WebSocket carrying an Origin; a scratch-tab about:blank ready-race). Shipped as PR #26. Added Taskfile launchers for per-profile debug Chrome. Then pivoted to a Perth-ecosystem networking push: deep-researched venture ideas for 4 WA innovation leaders (Brodie / Charlie / Jo / Zane) via a 9-agent workflow with adversarial stress-testing, and helped ratify a team unified-CEO-report norm.

Recap reflects available context + git facts; this is a long, compacted, multi-day session (files dated 07-10 → 07-15) so earlier steps may be partially captured. File lists are from git (the branch) + an on-disk mtime scan, not memory.

Timeline

  1. F1 — per-source freshness (count, most_recent, age_seconds) in store.Stats, TDD, 100% pkg coverage.
  2. F2 — MCP freshness tool + GET /stats + an options-page "Source freshness" readout.
  3. F3 — user-gesture "Refresh now" that drives an already-open tab (no surprise tab-opening).
  4. F4/F5capture_url + refresh_source over a hand-rolled CDP client (internal/cdp) + a Go collection-walker (internal/harvest); omnictx refresh CLI for opt-in cron self-healing.
  5. Two-axis code review → fixes: denylist-wins on the active lane, active-refresh logging, freshness fail-safe-to-stale.
  6. Live stress test (real store + debug Chrome) → 17/17 checks; found + fixed the no-Origin WebSocket bug and the ready-race; hackernews freshness 48h → <60s.
  7. PR #26 pushed over SSH; opened. (Jenkins CI polls main, so it verifies post-merge.)
  8. task chrome:cdp:* — per-profile persistent debug-Chrome launchers (personal/gusmao/mia) + omnictx:refresh; live-verified (refresh walked HN over real Chrome, ingested delta).
  9. Allowlist task — parsed a bookmarks export → 21 content-only hosts + a paste-in snippet for the myimaginationai profile.
  10. Deep-research workflow (9 agents) — a venture idea per WA leader, skeptic-stress-tested, ranked into a portfolio (Brodie #1).
  11. Unified-CEO-report norm — confirmed already ratified by parallel sessions; registered this session on the live board.

Artifacts produced

New files this session. Go/JS files are committed on feat/harvest-freshness (git); workspace files are gitignored (mtime/disk).

FileWhat it is
internal/cdp/cdp.goLoopback CDP client: scratch-tab open/navigate/evaluate/close; loopback-only guard.
internal/cdp/ws.goHand-rolled RFC 6455 WebSocket client that sends no Origin (Chrome 136+ rejects one); zero new modules.
internal/cdp/session.goCDP JSON command/response session + readiness gate.
internal/harvest/harvest.goServer-side collection walker: adapter walk → frontier delta → ingest, bounded pages.
internal/harvest/sources.goRefreshable-source registry (hackernews walk expr; reddit/lobsters/googlenews added later by a parallel session).
internal/denylist/denylist.goServer-side sensitive-host guard (mirrors the extension builtin) — denylist-wins on the active lane.
scripts/stress-test.mjsLive end-to-end stress harness (real store + real CDP Chrome) → JSON results.
scripts/stress-report.mjsRenders the stress results into a self-contained HTML report.
tools/omnictx-cdp/launch-chrome-cdp.ps1Launch a dedicated persistent debug Chrome per profile (own port/user-data-dir).
tools/omnictx-cdp/stop-chrome-cdp.ps1Stop/report the omnictx debug-Chrome instances (never touches normal Chrome).
downloads/omnictx-stress-report.htmlRendered live-stress-test report (17/17; the freshness before/after proof).
downloads/omnictx-allowlist-snippet.jsPaste-in snippet to merge 21 bookmarked content hosts into the myimaginationai allowlist.
perth-network-big-ideas (workflow)The 9-agent deep-research workflow: research → skeptic stress-test → ranked portfolio.

Also on the branch but authored by a parallel session (not attributed to this one): extension/src/adapters/{reddit,lobsters,googlenews}.js + tests, scripts/stress-test-multi.mjs, and CDP commits 16d7cfd / 9647981 / 6944668.

Files edited

FileChange
internal/store/store.go (+ test)Per-source freshness via one GROUP BY; injected clock; derive corpus most_recent.
internal/mcp/mcp.go, mcp_serve.go (+ test)New tools: freshness, capture_url, refresh_source; cap→capturer, fail-safe-to-stale.
internal/api/api.go (+ stats_test)GET /stats endpoint (token-guarded, fail-open).
internal/app/app.go, cli.go (+ tests)Wire CDP client + capturer + refresher; omnictx refresh subcommand w/ --min-interval guard.
internal/browsercap/browsercap.go (+ test)CaptureURL orchestration + ErrDenied + active-capture logging.
extension/src/lib/{client,harvest}.jsclient.stats(); pure pickRefreshTab.
extension/src/options/options.{js,html}"Source freshness" readout + "Refresh now" gesture + formatAge.
extension/src/background/{harvest-loop,service-worker}.jsMSG_REFRESH_SOURCE + handler routing.
docs/DESIGN.mdDocumented the new MCP tools in the surface list.
Taskfile.yml workspaceAdded chrome:cdp[:all|:stop|:status] + omnictx:refresh tasks.
wiki/live.md workspaceRegistered this session's line on the coordination board.

Key decisions

Tasks

This session created/updated no adb tickets — the omnictx work is tracked in its own repo via GitHub PR #26, not the valter-workspace adb board. The adb board (available this run) holds the separate venture-validation tickets (FIFO / questionnaire / Vibe-to-Prod), untouched here. The deep-research portfolio is a candidate for a new venture-validation ticket if you want it on the CEO dashboard.

Sources & caveats

Fact sources used this run: git (branch diff) on-disk mtime adb (available) gstack (thin) transcript — grabbed a DIFFERENT concurrent session, ignored.

Recap reflects available context + git facts; this is a long/compacted, multi-day session so earlier steps may not be fully captured. File lists are derived from git + the mtime scan, not memory. The feat/harvest-freshness branch is shared with parallel sessions; commits/files noted as "parallel session" are excluded from this session's attribution. The gatherer's default git block (branch main, ?? config/) was stale for this work and was not used.