The BEACON GUI: a browser-native referee for AI weather models

BEACON compares AI weather models against conventional numerical weather prediction using standardized METplus verification, entirely in the browser, with no server. Every interface BEACON has had so far — including the current one — was a prototype; this document designs the first product-grade BEACON GUI. The governing design brief asks for something harder than a dashboard refresh: one product that serves a verification scientist and a program manager without betraying either, and that lets a user redefine what “best” means for their mission — honestly.

This proposal rests on three decisions:

Everything else — the tiered store contract, the question-shaped navigation, the honesty gates in the profile roll-up — follows from these three.

Three bodies of prior work were surveyed for this design: the existing BEACON visualization and its prototypes, the MET-AL lab (thirteen browser experiments in modern MET verification), and the metplus-data-store project (a full re-encoding of a real 62 GB verification archive into browser-streamable stores, with parity tests). The new design inherits deliberately — each asset below was adopted, adapted, or declined on its merits.

The brief fixes the platform: browser-only, no backend, potentially slow local queries handled gracefully. The design answers with a tiered static-store contract — every tier is a plain file layout on static hosting, readable via HTTP range requests, and every tier is optional except the first two. The manifest tells the app what it has; the app never guesses.

A small JSON document, fetched first, that declares per campaign: the model registry (§3.4), observation sources, variables and levels with their coverage, the valid-time span (the brief's date-filter bound), statistic families present, byte sizes and URLs of every store, and explicit capability flags — for example ensemble: absent in the current archive, which has no probabilistic line types. Capability flags are what let the UI distinguish “this dataset can't answer that yet” from “no data in this slice” (§5.4).

One sorted Parquet file per campaign holding the long/tidy METplus stat table: full MET header columns plus line_type, stat_name, stat_value, and confidence bounds where MET emitted them. The surveyed archive proves the scale argument: 1.53 million rows — four models, 24 cycles, twelve variables, six line types — compresses to 8.8 MB, and DuckDB-WASM answers grouped queries against it over HTTP range reads in an existing working demo.

The load-bearing property: the spine keeps the aggregable line types (CTC counts, SL1L2/VL1L2 partial sums) next to the derived ones (CNT, CTS, VCNT). Derived statistics answer “what did MET conclude for this exact slice?”; aggregables let the client honestly recompute statistics for any slice — across cycles, lead windows, or whatever a Decision Profile weights — without committing the mean-of-ratios error (§4.2).

The brief's drill-down job says every summary is a path into the evidence behind it. Post-processed statistics dead-end at the aggregate; Tier 2 extends the path. It holds, per campaign: the point-observation table (the surveyed archive's is 11.7 M rows in a 44 MB Parquet — station, position, variable, level, value, QC flag) and matched-pair output — MET's MPR line type, now decided-enabled in the pipeline (§12), landing as its own Parquet table rather than in the spine to protect the spine's size. Matched pairs are what make station-level drill-down and conditional verification (§8) possible. Paired forecast/observed grids also exist in the archive but need a small cataloging pass before browser use. Tier 2 is manifest-gated: when a campaign lacks it, drill-down bottoms out at the source stat rows, honestly labeled.

The raw archive contains identifiers that must not reach a public UI — model names and verification masks that encode the pilot site. Sanitization is therefore not an editing pass but a build stage: a display-name registry maps every raw identifier to a public name (e.g., a site-specific baseline renders as “Regional WRF”), and the build fails if any unmapped or forbidden string survives into a published store or bundle. The MET-AL lab already runs exactly this leak check for its public data bundle; the BEACON GUI promotes it to a contract requirement. The UI renders only registry names; provenance panels (§4.4) show registry names too, with raw identifiers available only in non-public builds.

The app is host-agnostic by construction — any static host with HTTP range support satisfies the contract — so demo and production can differ in origin and freshness without differing in code. Two shapes are designed:

• Demo (this cycle): Cloudflare, static snapshot. The app deploys to static pages hosting with the Tier 0 manifest and the Tier 1 spine bundled into the deploy itself — at ~9 MB the spine ships same-origin, which eliminates CORS, proxies, and cold-fetch latency in one move. Larger tiers (the 44 MB point-obs table) serve from the existing R2 bucket behind its gating Worker, already live for the sibling lab. The data is explicitly a pinned snapshot: the manifest declares data_mode: "snapshot" with an as-of date, and the UI badges every surface with it, so a demo verdict is never mistaken for a live one.
• Production (open decision, §12): shape TBD. Today the archive lives on an on-premise server that also hosted the prototype GUI behind Caddy. Caddy serves range requests natively, so the same store contract works there unchanged — with data_mode: "live" and the manifest regenerated as campaigns are processed. Nothing in the app binds to either origin.

The engine runs single-threaded by default, which avoids cross-origin isolation headers entirely; threading is an optional optimization where headers can be set. A failed engine initialization is the one prominent error state (§5.4); a failed individual query degrades to a quiet empty state.

DuckDB-WASM (in a Web Worker) does what SQL is good at: slicing the spine by scope, grouping into series, feeding the cascade filters. The stat engine — the adopted MET-parity math library — does what SQL must not be trusted to do: derive verification statistics from aggregables, aggregate across rows correctly, resample for uncertainty, and normalize for profile roll-ups. Queries are keyed by scope and cancellable; a superseded query's results are discarded before render, which is the mechanism behind the brief's no-stale-flicker rule.

Verification statistics do not average. The RMSE of two days is not the mean of two RMSEs; a hit rate over a month is not the mean of daily hit rates. The only honest aggregation sums the ingredients — contingency counts, partial sums — and derives the statistic once, at the end (ratio of sums, never mean of ratios). Because the spine carries those ingredients, every number in the BEACON GUI that spans more than one MET output row is derived this way, by a library whose aggregation primitives were validated against MET itself.

Where MET emitted confidence bounds, the UI shows them. Where a user asks “is this difference real?” the design does not let CI overlap masquerade as the answer — overlapping ranges do not demonstrate equivalence, and the brief flags this as a correctness requirement. Instead, head-to-head significance is computed the way the verification community actually does it: paired resampling on the difference series (both models scored on the same cases, resampled together), rendered as a significance mark with its confidence level. A surveyed MET-AL prototype already implements this grid client-side; the microcopy rule is fixed: “ranges that overlap do not mean the models are equivalent.”

A referee's authority rests on its arithmetic being checkable. Because the spine carries both ingredients and MET's own conclusions, the client can prove its math on the user's machine, continuously.

Concretely, every displayed statistic offers a provenance panel: which stat rows contributed, the derivation (“RMSE from summed squared errors over N pairs”), the aggregation span, and a parity check against MET's derived rows where one exists. This generalizes the “oracle parity” mechanism already prototyped in the surveyed stores — and it is the feature that lets an expert trust a verdict they didn't compute themselves.

The brief lists eight jobs-to-be-done and forbids inheriting the old screen structure. The BEACON GUI organizes around the questions users arrive with, in three surfaces plus one overlay:

• Standing — “Which model is doing best — for my needs?” The landing surface: a profile-contextualized verdict, readable with zero terminology (jobs 1, 8).
• Compare — “Where, and by how much?” Skill vs. lead time, head-to-head across variables and levels, and the cross-metric matrix where each measure is judged on its own terms (jobs 2, 3, 4, 5).
• Evidence — “Show me the numbers behind that.” The drill-down terminus: filtered stat tables, series detail, provenance, and — when Tier 2 data exists — station-level observations (job 7).
• Profile (overlay) — the active Decision Profile: inspect, switch, edit, share (job 8, §6). It is an overlay, not a page, because it contextualizes every surface rather than owning one.

Ensemble calibration (job 6) is designed as a Compare capability that the manifest currently reports absent — it renders as an honest “not available in this dataset” explainer until probabilistic line types arrive (§5.4, §11).

The data's selection dimensions cascade — models constrain observation sources, which constrain variables and levels, and so on — and many combinations simply have no data. The design treats the current selection as a first-class object: a persistent scope bar summarizing model set, observation source, variable, level, statistic, and valid-time range.

• Unavailable options stay visible, annotated — never silently hidden. An option without data under the current scope renders dimmed with a reason on demand (“no CREDIT runs verified against this source”), so the structure of what exists stays legible.
• Analysis-fixed dimensions explain themselves in place. When a view fixes a dimension (vector-wind verification implies vector statistics), the scope bar shows that segment as set by the analysis, with one line of why — a consequence, not a lockout.
• Date controls operate on forecast valid time, bounded by the manifest's actual span, with the bound shown.

Every quantity in the product has three registered renderings, from one translation registry:

The lens is a single global control. Plain is the default for a first visit; the chosen lens rides in the URL, so a shared link reproduces not just the data view but the register it was read in. Crucially the lens is per-rendering, not per-mode: hovering (or focusing) any labeled quantity reveals its other two renderings, so an expert reading Plain-language output can always see the exact notation underneath, and a newcomer in Technical view is one gesture from an explanation. Nobody is trapped in a mode that wasn't built for them — this replaces the executive/advanced split.

Every data region distinguishes, visibly and textually:

• Loading — a quiet skeleton in the region's final geometry; no spinners over stale content, and superseded results never flash (§4.1).
• No data for this selection — a designed empty state that names the narrowing choice (“Nothing verified for this variable in March. Try widening the date range”), because empty is a normal property of sparse verification data, not a failure.
• Capability unavailable — manifest-driven: “This dataset contains no ensemble output, so calibration can't be assessed. Here's what that would show.” The surface stays discoverable and self-explaining rather than disappearing.

One error is deliberately loud: the data engine failing to initialize, which blocks everything and says so plainly. Any single failed query degrades to the quiet empty state.

• Brand. NSF NCAR identity applied through a token layer (type, color, wave motifs where restrained) — reviewed against the brand toolkit rather than improvised. The one systematic extension: amber is reserved product-wide for value-judgment context, so brand color and epistemic meaning never collide.
• Accessibility. WCAG AA as an acceptance gate: full keyboard operability (the scope bar, lens, and profile editor are the hard cases and get explicit keyboard specs), visible focus, ARIA landmarks per surface, chart data reachable as tables, reduced-motion honored (state transitions become instant swaps).
• Responsive. Dense comparison grids don't shrink — they re-rank: on narrow viewports the head-to-head grid becomes a sorted list of the largest honest differences with the full grid one tap away. The scope bar collapses to a summary chip that opens a full-screen editor.
• Onboarding. The Standing surface is the onboarding: it teaches profile, uncertainty, and drill-down by being read. First-visit affordances are limited to three dismissible, non-blocking anchors (what a profile is, what the whiskers mean, where the evidence lives).

The brief's central truth: “best” is not absolute. A Decision Profile captures a use case's priorities and rolls verification evidence into a standing under those priorities — transparently, recomputably, and labeled as the value judgment it is.

A small, serializable object — not a UI arrangement:

• Identity: name, one-sentence description of who it's for.
• Priorities: weights over outcome facets — variable groups (temperature, wind, precipitation, pressure), lead-time windows (nowcast, day-one, medium-range), verification regions (where accuracy matters most — MET's masking regions, §8), and statistic families (systematic error, random error, event detection) — not over 86 raw statistics. Facets keep weights meaningful to a non-expert and map deterministically onto stat rows for the engine.
• Roll-up method: the normalization and blend used to make facets comparable (§6.3) — named and inspectable, never implicit.
• Optionally, a cost model (§7), so Impact Translation travels with the profile.

Preset candidates below come from research on thirty weather-dependent professional cohorts; the catalogue and every weight in it are content to be set with domain experts. Decision (§12): presets ship draft-labeled — marked draft in-product until expert review, rather than waiting on it. General (balanced) is the clearly-labeled default, active from first paint so a visitor gets a coherent answer with zero configuration. One preset is not a draft: MET's own reference skill-score index (§8), whose weights are fixed by the verification community and double as a calibration target for the roll-up engine.

Normalization (stage 3) makes non-comparable measures blendable, so it must be inspectable, not a black box. The proposed default: within each facet, each model's aggregated statistic maps to a relative standing on 0–1 against the model field in the current scope, after applying the metric's direction (a data property — reducing a lower-is-better error always reads as improvement). The method is named in the UI, its intermediate values are visible in the “how this was computed” panel, and the architecture treats it as a pluggable strategy so domain experts can evaluate alternatives (e.g., skill-score-referenced scaling) without a redesign.

Sparse data meets weighted priorities constantly, and the brief demands the collision be visible. Three mechanisms:

• Coverage meter on every verdict: the share of the profile's weighted evidence actually present in scope, with the missing facets named (“wind gusts: weighted 20%, not verified in this dataset”).
• No silent renormalization. Missing weight is reported as missing; the verdict never quietly pretends the profile was different from what the user chose.
• A verdict floor. Below a coverage threshold the standing is withheld — “not enough of what this profile values has been verified here” — because an honest refusal beats a confident distortion.

One capability, two editors over the same object — neither audience gets a dumbed-down or gated-off version:

• Guided: plain-language importance choices (“matters most / matters / doesn't matter”) over outcome facets, phrased as outcomes (“catching rare severe events”, “getting tomorrow's wind right”). Three decisions produce a legitimate profile.
• Precise: numeric facet weights, statistic-family selection within facets, normalization strategy, coverage floor. Switching depths never loses state — Guided choices are coarse positions of the same weights Precise exposes.

The active profile — preset or custom — serializes compactly into the URL. A colleague opening a shared link sees the same verdict under the same priorities, with the same amber context. Named presets travel by id; custom profiles travel by value; server-side persistence stays out of scope per the brief.

Impact Translation re-expresses skill differences in consequences — above all money — so a decision-maker can grasp what a difference in accuracy is worth. It is engineered as the third vocabulary (§5.3): an optional lens over the same quantities, never a replacement for them.

The cost model is a visible object. Each translation rests on a named cost model attached to the profile: an exposure unit (“per event”, “per operating day”), parameters (“imbalance penalty $/MWh”, “cost of an unnecessary protective action”), a mapping from error statistics to expected cost, and a source note for every parameter. Choosing the Impact lens opens with the assumptions, not just the number.

Defaults are illustrations, and say so. No supplier of validated cost parameters exists yet (decision, §12), so v1 ships illustrative defaults: order-of-magnitude parameter sets per preset, sourced from public literature where possible, every one user-adjustable in place. The framing is fixed accordingly — figures are labeled Illustrative estimate — theoretical, based on adjustable assumptions, the assumptions drawer opens on first use of the lens, and the disclaimer travels with shared links. The lens's honest job at this stage is to teach how skill converts to consequence and let users bring their own numbers — not to assert what anything costs.

Ranges, labels, and challengeability. Figures render as ranges (parameter bounds propagated through the mapping). Parameter edits recompute live and travel with the shared profile — turning a suspicious number into a sensitivity analysis. The technical rendering of the underlying statistic stays one gesture away, so an expert can always interrogate the conversion. A confidently wrong dollar figure is the failure mode this section exists to prevent; when a profile has no cost model, the Impact lens says exactly that instead of improvising one.

The brief describes the product; MET describes what verification can honestly say. Several MET/METplus capabilities the brief never mentions bear directly on its central goal — fair, use-case-specific, region-specific comparison — and most cost little to adopt because they arrive as new rows in the existing store contract, not new architecture. Two deserve the headline:

Fairness across model resolutions is a MET capability, not a UI nicety. Point-to-point scoring double-penalizes a high-resolution model — a storm predicted correctly but displaced one grid cell scores worse than no storm at all — which systematically flatters coarse global models in exactly the comparisons BEACON exists to referee. MET's neighborhood methods (fractions skill score) and station-neighborhood analogue (HiRA) are the community's answer, and a referee comparing 2.5 km regional baselines against 0.25° global AI models should adopt them.

MET already ships a weighted composite — use it as the profiles' anchor. MET's skill-score index (the GO-Index family) is a community-defined weighted blend of skill scores across variables, levels, and leads. It is precedent that weighted composites are legitimate verification practice, a shipped preset whose weights BEACON did not choose, and — because MET can compute it offline — an end-to-end parity target for the entire profile roll-up engine, extending the trust loop (§4.4) from individual statistics to the composite itself.

Only the last row adds a tier; everything else is new rows in existing stores plus facet vocabulary. None of it adds screens — Compare and Evidence absorb these as dimensions and drill-downs, and the scope bar's capability-awareness (§5.4) keeps absent ones honest. This is deliberate: adopting more of MET must deepen the referee, not sprawl the product.

The brief's appendix defines the review test: every “must honor” item satisfied, by mechanisms this document can point to.

Every predecessor was a prototype, so continuity with prior code carries no weight here — each layer is chosen on current merits among modern, open-source options. The load-bearing fact: the data layer (stores, worker, stat engine) is framework-agnostic by construction, so the UI-framework choice is deliberately the most reversible one on this list.

Excluded from this cycle by the brief, but the architecture leaves each a clean socket rather than a rework:

• Ensemble calibration — the Compare surface reserves it; the manifest's capability flag flips it on the day probabilistic line types (ECNT, RHIST, PCT) appear in a campaign. No redesign, new data.
• Spatial verification maps — Tier 3's common-grid Zarr contract exists and is already built upstream; forecast/analysis/error fields become an Evidence extension in a future cycle.
• Interactive thresholds — recomputing categorical skill live from paired fields (the “oracle” pattern, already prototyped) belongs to the same future cycle as fields.
• Station time series & conditional verification — no longer deferred by decision, only by data: MPR emission is enabled (§12), and the Evidence surface receives it as soon as the next campaign is processed (§3.3, §8).
• Object-based verification (MODE/MTD) — Tier 4's object tables (§8); a validated browser MODE engine already exists in the surveyed lab when the cycle opens.
• Reliability diagrams, rank histograms, dark mode, image export — out of scope per the brief; nothing here depends on their absence.

Resolved with the product owner, 2026-07-09:

Still open: