Do BigQuery descriptions actually exist? An audit
- Run: 2026-08-28, against
lantern-app-dev, read-only, viaINFORMATION_SCHEMA. - Why it gates everything: the operator's constraint is "ideally these definitions live in BQ first that we then pull into the definitions here." If BigQuery carries no descriptions, the first piece of work is populating BigQuery, not building a portal page.
- Issue:
#987: a configurations page, a BigQuery-sourced schema viewer, and the BigQuery page becomes the Report Builder
The answer: build it. The dataset the viewer is actually about is in good shape.
analytics is the dataset the Report Builder and the schema viewer are for, and it is the best-described one in the project. Its three base tables carry a description on every single column.
analytics object | Type | Table description | Columns | Described |
|---|---|---|---|---|
events | BASE TABLE | yes | 12 | 12 (100%) |
ad_delivery_daily | BASE TABLE | yes | 13 | 13 (100%) |
event_counts_daily | BASE TABLE | no | 7 | 7 (100%) |
recent_user_events | VIEW | no | 10 | 0 |
recent_system_events | VIEW | no | 10 | 0 |
The dataset itself is described too: "Lantern analytics events - unified pipeline for client and server-side tracking."
The two zeros are views, and that is normal rather than a gap. A BigQuery view does not inherit column descriptions from the table underneath it. Both views read from events, whose columns are fully described, so a viewer that resolves a view through to its base table shows complete information for all five objects. That resolution is a design decision, not a data problem, and it is cheaper than describing the views by hand.
Project-wide coverage, for completeness
| Dataset | Region | Tables | Views | Dataset desc | Tables with desc | Columns | Columns described |
|---|---|---|---|---|---|---|---|
analytics | us-central1 | 3 | 2 | yes | 2 / 5 | 52 | 32 (62%) |
billing_attrib | US | 5 | 0 | yes | 5 / 5 | 40 | 0 |
billing_export_gcp | US | 3 | 0 | no | 1 / 3 | 227 | 69 (30%) |
billing_exports_gcp | US | 1 | 0 | not checked | not checked | 20 | 0 |
billing_marts | US | 0 | 2 | yes | 2 / 2 | 20 | 0 |
billing_norm | US | 1 | 1 | yes | 2 / 2 | 24 | 0 |
billing_raw | US | 5 | 0 | yes | 0 / 5 | 78 | 0 |
ops | US | 1 | 0 | yes | 0 / 1 | 10 | 0 |
logs | us-central1 | log-bucket views | yes | not exposed |
Totals across the datasets that expose a schema: 24 tables and views, 471 columns, 101 described (21%).
Two numbers in that table are not what they look like.
billing_export_gcp's 69 described columns are Google's, not ours. That is the standard GCP billing export, and its schema ships with descriptions written by Google. Excluding it, columns we have described ourselves are 32 of 244, and all 32 are inanalytics.logsis a log-bucket linked dataset, created by the log sink rather than by us. It has a description for the same reason: Google wrote it.
The billing datasets are described at dataset and table level and not at column level.
Nine of eleven billing tables carry a table description; not one carries a column description. That is a real gap if the schema viewer ever points at billing, but billing is not what this project is about, and nothing here blocks on it.
One finding that changes how the viewer must enumerate
A region-scoped INFORMATION_SCHEMA sweep silently missed a dataset. billing_exports_gcp is reported by the datasets API as being in location US, but region-us.INFORMATION_SCHEMA.TABLES does not return its one table. Querying the dataset's own INFORMATION_SCHEMA directly finds it immediately.
Why that matters for deliverable 2: the obvious way to build a schema reader is one sweep over region-us plus one over region-us-central1. That approach would have silently omitted a dataset from this very audit, and it would fail the same way in the portal, with no error and no empty state, just a dataset that is quietly not there.
So the reader should enumerate datasets from the datasets API first, then query each dataset's own INFORMATION_SCHEMA, which costs one job per dataset instead of one per region. With nine datasets that is a fine trade for not silently losing one.
A second reason to prefer it: INFORMATION_SCHEMA is region-scoped, so a region sweep hard-codes an assumption about where datasets live. The project already spans two regions, and nothing stops a third appearing.
What this does NOT settle
- Whether the descriptions are any GOOD. This audit counted them; it did not read them for quality.
eventsandad_delivery_dailyare fully described, but whether those descriptions are the definitions a reader needs is a separate read-through. - Whether anything keeps them current. Nothing found so far writes descriptions as part of a schema change, so they are hand-maintained and can drift silently. Worth deciding before the portal presents them as authoritative.
How to re-run this
Per-dataset, which is the enumeration this audit recommends:
SELECT
COUNT(*) AS columns_total,
COUNTIF(description IS NOT NULL AND TRIM(description) != '') AS columns_described
FROM `lantern-app-dev.<dataset>.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS`Table-level descriptions live in INFORMATION_SCHEMA.TABLE_OPTIONS under option_name = 'description', and dataset-level ones in INFORMATION_SCHEMA.SCHEMATA_OPTIONS. Both quote their values, so an empty description reads as "" rather than as an empty string.