Cloud Run request logs and the user agent

• Status: Investigated 2026-08-30 on lantern-app-dev, read only. Nothing in GCP was changed. Every number below came from a command that was actually run; the commands are in the last section.
• Issue: #1004: fix(services): every Cloud Run service logs the raw user-agent, so "we collect no device data" is not accurate
• Sibling docs: README.md is the decision brief for #1003. user-agent-redaction.md covers the application-side half.
• Scope: dev only. Prod is out of scope and no prod work is proposed here.

The short version

Google writes its own log of every request to every one of our services, and it records the visitor's browser user agent and their IP address.

• We do not write that log. Our code runs, returns, and then Google's serving layer writes the line. No change to our code can stop it. It can only be stopped at the point where Google decides which logs to keep.
• It records more than the user agent: the IP address is in there too, which is the stronger identifier of the two.

It is kept in two places, for 30 days and for 90 days, and both were measured rather than assumed.

• Every request is stored twice, once in the project's default store (30 days) and once in a second store we created in March 2026 (90 days).
• Verified by asking each store for its oldest surviving entry: the 90 day store goes back to exactly 2026-06-01, the 30 day store to exactly 2026-07-31. Both match their configured setting, so retention is behaving.

BigQuery has a WINDOW onto it, not a copy, and that turns out to matter a lot.

• There is a BigQuery dataset called logs, and it does expose these logs to SQL. But it is marked LINKED and contains one view and zero tables. It is a lens pointed at the 90 day store, not a second store.
• So when the 90 days expire, it disappears from BigQuery at the same moment. Nothing outlives the log store, and there is nothing extra to clean up.
• Checked for anything that could quietly make a real copy: no scheduled query, no BigQuery sink, no materialized table. The only scheduled BigQuery job on the project is the billing pricing transfer.

Recommendation: drop these logs before they are stored, in both places. The price is visibility into 18 errors per 90 days.

• The privacy win is real and it is bigger than the user agent alone, because it takes the IP address with it.
• The price is small for a reason worth stating plainly: almost everything in Google's log is already in ours.

Why the price is that small

Latency and status codes do NOT come from this log, so we do not lose them.

• Cloud Run publishes request_count and request_latencies as monitoring metrics on a completely separate pipeline, broken down by service, response code and route.
• Verified with live numbers: 19 active series across all our services in the last 2 days, including the 4xx and 5xx classes. Deleting the log does not touch any of it.

Our own application log already records the same request, and it is the one we can actually scrub.

• Every service runs pino-http, which writes a "request completed" line per request carrying method, URL, status code and response time.
• That log is roughly four times larger than Google's request log (3.3 MB a day against 0.8 MB), so the platform log is mostly a duplicate.
• It is also the log we control. Google's is not.

The one thing we genuinely lose: requests that die before our code runs.

• If the serving layer times out or a container fails to start, there is no application line, because nothing in our process ever handled the request. Only Google's log has it.
• Verified on a real case: a docs-api 504 at 2026-08-28T15:32:12Z exists in Google's request log and has no matching application line at all.
• Sizing it: 18 requests returned 5xx across the entire last 90 days. That is the whole population of errors this log would be the only witness to, and only some of those 18 never reached our code.

One documented workflow breaks, and it has a same-day replacement.

• docs/engineering/architecture/api/VENUES.md documents a debugging filter built on httpRequest.status and httpRequest.requestUrl for finding failed venue imports.
• It stops matching. The replacement is the equivalent filter over our own log line (jsonPayload.res.statusCode and jsonPayload.req.url), which carries the same two fields. That doc needs updating in the same change.

Nothing else depends on it.

• Zero log-based metrics exist on the project.
• Three alert policies exist and none of them reads this log. They watch scheduler job errors and analytics write failures, all on other logs.

What are the options, and what does each cost?

Option	What it does	What it costs	What we lose
Drop it before storage (recommended)	Google still generates the line, we refuse to store it. Takes the user agent and the IP together.	Two commands. No ongoing infrastructure.	The per-request row for requests that never reached our code: 18 candidates in 90 days. One doc filter to update.
Drop only the successful ones	Keep the entries where something went wrong, drop the rest.	Same two commands, narrower filter.	Nothing operationally. But it fails the privacy goal: every kept error entry still carries a real user's agent and IP.
Scrub and re-write it	Route the log through a queue, strip the fields with our own service, write it back.	A new service to build and run forever.	It does not actually work. The raw entries sit in the queue for its retention window, so Google still holds them; and the rewritten entry is no longer a native request log, so the filters it was meant to preserve break anyway.
Shorten how long we keep it	Same data, held for less time.	One command.	It fails the standard. The data is still collected and still held, just briefly. A request served during the window still finds it. It also shortens every other log in the same store, not just this one.
Hide the field from readers	Mark the field access-restricted on the store.	One command.	It fails the standard for the same reason, more subtly. The value is still written and still stored. It changes who can read it, not whether it exists.

Is there an option that keeps the diagnostics and drops the field?

Yes, and it is the recommendation plus finishing the work already started.

• Google's log has nothing we need that our own log cannot carry. So the way to keep the diagnostics is not to preserve Google's line, it is to make ours good enough and then drop Google's.
• That means three things together: drop the platform log, deploy the redaction we already wrote, and extend it to the rest of the device headers (below).
• The honest caveat, stated rather than dressed up: the requests that never reach our code have no replacement. There is no configuration anywhere that keeps that row and nulls one field in it. If we want those 18, we keep the user agent. That is the actual trade, and 18 in 90 days is what it is worth.

What did this investigation find that outranks the thing it was looking for?

The redaction we believe is live is NOT deployed. Not on any service.

• Both commits sit on feat/admin-and-merchant-portals and neither is on origin/dev: 54ad0b96 (user agent) and b0dbbd8a (App Check token).
• Measured, not inferred: across the last 14 days, zero log entries on the whole project show a redacted user agent, and zero show a redacted App Check token. Raw App Check tokens were still arriving in analytics-api on 2026-08-28, and a raw browser user agent was still arriving in docs-api on 2026-08-30.
• If it was demonstrated working, it was demonstrated on a locally run stack. On deployed dev, the raw values are still landing today.

Even once deployed, redacting user-agent alone does not close the surface, because the whole header block is logged.

• pino-http serializes every header. The redact list names user-agent and stops there. Confirmed still present in docs-api logs on 2026-08-30:

Header still logged	What it reveals
sec-ch-ua-platform	The operating system, in plain text. The live value is "Windows".
sec-ch-ua	The browser and its major version.
sec-ch-ua-mobile	Phone or desktop.
accept-language	Language and region. The live value is en-US.
x-forwarded-for	The visitor's IP address.
forwarded	The IP address again, in the other format.
referer	The page they came from.

• Together these reconstruct most of a user agent. Removing one header from a dump that still contains the same class of data in six other headers is not the fix it appears to be.
• This belongs on #1004, because it is the same claim: "we collect no device data" is not accurate while these are in the log.

Two things checked and found clean, worth recording so nobody re-derives them.

• The analytics events table carries no device data. Twelve columns, none device related. Confirmed again here.
• There is no load balancer in front of anything. Zero URL maps, zero backend services, and no load balancer request logs on the project. Cloud Run's own log is the only request log GCP writes for us, so there is no second source hiding behind this one.

One surface that is out of reach, recorded so it is not mistaken for a gap.

• Audit logs are held 400 days in a locked store, and locked means the retention cannot be shortened and the log cannot be excluded. Those entries do carry a browser user agent and IP.
• But the browser in them is the operator's, using the Google Cloud console. They record who administered the project, not who used the app. It is not an end-user surface and there is no action here.

For whoever implements it

The two commands, written out. NOT run.

gcloud logging sinks update _Default \
  --project=lantern-app-dev \
  --add-exclusion=name=drop-cloud-run-request-logs,description='Cloud Run request logs carry httpRequest.userAgent and httpRequest.remoteIp. See issue 1004.',filter='logName="projects/lantern-app-dev/logs/run.googleapis.com%2Frequests"'

gcloud logging sinks update cloud-run-to-analytics \
  --project=lantern-app-dev \
  --add-exclusion=name=drop-cloud-run-request-logs,description='Cloud Run request logs carry httpRequest.userAgent and httpRequest.remoteIp. See issue 1004.',filter='logName="projects/lantern-app-dev/logs/run.googleapis.com%2Frequests"'

• Both are required. Excluding one still leaves a full copy in the other. _Default is the 30 day store, cloud-run-to-analytics feeds the 90 day store and the BigQuery view.
• gcloud is at /home/mechelle/google-cloud-sdk/bin/gcloud and is not on PATH. An earlier session reported it missing on that basis. Use the absolute path.
• The third sink, _Required, does not carry this log and needs nothing.

An exclusion is forward-looking only. The backlog stays until it ages out.

• Up to 90 days of already-stored entries survive the change and expire on their own schedule.
• Purging them immediately is a separate, irreversible step (gcloud logging logs delete) that deletes every entry in that log at once. It is not part of the recommendation and should be a deliberate decision of its own.

Verify it worked by these, not by the absence of an error.

1. gcloud logging sinks describe _Default and ... describe cloud-run-to-analytics both show the exclusion.
2. Make a request to any service, wait a minute, then read the request log with a short freshness. Zero rows is the pass.
3. Confirm run.googleapis.com/request_count still reports in monitoring, so the diagnostics really did survive.
4. Confirm our own application line is still written for that same request.

The follow-on work this investigation implies, none of it done here.

• Get 54ad0b96 and b0dbbd8a deployed. They are written and not live.
• Extend the redact list to the seven headers in the table above, or replace the default header serializer with an allowlist so a newly invented header does not arrive pre-logged. The allowlist is the version that stops the next one.
• Update the debugging filter in VENUES.md to read our log instead of Google's.

What does this look like for prod later?

Identical in shape and cheaper to do, because prod has no history yet: the same two sink exclusions apply, and setting them before prod carries traffic means no backlog ever accumulates. Recorded so the answer is not re-derived. Not proposed and not scheduled, per the standing preference that prod stays untouched until the migration is finished.

The commands this document's numbers came from

All read only, all against lantern-app-dev.

Question	Command
Which sinks exist, and do any exclude anything	gcloud logging sinks list and gcloud logging sinks describe <name>
Which stores exist and for how long	gcloud logging buckets list
Does BigQuery hold a copy	gcloud logging links list --bucket=lantern-api-logs --location=us-central1, then the BigQuery REST dataset and table listing for logs
Is anything materializing it	BigQuery Data Transfer config listing in us and us-central1
Does the log really carry the user agent	gcloud logging read 'logName="projects/lantern-app-dev/logs/run.googleapis.com%2Frequests"'
Is it in both stores	the same read with --bucket/--location/--view against each
How long does it really live	the same read with --order=asc --freshness=365d against each store
How big is it	monitoring logging.googleapis.com/byte_count, grouped by log, 7 days
Do latency and status survive	monitoring run.googleapis.com/request_count, grouped by service and response class, 2 days
What depends on it	gcloud logging metrics list and gcloud alpha monitoring policies list
Is there a second request-log source	gcloud compute url-maps list, gcloud compute backend-services list, and a read for resource.type="http_load_balancer"
Is the redaction deployed	reads for jsonPayload.req.headers."user-agent"="[REDACTED]" and the App Check equivalent, 14 days, whole project
What else is in our own log	a read of one run.googleapis.com%2Fstdout entry from docs-api