Skip to content

feat(data-warehouse): scaffold google merchant center source - #76410

Open
Gilbert09 wants to merge 5 commits into
masterfrom
posthog-code/scaffold-google-merchant-center-source
Open

feat(data-warehouse): scaffold google merchant center source#76410
Gilbert09 wants to merge 5 commits into
masterfrom
posthog-code/scaffold-google-merchant-center-source

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

A user asked for Google Merchant Center as a data warehouse source in the source-request survey. We don't have it yet.

Changes

Scaffolds Google Merchant Center as a new source stub, following the /implementing-warehouse-sources skill. This is scaffolding only. The source ships with unreleasedSource=True, so it stays hidden from users and does not yet sync anything.

  • New ExternalDataSourceType.GOOGLEMERCHANTCENTER enum value and matching externalDataSources entry in schema-general.ts (regenerated posthog/schema_enums.py + schema.json via pnpm run schema:build).
  • New source class GoogleMerchantCenterSource (category E_COMMERCE) under sources/google_merchant_center/, registered in _load_all.py.
  • Generated config module via pnpm run generate:source-configs.
  • Icon added at frontend/public/services/google_merchant_center.png (via Logo.dev).
  • Linear Django migration 0116_google_merchant_center_source (choices-only AlterField, no-op SQL) with max_migration.txt updated.
  • SOURCES.md scaffolded-list entry.

The other requested survey entry ("yes") was junk, not an identifiable source, so it was skipped.

How did you test this code?

Agent-run automated checks only (no manual/live-API testing):

  • pnpm run generate:source-configs and pnpm run schema:build regenerate cleanly.
  • Source registers and get_source_config resolves (verified via manage.py shell).
  • Migration verified linear: makemigrations --check --dry-run reports no changes, single leaf 0116, sqlmigrate is a no-op, applied OK locally.
  • pytest over the source suites passes (test_source_categories, test_source_versions, test_source_config_generator, test_all_sources_import, test_source_loading).
  • ruff check/ruff format clean on touched files.

Note

hogli ci:preflight's migration check failed locally with a ClickHouse ServerError from an unhealthy local dev ClickHouse. This change contains no ClickHouse migrations, and the Postgres migration was verified linear through the checks above. Real CI runs against a healthy ClickHouse.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No user-facing docs yet: the source is hidden (unreleasedSource=True) and has no sync logic. Docs come with the full implementation.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored by Claude (Claude Code). The task was to triage survey-requested warehouse sources and scaffold the legitimate ones per the /implementing-warehouse-sources skill (which I invoked, along with /django-migrations).

Decisions:

  • "yes" was skipped as junk (not an identifiable source).
  • "googlemerchantcenter" maps to Google Merchant Center, a real Google product for e-commerce product feeds. Verified a brand logo exists via Logo.dev; confirmed no existing source and no in-flight PR (searched open PRs and the maintainer's own open PRs).
  • The config generator regenerated ~20 unrelated generated-config modules that are stale on master. I reverted those so this PR touches only the new source's files.

Created with PostHog Code

Scaffolds Google Merchant Center as a new data warehouse source stub in response to a user request from the source survey.

This is scaffolding only: the source is registered with `unreleasedSource=True` (hidden from users) and has no sync logic yet. Wires up the `ExternalDataSourceType` enum, frontend `externalDataSources` entry, generated config, registration, icon, and a linear Django migration.

Generated-By: PostHog Code
Task-Id: b9f98a01-e862-45f0-b24e-ac7aff43c837
@Gilbert09 Gilbert09 added the skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs label Aug 2, 2026
Copilot AI review requested due to automatic review settings August 2, 2026 09:37
@trunk-io

trunk-io Bot commented Aug 2, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@github-actions
github-actions Bot requested a deployment to preview-pr-76410 August 2, 2026 09:37 In progress
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

🦔 Hogbox preview · 🔧 building…

Spinning up PostHog for this PR on a hogland hogbox — this comment updates in place when it's ready (usually a few minutes).

commit b9ba09d · build log

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Bundle size — 🔺 +69 B (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 65.51 MiB · 🔺 +69 B (+0.0%)

No file changed by more than 1000 B.

Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report

Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root Eager (shipped) Δ vs base Budget
entry (logged-out pages, app bootstrap)
src/index.tsx
1.25 MiB · 22 files no change ███░░░░░░░ 27.7% of 4.51 MiB
authenticated shell (every logged-in page)
src/scenes/AuthenticatedShell.tsx
8.14 MiB · 3,035 files 🔺 +23 B (+0.0%) ████████░░ 83.8% of 9.71 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx
Size File
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB src/RootErrorBoundary.tsx
912 B ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B src/scenes/ChunkLoadErrorBoundary.tsx
762 B src/index.tsx
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
Size File
285.5 KiB ../node_modules/.pnpm/posthog-js@1.410.1/node_modules/posthog-js/dist/rrweb.js
267.7 KiB ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
235.5 KiB src/taxonomy/core-filter-definitions-by-group.json
231.5 KiB ../node_modules/.pnpm/posthog-js@1.410.1/node_modules/posthog-js/dist/module.js
154.3 KiB ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.3 KiB src/lib/api.ts
94.7 KiB ../packages/quill/packages/quill/dist/index.js
93.3 KiB ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.6 KiB ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js

Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479

Toolbar bundle — eager 2.19 MiB within budget

What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.

Metric Size Δ vs base Budget
Eager (shipped)
entry + static imports
2.19 MiB · 17 files no change ████░░░░░░ 38.3% of 5.72 MiB
Deferred (lazy) 2.08 MiB · 33 files no change n/a — loads on demand
Loader dist/toolbar.js 1.1 KiB no change █░░░░░░░░░ 5.8% of 19.5 KiB
Largest eagerly-shipped chunks
Size File
718.3 KiB dist/toolbar/toolbar-app-FT4UJ3IP.css
551.4 KiB dist/toolbar/chunk-chunk-TKZ3SULZ.js
484.6 KiB dist/toolbar/chunk-chunk-T44C2V5C.js
133.6 KiB dist/toolbar/chunk-chunk-PETGQ2AR.js
131.8 KiB dist/toolbar/chunk-chunk-T5KY5WYR.js
71.0 KiB dist/toolbar/toolbar-app-HOL6RSL7.js
69.0 KiB dist/toolbar/chunk-chunk-27JL52RE.js
35.6 KiB dist/toolbar/chunk-chunk-T3UPWJAA.js
20.9 KiB dist/toolbar/chunk-chunk-YJS276P3.js
12.2 KiB dist/toolbar/chunk-chunk-PIK3PADE.js

Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile

Dist folder size — 🔺 +17.2 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1383.91 MiB · 🔺 +17.2 KiB (+0.0%)

Playwright — all passed

All tests passed.

View test results →

⚠️ Backend snapshots — 2 updated (2 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 2 snapshots (2 modified, 0 added, 0 deleted)

What this means:

  • Query snapshots have been automatically updated to match current output
  • These changes reflect modifications to database queries or schema

Next steps:

  • Review the query changes to ensure they're intentional
  • If unexpected, investigate what caused the query to change

Review snapshot changes →

⚠️ Django migration SQL — 1 new migration to review

We've detected new migrations on this PR. Review the SQL output for each migration:

products/warehouse_sources/backend/migrations/0116_google_merchant_center_source.py

BEGIN;
--
-- Alter field source_type on externaldatasource
--
-- (no-op)
--
-- Alter field source_type on pendingsourcecredential
--
-- (no-op)
COMMIT;

Last updated: 2026-08-03 11:48 UTC (b9ba09d)

Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 0 Safe | 1 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

warehouse_sources.0116_google_merchant_center_source
  └─ #1 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: externaldatasource, field: source_type, field_type: CharField
  └─ #2 ⚠️ AlterField
     Field alteration may cause table locks or data loss (check if changing type or constraints)
     model: pendingsourcecredential, field: source_type, field_type: CharField

Last updated: 2026-08-03 11:48 UTC (b9ba09d)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR scaffolds Google Merchant Center as a new (hidden) data warehouse source stub within products/warehouse_sources, wiring it into the existing source registry + generated schema artifacts while ensuring it remains unavailable to users via unreleasedSource=True.

Changes:

  • Adds a new ExternalDataSourceType entry for Google Merchant Center and propagates it through generated schema enums/artifacts.
  • Introduces a new GoogleMerchantCenterSource stub (config-only) and registers it in the source loader + source catalog list.
  • Adds a Django migration updating source_type choices for the warehouse-sources models.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
products/warehouse_sources/backend/types.py Adds ExternalDataSourceType.GOOGLEMERCHANTCENTER.
products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md Adds google_merchant_center to the source scaffold list.
products/warehouse_sources/backend/temporal/data_imports/sources/google_merchant_center/source.py New stub source definition with unreleasedSource=True and E-commerce category.
products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/googlemerchantcenter.py Generated empty config class for the new source.
products/warehouse_sources/backend/temporal/data_imports/sources/_load_all.py Ensures the new source module is imported/registered.
products/warehouse_sources/backend/migrations/max_migration.txt Advances recorded max migration to 0116_google_merchant_center_source.
products/warehouse_sources/backend/migrations/0116_google_merchant_center_source.py Choices-only migration updating source_type allowed values.
posthog/schema_enums.py Adds schema enum value GOOGLE_MERCHANT_CENTER.
frontend/src/queries/schema/schema-general.ts Adds GoogleMerchantCenter to the externalDataSources list.
frontend/src/queries/schema.json Adds GoogleMerchantCenter to the generated schema JSON.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add GoogleMerchantCenter to the kea-typegen source-type unions in sourceCatalogLogic and sourceWizardLogic, keeping the generated blocks in sync with the new externalDataSources entry. Fixes the frontend typechecking / schema-up-to-date check.

Generated-By: PostHog Code
Task-Id: b9f98a01-e862-45f0-b24e-ac7aff43c837
@posthog

posthog Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

👋 Visual changes detected for this PR.

Review and approve in PostHog Visual Review

If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix.

@Gilbert09
Gilbert09 requested a review from a team August 3, 2026 11:14

@fuziontech fuziontech left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review agent on behalf of @fuziontech.

The scaffold follows the existing unreleased-source pattern, keeps the connector hidden until implementation, and updates the registry, enums, migration chain, generated configs/types, icon, and kea typegen consistently. Focused source registry/config/category/version tests passed (3,861 tests), along with Ruff and Python compilation checks. No blocking issues found.

…er-source

Generated-By: PostHog Code
Task-Id: 08bac2d1-a56a-4647-84ad-8dc72c8dd711
@trunk-io

trunk-io Bot commented Aug 3, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@github-actions
github-actions Bot requested a deployment to preview-pr-76410 August 3, 2026 11:45 In progress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants