[FEATURE] Ship agent-skill guidance for configuring data sources and expectations - #12061
Merged
Merged
Conversation
The schema generator's expectation list is curated: a generated JSON schema is only complete if the expectation's class defines the catalog metadata block (short description, data quality issues, supported data sources). Five registered expectations define no such block, so emitting schemas for them would pass off structurally incomplete files as real ones. Name them explicitly instead, so a completeness check can tell a known, documented gap apart from an accidental one. Also drop a duplicated entry from the generator's list; it produced no second file and only invited confusion about which expectations are covered.
Consumers that want to know what Great Expectations can do currently have to re-parse every schema file to find out. Emit two generated indexes alongside the schemas instead, from the same generation step, so they cannot drift apart. The datasource index maps each schema file to its exact add_or_update_* factory method. Most types snake-case cleanly from the class name but six of twenty-six do not, so the mapping is read from the live type registry at generation time and frozen into shipped data. Because the method name is reconstructed from the registered type name, each one is checked against the real factory surface, so a change to the naming convention fails here rather than shipping an index pointing at methods that do not exist. The expectation index maps each type to its schema file and catalog metadata, and names the registered expectations that have no curated metadata block under documented_absent, so a completeness check can tell a known gap from an accidental one. Both emitters return their serialized content rather than writing it, so a drift check can regenerate in memory and compare against the checked-in file without duplicating the extraction logic it is meant to guard.
A generated file that nothing checks will silently rot. Add assertions that regenerate each index in memory and compare it byte-for-byte to the checked-in copy, that every datasource index entry names a factory method which actually exists, and that every shipped expectation is either cataloged or explicitly recorded as absent - with the catalog and the absence list required to be disjoint, so an entry that gains a schema later cannot linger in both. The completeness check compares against expectations whose implementation lives under the great_expectations package rather than the whole registry. The registry is process-global and test modules register expectations at import time, so an unfiltered comparison would depend on which other test modules happened to be collected. The datasource index guard lives in its own module because the existing schema test file is skipped on every supported Python, so an assertion added there could never run. Hoist the generator's expectation list to a module-level constant so a drift check can regenerate the index without restating the list, which would defeat the purpose of the check.
The expectation and datasource schema catalogs describe what this package can do, and tooling is expected to read them from an installed package rather than from a source checkout. Until now no JSON shipped at all, so those catalogs existed only for people working from a clone. Scope the globs to the two schema trees rather than matching JSON anywhere under the package, so an unrelated JSON file added elsewhere later does not silently become part of the distribution.
Resolving the expectation classes by name makes their type opaque to the checker, so the attribute-defined suppression on the schema call is now unused and fails the type check under warn-unused-ignores.
Three reference documents an agent consults while operating a session: finding or announcing the working context, writing an in-memory session out to a real project, and interpreting slow or failing data operations. Each procedure was executed against a live context rather than written from the API surface, which changed several of them: Neither an unusable GX_HOME nor a discarded cloud configuration produces any signal at the context entry point - both look identical to having no project - so the agent checks the environment itself instead of waiting for an error that never arrives. Adding a data source under a name that already exists replaces it and drops every asset on it, so the write-out procedure fetches first and adds only what is missing, and says plainly what the replacing call would destroy. A time budget is a check-in, not a cancellation: the query is already running on the platform's own compute and keeps costing whether or not the client waits, so the operation is polled on a worker and the user decides whether to keep waiting or narrow the data. Scope reduction leads with a narrower batch definition, because assets describe what data a project works against and batch definitions are how it gets sliced. A row limit through a query asset is offered only when no column supports partitioning, and only as a temporary exploration step.
The entry document carries the guided flow from an empty session to a batch definition that has been proven to read data. Two behaviors it encodes are not obvious from the API surface: Retrieving a batch does not prove a batch definition works. Batch retrieval is lazy for SQL query assets, so a query over a nonexistent table returns a batch and only fails when something reads through it. The flow therefore ends at a head probe, not at retrieval. Replacing a data source drops every asset on it. Reusing an existing data source to add one asset would silently destroy the user's other assets, so the flow fetches first and falls back to the replacing factory only when nothing is there to reuse. Assets and batch definitions have no update-in-place factory at all and raise on a duplicate name, so they reuse or delete-then-recreate. The catalog reference derives every configurable type, its factory name, and its asset and batch-definition surface from the shipped schema index at runtime. The factory naming is irregular in six of twenty-six cases, and a hand-maintained list would go stale the first time a type is added.
The entry document turns checks a user describes in words into expectations drawn from the shipped catalog, runs them, and reports each result. Four behaviors it encodes are not evident from the API. A suite must be registered before expectations are added to it. Expectations added to an unregistered suite validate normally and are then silently lost, because nothing is wired to a store. The handle the store returns is the one to build on. Replacing a suite discards its contents. The upsert factory saves the incoming suite over the stored one rather than merging, so passing a fresh empty suite under an existing name zeroes it with no error. The flow fetches an existing suite and reuses it, and treats replacement as a separate deliberate act. Per-expectation results do not come back in the order they were added. Pairing results to expectations by position appears to work on small suites and silently mislabels larger ones, so results are paired by their own configuration. An empty result means a metric errored rather than data failing, but only when the expectation also did not succeed: a passing expectation can carry an empty result too. The two cases are reported differently, because a broken column is a configuration problem and a failed expectation is a finding about the data. Suite names containing dots fan the store out into nested directories, so they are prohibited.
Agent platforms discover a skill by parsing its frontmatter and following its relative references, so a skill that drifts out of the format stops being found rather than failing loudly. These tests hold the shipped skills to the format: frontmatter parses, the declared name matches the directory, the description stays within limits, and every reference resolves to a real file no more than one directory below the skill root. The three session references are carried by both skills, because a skill directory must be self-contained. They are compared byte for byte, and the failure names the copy direction so drift is a one-command fix rather than a merge. Each check ships with a companion test that builds the violation it is meant to catch and asserts the check reports it. A check that cannot fail is worse than no check, because it reads as coverage. Discovery scans for entry documents rather than naming the skills, and asserts it found some, so an empty scan fails instead of passing quietly.
Guidance that carries code is only as good as the code it carries. The blocks tagged executable now run in order against a throwaway session backed by pandas and sqlite, and the run has to reach three end states: a batch definition that reads data, a validation result carrying an entry per expectation, and a written-out directory that a fresh file-backed context loads with its batch definitions and suites intact. Every other block still has to parse. The negative paths are pinned too, because each one is a claim the guidance makes. A query asset over a missing table hands back a batch and only fails when something reads through it. An expectation whose metric errored returns an empty payload while one that genuinely failed returns a populated one -- and since an expectation that passed also returns an empty payload, both halves of that test are load-bearing. Empty tables and all-null columns produce results rather than raising, including the case where a range check passes because nulls count as missing rather than unexpected. An empty partition window fails at retrieval, before any probe. The configure snippet is exercised against a session that already holds the data source, so it fails if it is ever rewritten to replace rather than reuse. Replacing a data source drops every asset on it, and a snippet that teaches the safe form should stop compiling the moment it stops teaching it. Result ordering gets the same treatment: results are grouped by column and expectations whose metric errored are moved to the front, so an example that pairs results with expectations by position is documenting a coincidence. The text now describes the mechanism, and the tests hold it to that.
Agent platforms look for skills in fixed project directories, so the skills have to be copied out of the package and into the project. The delicate part is not the copy -- it is that the command has to stay safe to re-run against a directory that may hold the user's own work. Each installed skill carries a manifest recording the version that wrote it and a hash of what was written. That is what makes ownership decidable. A copy matching its manifest is left untouched, byte for byte. A copy that still matches its manifest but trails the package is replaced. A copy that no longer matches has been edited by the user, so it is refused and left alone until the run explicitly forces it. A directory with no manifest was never ours and is refused always, including under force -- ownership is not something a flag can assert. Ownership is settled before versions are compared, because the reverse order turns a package having moved on into a licence to overwrite work the user did by hand. Writes stage beside the destination and swap in, so an interrupted run leaves either the previous skill intact or a staging directory the next run clears -- never a half-written skill. Anything that goes wrong with one skill is recorded and reported rather than raised, so one unreadable path costs one destination instead of the whole run. The package hash covers symlinks by their target rather than by what they point at, and the walk that feeds it never descends through a linked directory. Both matter: dereferencing would copy content from outside the package into the project, and directory recursion through links varies by Python version, which would make the same tree hash differently on different interpreters. Discovery resolves the package through the import system rather than walking up from a file path, so it works the same for a wheel, an editable install, and a source checkout. Finding no skills at all is raised rather than reported: nothing to install is a packaging defect, and reporting it as a run where nothing went wrong would hide it.
The skills ship inside the package, but agent platforms look for them in the project. This is the command that bridges the two, invoked as a module rather than a console script so it needs no entry-point wiring and cannot collide with anything already on the path. Install accepts a project root, a target, symlink mode, and a force flag; list reports what is bundled and what each target currently holds, naming the version that installed it so a project left behind by an upgrade is visible before anything is changed. The report is grouped by outcome and the command succeeds only when nothing failed. Two details are less obvious than they look. Failures carry a kind rather than only a message, because the reason a destination was refused cannot be recovered afterwards by looking at it. A skill refused for local edits and a skill whose subdirectory could not be read both still exist and both still hold a valid manifest. Inferring one from the other means telling a user to hunt for a stray file when the real problem is a permission bit. Only the code that made the decision knows, so it now says. The project root defaults to the working directory lazily rather than while the arguments are being defined. Reading the working directory can fail -- it has been deleted or unmounted -- and doing it during argument definition puts that failure outside the handler that turns it into a message, so every invocation raised, including the ones that never needed a directory at all.
The installer decides when to overwrite files in a directory someone else owns, so these tests check the project on disk rather than the report the installer returns. A report can say a skill was left untouched while the bytes underneath it moved; only the filesystem settles it. Modification times and inodes are compared alongside contents, because rewriting a file with identical bytes is still a write. Every check ships with a companion that breaks the installer in the one way that check exists to catch, and asserts the check notices. A test suite that cannot fail is worse than none, because it reads as coverage. Building those companions turned up a recurring weakness in the tests themselves. A helper that sets up more than the behaviour under test requires quietly removes a dimension from every test that uses it: one helper always paired an edit with a version change, so nothing could distinguish a stale skill from an edited one, and deleting that distinction from the installer left the whole suite green. The same shape hid three more, including a re-run that asks for the other install mode and a rebuild that changes content without changing the version -- the case an editable install lives in. The helpers now take those dimensions as explicit arguments and verify them, so a run set up to prove something it does not prove reports that instead of passing. Each condition the installer tests before deciding a skill is current or unmodified is now covered by its own test, one at a time.
The skills are read from the installed package at runtime, so they have to travel with it rather than exist only in the source tree. Matched by file rather than by directory, since a package_data pattern only ever selects files and wheels cannot record an empty directory.
Packaging rules fail silently: a glob that stops matching drops files from the wheel while every source-tree test stays green. This checks the artifact users actually install, after installation, so a packaging regression fails loudly instead of shipping.
Captures why the guidance is version-matched by living inside the distribution, why the command surface is module invocation rather than a console script, and why installing copies into a project by default with an ownership manifest instead of writing over what it did not create.
The rationale is the same; the phrasing no longer dates itself against whatever the test suite happened to cover at the time it was written.
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This was referenced Aug 13, 2026
…lls-and-installer # Conflicts: # setup.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Great Expectations is increasingly configured and validated through a coding agent rather than by hand-written Python. An agent's general programming knowledge doesn't encode the current, correct sequence of calls for this library — which factory method a given connection type needs, what order a suite has to be registered in before expectations are added to it, how to handle a secret without ever printing it to the conversation. Left to infer this, an agent produces plausible-looking code that's subtly wrong about as often as it's right, and a user who doesn't already know the right pattern has no way to tell the two apart.
This ships guidance that closes that gap, in a place and format an agent's own tooling already knows how to find and read.
What's added
Two "skills" — self-contained guidance directories, each with one entry document plus supporting reference material one directory below it. This is an open, multi-vendor format that several coding-agent platforms already read, so publishing in this shape makes the guidance usable by any agent whose platform speaks it, without a separate integration per agent:
gx-configure-data-source— takes an agent from "here is my data" to a verified working batch definition: connect a data source, define an asset, add a batch definition, and prove it by actually reading through it. Its catalog reference derives every configurable type, factory name, and asset/batch-definition surface from the shipped schema index at runtime, rather than a hand-maintained list that would go stale.gx-configure-expectations— turns data quality checks a user describes in words into expectations drawn from the shipped catalog, runs them, and reports results grouped and interpreted correctly (a metric error and a failed expectation are not the same thing, and results don't come back in the order they were added).Both skills share reference material on session handling: finding or announcing the working context, writing an in-memory session out to a real project, and interpreting slow or failing data operations without either blocking indefinitely or silently giving up.
Why the guidance ships inside the package
The correct calling sequence for a fluent factory method or a suite registration is a function of the exact
great_expectationsversion installed. A separately hosted or generated copy can drift out of sync with any given install the moment either changes independently — silently handing an agent instructions for an API surface that no longer matches what's on disk. Shipping the content in the package ties its version to the code's version by construction, so installing or upgrading the package is what keeps the guidance current.See
docs/adr/0006-ship-agent-skills-with-the-package.mdfor the full rationale, including why the command surface ispython -m great_expectationsrather than a new console-script entry point (GX previously shipped and removed one; module invocation needs no packaging-level entry-point wiring and can't collide with anything onPATH).Command surface
installplaces the bundled skills into the discovery directories a project's coding agent reads (.agents/skillsfor Codex/Cursor,.claude/skillsfor Claude Code/Cursor), by copy by default or by symlink on request.listreports what's bundled and what each target currently holds.Install model — safety properties
The installer treats the destination as belonging to the user, not the package:
--force.This makes the command safe to run again after every upgrade, or simply on the suspicion it was never run at all.
What's tested
210 tests cover: installer safety (each safety property above has a paired test that builds the one situation it exists to prevent and confirms the installer catches it); skill format conformance (frontmatter parses, declared name matches its directory, every reference resolves to a real file, the shared reference documents stay byte-identical across both skills); and the skills' own executable guidance (every code block tagged runnable is executed in order against a real session, including the negative paths the guidance warns about — a query over a missing table, an expectation whose metric errors versus one that genuinely fails, an all-null column, an empty result window).
A CI check (
ci/checks/check_installed_agent_skills.py) additionally verifies the installed distribution — built andpip install-ed, not the source tree — actually carries the skills and schema catalogs, and that their installed content hashes match their own ownership manifest. A packaging regression such as a glob that stops matching then fails loudly instead of shipping silently.Note that this check is added to the
import_gxjob, andci.ymlruns onpull_request_target— so the workflow definition comes from the base branch. The new step will not execute on this PR; it becomes live once this merges. Reviewers should expect that rather than read its absence as a pass.Packaging
Adds one
package_dataglob tosetup.py:.agents/skills/**/*.md, scoped to the skills tree rather than a blanket markdown pattern, and matched by file rather than by directory (apackage_datapattern only ever selects files, and wheels can't record an empty directory).No
MANIFEST.inchange is needed —package_datacovers the sdist as well as the wheel, the same property already verified on the catalog PR below.Stacking
This PR is stacked on #12055 and should merge after it — it depends on the schema catalog indexes that PR ships, which the data-source skill's catalog reference reads at runtime.