AGENTS.md

AGENTS.md

This file provides guidance to AI coding agents when working with code in this repository. It is tool-neutral and intended to be shared across agent runtimes.

Memory Bank

• Use ai-memory/README.md for durable project context and architecture notes.
• If present, use ai-memory/PROMPT.md and ai-memory/TASKS.md for active feature/task context.
• Keep memory-bank updates concise and aligned with implemented behavior.

Project Overview

ElasticGraph is a schema-driven, scalable, cloud-native, batteries-included GraphQL platform backed by Elasticsearch/OpenSearch. It's designed as a modular system with a small core and numerous built-in extensions, organized as a Ruby monorepo containing 20+ gems.

Common Commands

Testing

• script/run_specs - Run entire test suite (uses flatware for parallelization)
• script/run_gem_specs [gem_name] - Run tests for a specific gem (e.g., elasticgraph-support)
• bundle exec rspec [path] - Run specific test file or directory
• bundle exec rspec --only-failures - Run only previously failed tests
• bundle exec rspec --next-failure - Run failures one at a time (for iterative debugging)
• script/flatware_rspec [path] - Run tests in parallel (faster for large test runs, slower for small subsets)

Important: Integration/acceptance tests require a running datastore:

bundle exec rake elasticsearch:test:boot
# or
bundle exec rake opensearch:test:boot

Test runtime expectations (avoid waiting on a stuck run)

• Expected runtimes: script/run_gem_specs elasticgraph-graphql ≈ 1-2 min; full script/run_specs ≈ 5-8 min. If a test run exceeds ~3x the expected time, do NOT keep waiting — the datastore is likely in a bad state.
• Diagnosing a stuck/slow run: check curl -s localhost:9234/_cluster/health and look for a large active_shards count, and check log/datastore_client.test.log for repeated 429 rejected_execution_exception errors. Recover by killing the test run and re-booting the datastore (bundle exec rake elasticsearch:test:boot or opensearch:test:boot), then re-run.
• Always pipe long test runs to a file (e.g. > /tmp/specs.log 2>&1) rather than through tail/grep, so you can observe incremental progress and distinguish "still working" from "hung".

Build & Validation

• script/quick_build - Run abridged CI build (recommended before opening PRs)
• script/lint - Run linter (Standard Ruby)
• script/lint --fix - Auto-fix linting issues
• script/type_check - Run Steep type checker
• script/spellcheck - Check spelling (uses codespell)
• script/spellcheck -w - Auto-fix spelling issues

Schema & Artifacts

• bundle exec rake schema_artifacts:dump - Regenerate schema artifacts after schema definition changes
• Schema definition files: config/schema.rb and config/schema/*.rb

Local Development

• bundle exec rake boot_locally - Boot ElasticGraph locally (from a new project)
• bundle exec rake site:serve - Serve project website locally at http://localhost:4000/elasticgraph/
• bundle exec rake site:preview_docs:[gem_name] - Preview API docs for a specific gem (faster feedback loop)

Documentation

• API documentation uses YARD
• 100% documentation coverage is required for all public methods and classes.
• Website source: config/site/
• Example queries: config/site/examples/*/queries/
• When writing links in documentation, use permalinks (links to a specific commit/version)
• Prefer relative links to ElasticGraph documentation over external links to ruby-doc.org

Validated Code Snippets in User Guides

One of the website's guiding principles is that all code snippets in guides must be validated — pulled from real example projects under config/site/examples/ rather than hand-written inline in the markdown. This ensures CI catches any future API breakage that would invalidate the docs.

When you add or modify code examples in config/site/src/guides/*.md, do this from the start (don't write inline code='...' blocks first and migrate later):

1. Pick or create an example project under config/site/examples/<name>/. Each project needs at minimum:
   • schema.rb — schema definition file (validated by schema_artifacts:dump)
   • local_settings.yaml — points schema_artifacts.directory at config/site/examples/<name>/schema_artifacts
   • Optional: queries/<category>/*.graphql (validated against the schema by the query registry)
   • Optional: additional .rb files for snippets that don't need to execute (e.g. Rakefile-style examples)
2. Mark snippet ranges in the source files with # :snippet-start: <name> / # :snippet-end: comment fences. The fenced region becomes available as <example>.snippets.<file>.<name>. Whole files are also exposed as <example>.files.<file>, but prefer fenced snippets — they let you keep the standard Block copyright header at the top of the file (and any boilerplate like schema.json_schema_version 1) without it leaking into the rendered docs. Trim each snippet to just what's pertinent to the guide.
3. Reference snippets from markdown with the data parameter:

   {% include copyable_code_snippet.html language="ruby" data="<example>.snippets.<file>.<name>" %}
   

   Never write inline code='...' blocks for guide content — those bypass validation.
4. Run bundle exec rake site:examples:<name>:extract_snippets to regenerate config/site/src/_data/<name>.yaml (this file is gitignored — it gets rebuilt on the fly).
5. Preview with bundle exec rake site:serve to confirm the snippet renders correctly before opening a PR.

For an end-to-end example, see config/site/examples/custom_resolver/ and how its snippets are pulled into config/site/src/guides/custom-graphql-resolvers.md.

Architecture

Monorepo Structure

All gems follow the pattern: elasticgraph-[name]/ containing:

• lib/elastic_graph/[name]/ - Source code
• spec/ - RSpec test suite
• Gemfile - Symlinked from root Gemfile
• [name].gemspec - Gem specification

Gem Categories

Core Libraries (8 gems): Always included in production deployments

• elasticgraph-admin: Datastore administration
• elasticgraph-datastore_core: Core datastore logic
• elasticgraph-graphql: GraphQL query engine
• elasticgraph-indexer: Data indexing
• elasticgraph-schema_artifacts: Schema artifact access
• elasticgraph-support: Shared utilities
• elasticgraph-rack: Rack server
• elasticgraph-graphiql: GraphiQL IDE

Local Development Libraries (3 gems):

• elasticgraph: Project bootstrapping
• elasticgraph-local: Local development support
• elasticgraph-schema_definition: Schema definition DSL

Datastore Adapters (2 gems):

• elasticgraph-elasticsearch: Elasticsearch client wrapper
• elasticgraph-opensearch: OpenSearch client wrapper

Extensions (5 gems): Optional functionality

• elasticgraph-apollo: Apollo Federation support
• elasticgraph-health_check: Health checks
• elasticgraph-query_interceptor: Query interception
• elasticgraph-query_registry: Source-controlled query registry
• elasticgraph-warehouse: Data warehouse ingestion

AWS Lambda Integration (6 gems):

• elasticgraph-admin_lambda, elasticgraph-graphql_lambda, elasticgraph-indexer_lambda, elasticgraph-indexer_autoscaler_lambda, elasticgraph-warehouse_lambda, elasticgraph-lambda_support

Key Components

Query Engine (elasticgraph-graphql):

• ElasticGraph::GraphQL::DatastoreQuery: Intermediate query representation
• ElasticGraph::GraphQL::Aggregation: Aggregation query handling
• filter_node_interpreter.rb: Filter operator mapping

Schema Definition (elasticgraph-schema_definition):

• SchemaElementNames: Customizable GraphQL schema element names
• config/schema.rb: Main schema entry point
• config/schema/*.rb: Schema definition files (e.g., teams.rb, widgets.rb)

Test Infrastructure:

• spec_support/: Shared test utilities
• Test coverage maintained at 100%
• Uses FactoryBot for test data

Development Workflow

Adding Query API Features

When adding filtering predicates or aggregation functions:

1. Design Phase: Create GitHub Discussion, research datastore capabilities, design GraphQL API following ElasticGraph's guiding principles
2. Schema Definition: Update SchemaElementNames, built-in types, and test coverage
3. Query Translation: Implement GraphQL → datastore query translation with unit, integration, and acceptance tests
4. Documentation: Add user-facing docs with working examples to config/site/

Verification

After completing a chunk of work, run script/run_specs and script/type_check to verify correctness across all gems and type signatures.

Test Strategy

Three layers of testing:

• Unit tests: Build and inspect DatastoreQuery without execution
• Integration tests: Build and execute DatastoreQuery directly (skips GraphQL layer)
• Acceptance tests: End-to-end GraphQL queries with real datastore

Bundle Management

The repo uses a symlinked Gemfile approach:

• Root Gemfile defines development, site, and test dependencies
• Each gem has a symlinked Gemfile that includes its .gemspec and recursively resolves ElasticGraph gem dependencies from the repo

Custom gems can be added via Gemfile-custom (see Gemfile-custom.example), then run source script/enable_custom_gemfile.

Code Standards

• Linter: Standard Ruby (see .standard.yml)
• Type Checker: Steep (Steepfile)
• Test Framework: RSpec
• Coverage: 100% required (enforced by SimpleCov)
• Ruby Version: 3.4.x or 4.0.x
• Standard Comment Header: Required on most Ruby files (copyright notice)

Ruby Idioms

• Prefer filter_map over .select { ... }.map { ... } chains. For example, instead of:

  items.select { |item| item.value }.map { |item| item.value }

  Use:

  items.filter_map { |item| item.value }

• Prefer ::Data.define over ::Struct.new for immutable data classes. Use Struct only when mutability is required.
• Don't rely on exceptions for control flow (exceptions are slow). Handle edge cases explicitly instead (e.g., check for nil before calling a method that would raise ArgumentError).
• For constants accessed from multiple EG gems, define them in elasticgraph-support/lib/elastic_graph/constants.rb.
• Always put a blank space after # in comments. This applies to all comments, including RBS type annotation comments (e.g., # : String not #: String).
• Avoid defensive code for impossible cases. Don't add checks, tests, or handling for scenarios that should never occur due to the design of the system. If something can only happen due to a bug in the implementation, it's better to fail fast than to silently handle it.
• Drop unnecessary namespace prefixes. When code is already inside module ElasticGraph, use Indexer instead of ::ElasticGraph::Indexer, and Errors::ConfigError instead of ::ElasticGraph::Errors::ConfigError. Use fully qualified names only when necessary to avoid ambiguity.
• Prefer using prepend + super rather than alias_method when needing to hook in and override a method.
• Keep require statements alphabetically sorted (by the required path string). This applies to both lib and spec files. When adding a new require, insert it in the correct alphabetical position rather than appending it to the end. An exception can be made if the files have to be required in a specific order (this should be very rare).

RBS Type Signatures

• When defining RBS signatures for extension modules, prefer declaring the concrete type a module extends rather than defining custom interfaces. For example, use module IndexExtension : ::ElasticGraph::SchemaDefinition::Indexing::Index instead of creating a custom _IndexExtensionInterface.
• Place interface methods on the narrowest correct interface. If a method only exists on indexable types, put it on _IndexableType, not _Type.
• Define type aliases (e.g. type abstractType = InterfaceType | UnionType) for repeated union types rather than duplicating them across signatures.
• When Steep can't infer a narrowed type from an expression, prefer an inline RBS type annotation comment (e.g. # : SchemaElements::InterfaceType) over a _ = cast. Use _ = only as a last resort when an annotation won't work.
• For Data.define blocks where Steep can't type-check the dynamically generated initialize, use # @implements ClassName with a corresponding class ClassName < Data in the RBS file that declares the method signatures. Avoid __skip__ = def.
• Prefer &:method_name over explicit blocks. If Steep complains, add the missing method to the RBS rather than rewriting as { |x| x.method_name }.
• Keep @ivar declarations adjacent to their accessor method in the RBS, not grouped elsewhere.
• Be precise about collection element types in RBS — e.g. if a Set only ever contains UnionType, type it as Set[UnionType] not Set[UnionType | InterfaceType].
• Use @dynamic annotations only for methods that actually exist at runtime (provided by delegation, Struct, or included modules). Never use @dynamic for methods that would raise NoMethodError.

Testing

• Avoid duplicate tests. If two tests will always pass/fail together, keep only one.
• Use expect_to_return_non_nil_values_from_all_attributes to test wrapper classes (like WarehouseLambda, GraphQL, Indexer, etc.). This automatically exercises every zero-argument method and verifies all dependencies are built successfully.
• Use the :capture_logs RSpec tag instead of logger test doubles for verifying log output. Access logs with logged_jsons_of_type(message_type).
• Use build_* helper methods from spec/support/builds_*.rb to construct test objects. These helpers provide sensible defaults while allowing selective overrides for testing specific scenarios.
• Only tag a spec with :dont_validate_graphql_schema when it's actually required for the test to pass under VALIDATE_GRAPHQL_SCHEMAS=1 (the tag skips that validation). First try to fix the test's schema so it produces a valid GraphQL schema while still exercising what the test is meant to exercise; reach for the tag only when that isn't possible. Verify by running the spec with VALIDATE_GRAPHQL_SCHEMAS=1.

Important Patterns

Schema Definition

• When referencing derived type names (e.g. filter input types), never hardcode names like "StringFilterInput". Always use schema_def_state.type_ref("String").as_filter_input.name (or similar as_* methods on type references). Hardcoded names break when schema element names are customized (e.g. camelCase schemas).

Schema Artifacts

After schema definition changes, always run:

bundle exec rake schema_artifacts:dump

This updates:

• Generated GraphQL schema
• Datastore mappings
• Runtime metadata
• Datastore scripts (including auto-updating INDEX_DATA_UPDATE_SCRIPT_ID constant)

Datastore Configuration

Test datastore ports configured in config/settings/test.yaml.template. The Rakefile automatically clears ClusterConfigurationManager state files when booting test datastores.

Flatware Parallelization

Tests run in parallel via flatware when beneficial. The build scripts automatically determine when to use it based on test suite size.

Troubleshooting

• Datastore in bad state: Kill and restart rake [elasticsearch|opensearch]:test:boot
• Schema artifact issues: Run rake schema_artifacts:dump twice if updating update_index_data.painless script
• Memory resets between datastore boots: The boot_prep_for_tests task clears ClusterConfigurationManager state files

Repository Files

• Rakefile: Main task definitions, schema artifact automation, test prep hooks
• script/: Build and development scripts
• config/schema.rb: Schema entry point
• config/settings/: Environment configurations
• spec_support/: Shared test infrastructure
• CODEBASE_OVERVIEW.md: Detailed architecture and dependency diagrams
• CONTRIBUTING.md: Contribution guidelines and detailed feature development walkthrough
• MAINTAINERS_RUNBOOK.md: Maintenance tasks (releases, etc.)
• ai-memory/: AI agent memory bank (if using AI assistants)