Advanced developer docs #2569
Open
Advanced developer docs #2569
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds detailed pydoc and markdown documentation for all major ECS generator modules, following a consistent documentation pattern: Module Documentation (pydoc): - otel.py: OpenTelemetry integration and validation - markdown_fields.py: Markdown documentation generation - intermediate_files.py: Intermediate format generation - es_template.py: Elasticsearch template generation - csv_generator.py: CSV field reference export - beats.py: Beats field definition generation - ecs_helpers.py: Shared utility functions Each module now includes: - Comprehensive module-level docstrings explaining purpose and usage - Detailed function/method docstrings with Args, Returns, Examples - Clear explanations of key concepts and behaviors Markdown Guides (scripts/docs/): - README.md: Central documentation index and getting started guide - otel-integration.md: OTel semantic conventions integration - markdown-generator.md: Documentation generation with Jinja2 - intermediate-files.md: Flat and nested format representations - es-template.md: Elasticsearch template formats (composable & legacy) - csv-generator.md: CSV field reference for spreadsheets - beats-generator.md: Beats field definitions and default_field system - ecs-helpers.md: Utility functions reference and patterns Each guide includes: - Architecture diagrams and data flow explanations - Complete usage examples (CLI and programmatic) - Troubleshooting sections with common issues - Making changes guides for extensibility - Performance considerations - Testing strategies This documentation provides developers with everything needed to understand, modify, and extend the ECS build system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Significantly improved schema-pipeline.md to make complex concepts more accessible: Field Reuse Improvements: - Added visual ASCII diagrams showing before/after states of reuse operations - Detailed step-by-step examples of transitivity in action - Clear distinction between foreign reuse (transitive) and self-nesting (non-transitive) - Complete use case table explaining when to use each type - Reuse order explanation with concrete examples Subset Filtering Improvements: - Complete rewrite with motivation and benefits section - Visual representations of inclusion/exclusion - Comprehensive syntax guide with 4 different approaches - Three complete real-world subset examples (web, security, infrastructure) - Field options documentation (docs_only, index flags) - Multiple subset merging explained New Sections: - Quick Reference cheat sheets for rapid lookup - Common Patterns section with 4 practical examples - Enhanced troubleshooting with symptom → cause → solution format - Debugging commands with copy-paste Python snippets The documentation now uses visual learning, example-driven explanations, and troubleshooting-first approach to make field reuse and subset filtering concepts much easier to understand. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Added pydoc (docstrings) and markdown documentation for all Python scripts: Generator Scripts (scripts/generators/): - otel.py: OTel integration and validation - markdown_fields.py: Markdown documentation generation - intermediate_files.py: Intermediate format generation - es_template.py: Elasticsearch template generation - ecs_helpers.py: Shared utility functions - csv_generator.py: CSV field reference export - beats.py: Beats field definition generation Schema Processing (scripts/schema/): - loader.py: Schema loading and nesting - cleaner.py: Validation and normalization - finalizer.py: Field reuse and name calculation - visitor.py: Field traversal utilities - subset_filter.py: Subset filtering - exclude_filter.py: Exclude filtering Main Orchestrator: - generator.py: Complete pipeline documentation Documentation Files: - scripts/docs/README.md: Updated with all module links - scripts/docs/otel-integration.md: OTel integration guide - scripts/docs/markdown-generator.md: Markdown generation guide - scripts/docs/intermediate-files.md: Intermediate files guide - scripts/docs/es-template.md: Elasticsearch template guide - scripts/docs/ecs-helpers.md: Utility functions guide - scripts/docs/csv-generator.md: CSV generator guide - scripts/docs/beats-generator.md: Beats generator guide - scripts/docs/schema-pipeline.md: Complete pipeline documentation Documentation Style: - Google-style docstrings with Args, Returns, Raises, Examples - Comprehensive markdown guides with architecture, usage, troubleshooting - Visual examples and flow diagrams - Practical code examples - Common patterns and best practices This provides a complete developer reference for understanding, modifying, and extending the ECS generation pipeline. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
|
Documentation changes preview: https://docs-v3-preview.elastic.dev/elastic/ecs/pull/2569/reference/ |
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1. What does this PR do?
This adds advanced developer documentation for ECS tooling, which documents all steps of the generation pipeline, adds information on the ECS-OTel mapping process, adds pydoc to all functions. It also adds documentation on field re-use, subset and exclude filters.
2. Which ECS fields are affected/introduced?
N/A
3. Why is this change necessary?
To improve the ECS developer experience
4. Have you added/updated documentation?
YES
5. Have you built ECS and committed any newly generated files?
YES
6. Have you run the ECS validation tests locally?
YES
7. Anything else for the reviewers?
Commit Message
This adds complete developer-focused documentation to the ECS generation
pipeline, making the codebase significantly more accessible to contributors
and users.
This provides a complete reference for developers to understand, modify,
and extend the ECS generation pipeline.
🤖 Generated with Claude Code
Co-Authored-By: Claude [email protected]