diff --git a/README.md b/README.md index f9e94756b..2e711f17b 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@

- Scriptable interface to a powerful, multi-lingual language server. + The SWE that Never Sleeps

@@ -22,72 +22,48 @@
-[Codegen](https://docs.codegen.com) is a python library for manipulating codebases. +The Codegen SDK provides a programmatic interface to code agents provided by [Codegen](https://codegen.com). ```python -from codegen import Codebase - -# Codegen builds a complete graph connecting -# functions, classes, imports and their relationships -codebase = Codebase("./") - -# Work with code without dealing with syntax trees or parsing -for function in codebase.functions: - # Comprehensive static analysis for references, dependencies, etc. - if not function.usages: - # Auto-handles references and imports to maintain correctness - function.move_to_file("deprecated.py") -``` - -Write code that transforms code. Codegen combines the parsing power of [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) with the graph algorithms of [rustworkx](https://github.com/Qiskit/rustworkx) to enable scriptable, multi-language code manipulation at scale. +from codegen.agents.agent import Agent -## Installation and Usage +# Initialize the Agent with your organization ID and API token +agent = Agent( + org_id="YOUR_ORG_ID", # Find this at codegen.com/developer + token="YOUR_API_TOKEN", # Get this from codegen.com/developer + # base_url="https://codegen-sh-rest-api.modal.run", # Optional - defaults to production +) -We support +# Run an agent with a prompt +task = agent.run(prompt="Implement a new feature to sort users by last login.") -- Running Codegen in Python 3.12 - 3.13 (recommended: Python 3.13+) -- macOS and Linux - - macOS is supported - - Linux is supported on x86_64 and aarch64 with glibc 2.34+ - - Windows is supported via WSL. See [here](https://docs.codegen.com/building-with-codegen/codegen-with-wsl) for more details. -- Python, Typescript, Javascript and React codebases +# Check the initial status +print(task.status) -``` -# Install inside existing project -uv pip install codegen +# Refresh the task to get updated status (tasks can take time) +task.refresh() -# Install global CLI -uv tool install codegen --python 3.13 +# Check the updated status +print(task.status) -# Create a codemod for a given repo -cd path/to/repo -codegen init -codegen create test-function - -# Run the codemod -codegen run test-function - -# Create an isolated venv with codegen => open jupyter -codegen notebook +# Once task is complete, you can access the result +if task.status == "completed": + print(task.result) # Result often contains code, summaries, or links ``` -## Usage +## Installation and Usage -See [Getting Started](https://docs.codegen.com/introduction/getting-started) for a full tutorial. +Install the SDK using pip or uv: +```bash +pip install codegen +# or +uv pip install codegen ``` -from codegen import Codebase -``` - -## Troubleshooting - -Having issues? Here are some common problems and their solutions: -- **I'm hitting an UV error related to `[[ packages ]]`**: This means you're likely using an outdated version of UV. Try updating to the latest version with: `uv self update`. -- **I'm hitting an error about `No module named 'codegen.sdk.extensions.utils'`**: The compiled cython extensions are out of sync. Update them with `uv sync --reinstall-package codegen`. -- **I'm hitting a `RecursionError: maximum recursion depth exceeded` error while parsing my codebase**: If you are using python 3.12, try upgrading to 3.13. If you are already on 3.13, try upping the recursion limit with `sys.setrecursionlimit(10000)`. +Get started at [codegen.com](https://codegen.com) and get your API token at [codegen.com/developer](https://codegen.com/developer). -If you run into additional issues not listed here, please [join our slack community](https://community.codegen.com) and we'll help you out! +You can interact with your AI engineer via API, or chat with it in Slack, Linear, Github, or on our website. ## Resources @@ -96,18 +72,6 @@ If you run into additional issues not listed here, please [join our slack commun - [Contributing](CONTRIBUTING.md) - [Contact Us](https://codegen.com/contact) -## Why Codegen? - -Software development is fundamentally programmatic. Refactoring a codebase, enforcing patterns, or analyzing control flow - these are all operations that can (and should) be expressed as programs themselves. - -We built Codegen backwards from real-world refactors performed on enterprise codebases. Instead of starting with theoretical abstractions, we focused on creating APIs that match how developers actually think about code changes: - -- **Natural mental model**: Write transforms that read like your thought process - "move this function", "rename this variable", "add this parameter". No more wrestling with ASTs or manual import management. - -- **Battle-tested on complex codebases**: Handle Python, TypeScript, and React codebases with millions of lines of code. - -- **Built for advanced intelligences**: As AI developers become more sophisticated, they need expressive yet precise tools to manipulate code. Codegen provides a programmatic interface that both humans and AI can use to express complex transformations through code itself. - ## Contributing Please see our [Contributing Guide](CONTRIBUTING.md) for instructions on how to set up the development environment and submit contributions. diff --git a/docs/graph-sitter/about.mdx b/docs/graph-sitter/about.mdx new file mode 100644 index 000000000..d7ffe80a1 --- /dev/null +++ b/docs/graph-sitter/about.mdx @@ -0,0 +1,72 @@ +--- +title: "Codegen, Inc." +sidebarTitle: "About Us" +icon: "building" +iconType: "solid" +--- + + + +## Our Mission + +Our mission is to build fully-autonomous software engineering - the equivalent of self-driving cars for code. + +We believe the highest leverage path to autonomous development is enabling AI agents to "act via code." + +Just as self-driving cars need sophisticated sensors and controls to navigate the physical world, AI agents need powerful, precise tools to manipulate codebases. We're building that foundational layer: a programmatic interface that lets AI agents express complex code transformations through code itself. + +This approach creates a shared language that both humans and AI can use to: + +- Express powerful changes with precision and predictability +- Build sophisticated tools from primitive operations +- Create and maintain their own abstractions +- Scale transformations across massive codebases + +## The Team + +Based in San Francisco, we're a team of engineers and researchers passionate about: + +- Making large-scale code changes more accessible +- Building tools that work the way developers think +- Creating the infrastructure for AI-powered code manipulation +- Advancing the state of the art in program transformation + +## Open Source + +We believe in the power of open source software. Our core library, [codegen](https://github.com/codegen-sh/codegen-sdk), is freely available and open to contributions from the community. + +## Join Us + + + + We're hiring! Join us in building the future of code transformation. + + + Connect with other developers and share your Codegen experiences. + + + +## Connect with Us + + + + Follow us for updates and announcements + + + Connect with our team and stay updated on company news + + + + + Want to learn more about what we're building? Check out our [getting started + guide](/introduction/getting-started) or join our [community + Slack](https://community.codegen.com). + diff --git a/docs/graph-sitter/advanced-settings.mdx b/docs/graph-sitter/advanced-settings.mdx new file mode 100644 index 000000000..08f0e5472 --- /dev/null +++ b/docs/graph-sitter/advanced-settings.mdx @@ -0,0 +1,404 @@ +--- +title: "Advanced Settings" +sidebarTitle: "Advanced Settings" +icon: "memory" +iconType: "solid" +--- + +Codegen's [Codebase](/api-reference/core/Codebase) constructor accepts a `CodebaseConfig` object which is used to configure more advanced behaviors of the graph construction process. + +These flags are helpful for debugging problematic repos, optimizing Codegen's performance, or testing unreleased or experimental (potentially backwards-breaking) features. + + +**These are considered experimental features and may change in the future!** + +As such, they may have little to no testing or documentation. Many of these flags may also be unsupported in the future! + +If you need help, please visit our [community](/introduction/community). + + + +These configuration options are defined in [src/codegen/configs/models/codebase.py](https://github.com/codegen-sh/codegen/blob/develop/src/codegen/configs/models/codebase.py). + + +# Usage + +You can customize the behavior of the graph construction process when initializing a [Codebase](/api-reference/core/Codebase) by passing a `CodebaseConfig` object with the desired configuration flags. + +```python +from codegen import Codebase +from codegen.configs import CodebaseConfig + +# Initialize a Codebase with custom configuration +codebase = Codebase( + "", + config=CodebaseConfig( + flag1=..., + flag2=..., + ... + ) +) +``` + +# Table of Contents + +- [debug](#flag-debug) +- [verify-graph](#flag-verify-graph) +- [track-graph](#flag-track-graph) +- [method-usages](#flag-method-usages) +- [sync-enabled](#flag-sync-enabled) +- [full-range-index](#flag-full-range-index) +- [ignore-process-errors](#flag-ignore-process-errors) +- [disable-graph](#flag-disable-graph) +- [disable-file-parse](#flag-disable-file-parse) +- [exp-lazy-graph](#flag-exp-lazy-graph) +- [generics](#flag-generics) +- [import-resolution-paths](#flag-import-resolution-paths) +- [import-resolution-overrides](#flag-import-resolution-overrides) +- [py-resolve-syspath](#flag-py-resolve-syspath) +- [ts-dependency-manager](#flag-ts-dependency-manager) +- [ts-language-engine](#flag-ts-language-engine) +- [v8-ts-engine](#flag-v8-ts-engine) +- [unpacking-assignment-partial-removal](#flag-unpacking-assignment-partial-removal) + +# Configuration Flags + +## Flag: `debug` +> **Default: `False`** + +Enables verbose logging for debugging purposes. In its current form, it enables: +- Verbose logging when adding nodes to the graph +- Verbose logging during initial file parsing +- Additional assertions on graph creation +- Additional (costly) debug metrics on codebase construction +- etc. + + +This flag may be very noisy and significantly impact performance. It is generally not recommended to use. + + +## Flag: `verify_graph` +> **Default: `False`** + +Adds assertions for graph state during reset resync. Used to test and debug graph desyncs after a codebase reset. + +Runs `post_reset_validation` after a reset resync. + + +This is an internal debug flag. + + +## Flag: `track_graph` +> **Default: `False`** + +Keeps a copy of the original graph before a resync. Used in conjunction with `verify_graph` to test and debug graph desyncs. + +Original graph is saved as `ctx.old_graph`. + + +This is an internal debug flag. + + +## Flag: `method_usages` +> **Default: `True`** + +Enables and disables resolving method usages. + +**Example Codebase:** +```python +class Foo: + def bar(): + ... + +obj = Foo() +obj.bar() # Method Usage +``` + +**Codemod with `method_usages` on:** +```python +bar_func = codebase.get_class("Foo").get_method("bar") +len(bar_func.usages) # 1 +bar_func.usages # [obj.bar()] +``` + +**Codemod with `method_usages` off:** +```python +bar_func = codebase.get_class("Foo").get_method("bar") +len(bar_func.usages) # 0 +bar_func.usages # [] +``` + +Method usage resolution could be disabled for a marginal performance boost. However, it is generally recommended to leave it enabled. + +## Flag: `sync_enabled` +> **Default: `False`** + +Enables or disables graph sync during `codebase.commit`. + + +Implementation-specific details on sync graph can be found [here](https://github.com/codegen-sh/codegen/blob/develop/architecture/6.%20incremental-computation/C.%20Graph%20Recomputation.md). + + +This section won't go into the specific details of sync graph, but the general idea is that enabling sync graph will update the Codebase object to whatever new changes were made. + +**Example with `sync_enabled` on:** +```python +file = codebase.get_file(...) +file.insert_after("foobar = 1") +codebase.commit() + +foobar = codebase.get_symbol("foobar") +assert foobar # foobar is available after commit / graph sync +``` + +**Example with `sync_enabled` disabled:** +```python +file = codebase.get_file(...) +file.insert_after("foobar = 1") + +foobar = codebase.get_symbol("foobar", optional=True) +assert not foobar # foobar is not available after commit +``` + + +Enabling sync graph will have a performance impact on codebase commit, but will also unlock a bunch of operations that were previously not possible. + + +## Flag: `full_range_index` +> **Default: `False`** + +By default, Codebase maintains an internal range-to-node index for fast lookups. (i.e. `bytes 120 to 130 maps to node X`). +For optimization purposes, this only applies to nodes defined and handled by `parser.py`. + +Enabling `full_range_index` will create an additional index that maps **all** tree-sitter ranges to nodes. +This can be useful for debugging or when you need to build any applications that require a full range-to-node index (i.e. a codebase tree lookup). + + +This flag **significantly** increases memory usage! + + +## Flag: `ignore_process_errors` +> **Default: `True`** + +Controls whether to ignore errors that occur during external process execution (such as dependency manager or language engine). + +Disabling `ignore_process_errors` would make Codegen fail on errors that would otherwise be logged then ignored. + +## Flag: `disable_graph` +> **Default: `False`** + +Disables the graph construction process. Any operations that require the graph will no longer work. (In other words, this turns off import resolution and usage/dependency resolution) + +Functions that operate purely on AST such as getting and editing parameters or modifying function and class definitions will still work. + + +For codemods that do not require the graph (aka only AST/Syntax-level changes), **disabling graph parse could yield a 30%-40% decrease in parse time and memory usage**! + + +## Flag: `disable_file_parse` +> **Default: `False`** + +Disables **ALL** parsing, including file and graph parsing. This essentially treats all codebases as the "UNSUPPORTED" language mode. + +Nearly all functions except for editing primitives like `codebase.get_file` and `file.edit` will no longer work. + + +This flag is useful for any usages of Codegen that do **NOT** require any AST/CST/Graph parsing. (i.e. using Codegen purely as a file editing harness) + +If this is your use case, this **could decrease parse and memory usage by 95%.** + + +## Flag: `exp_lazy_graph` +> **Default: `False`** + +This experimental flag pushes the graph creation back until the graph is needed. This is an experimental feature and may have some unintended consequences. + +**Example Codemod:** +```python +from codegen import Codebase +from codegen.configs import CodebaseConfig + +# Enable lazy graph parsing +codebase = Codebase("", config=CodebaseConfig(exp_lazy_graph=True)) + +# The codebase object will be created immediately with no parsing done +# These all do not require graph parsing +codebase.files +codebase.directories +codebase.get_file("...") + +# These do require graph parsing, and will create the graph only if called +codebase.get_function("...") +codebase.get_class("...") +codebase.imports +``` + + +This may have a very slight performance boost. Use at your own risk! + + +## Flag: `generics` +> **Default: `True`** + +Enables and disables generic type resolution. + +**Example Codebase:** +```python +class Point: + def scale(cls, n: int): + pass + +class List[T](): + def pop(self) -> T: + ... + +l: List[Point] = [] +l.pop().scale(1) # Generic Usage +``` + +**Codemod with `generics` on:** +```python +bar_func = codebase.get_class("Point").get_method("scale") +len(bar_func.usages) # 1 +bar_func.usages # [l.pop().scale(1)] +``` + +**Codemod with `generics` off:** +```python +bar_func = codebase.get_class("Point").get_method("scale") +len(bar_func.usages) # 0 +bar_func.usages # [] +``` + + +Generic resolution is still largely WIP and experimental, and may not work in all cases. In some rare circumstances, disabling generics may result in a significant performance boost. + + +## Flag: `import_resolution_paths` +> **Default: `[]`** + +Controls alternative paths to resolve imports from. + +**Example Codebase:** +```python +# a/b/c/src.py +def update(): + pass + +# consumer.py +from c import src as operations + +operations.update() +``` + +**Codemod:** +```python +codebase.ctx.config.import_resolution_paths = ["a/b"] +``` + +## Flag: `import_resolution_overrides` +> **Default: `{}`** + +Controls import path overrides during import resolution. + +**Example** +`from a.b.c import d` with the override `a/b` -> `foo/bar` will internally resolve the import as `from foo.bar.c import d`. + +## Flag: `py_resolve_syspath` +> **Default: `False`** + +Enables and disables resolution of imports from `sys.path`. + + +For this to properly work, you must also set `allow_external` to `True`. + + +## Flag: `allow_external` +> **Default: `False`** + +Enables resolving imports, files, modules, and directories from outside of the repo path. + + +Turning this flag off may allow for bad actors to access files outside of the repo path! Use with caution! + + +## Flag: `ts_dependency_manager` +> **Default: `False`** + + +**This is an internal flag used for Codegen Cloud and should not be used externally!** + +This flag **WILL** nuke any existing `node_modules` folder! + + + +This flag also assumes many constants for Codegen Cloud. Very likely this will not work if run locally. + +Instead, just install `node_modules` as normal (either through `npm`, `pnpm`, or `yarn`) and skip this setting! + + +Enables Codegen's internal dependency installer for TypeScript. This will modify `package.json` and install the bare minimum set of installable dependencies. + + +More documentation on TypeScript dependency manager can be found [here](https://github.com/codegen-sh/codegen/blob/develop/architecture/external/dependency-manager.md) + + +## Flag: `ts_language_engine` +> **Default: `False`** + + +This feature was built primarily with Codegen Cloud in mind. As such, this assumes a valid NodeJS and TypeScript environment. + + +Enables using the TypeScript compiler to extract information from the codebase. Enables commands such as `inferred_return_type`. + + +This will increase memory usage and parsing time. Larger repos may even hit resource constraints with the bundled TypeScript compiler integration. + + +## Flag: `v8_ts_engine` +> **Default: `False`** + + +This feature flag requires `ts_language_engine` to be enabled as well. + + +Enables using the **V8-based TypeScript compiler** to extract information from the codebase. Enables commands such as `inferred_return_type`. + +The V8 implementation (as opposed to the default external-process based implementation) is less stable, but provides the entire TypeScript API to be used from within Codegen. + + +This will increase memory usage and parsing time. Larger repos may even hit resource constraints with the V8-based TypeScript compiler integration. + + +## Flag: `unpacking_assignment_partial_removal` +> **Default: `False`** + +Enables smarter removal of unpacking assignments. + +**Example Codebase:** +```python +a, b, c = (1, 2, 3) +``` + +**Codemod with `unpacking_assignment_partial_removal` on:** +```python +file = codebase.get_file(...) +b = file.get_symbol("b") +b.remove() +codebase.commit() + +file.symbols # [a, c] +file.source # "a, c = (1, 3)" +``` + +**Codemod with `unpacking_assignment_partial_removal` off:** +```python +file = codebase.get_file(...) +b = file.get_symbol("b") +b.remove() +codebase.commit() + +file.symbols # [] +file.source # "" +``` diff --git a/docs/graph-sitter/community.mdx b/docs/graph-sitter/community.mdx new file mode 100644 index 000000000..ed02a4276 --- /dev/null +++ b/docs/graph-sitter/community.mdx @@ -0,0 +1,48 @@ +--- +title: "Community & Contributing" +sidebarTitle: "Community" +icon: "people-group" +iconType: "solid" +--- + +import { + COMMUNITY_SLACK_URL, + CODEGEN_SDK_GITHUB_URL, +} from "/snippets/links.mdx"; + +Join the growing Codegen community! We're excited to have you be part of our journey to make codebase manipulation and transformation more accessible. + + + + Connect with the community, get help, and share your Codegen projects in our + active Slack workspace. + + + Star us on GitHub, report issues, submit PRs, and contribute to the project. + + + Follow us for updates, tips, and community highlights. + + + Learn how to use Codegen effectively with our comprehensive guides. + + + + + Please help us improve this library and documentation by submitting a PR! + + +## Contributing + +We welcome contributions of all kinds! Whether you're fixing a typo in documentation, reporting a bug, or implementing a new feature, we appreciate your help in making Codegen better. + +Check out our [Contributing Guide](https://github.com/codegen-sh/codegen-sdk/blob/develop/CONTRIBUTING.md) on GitHub to learn how to: + +- Set up your development environment +- Submit pull requests +- Report issues +- Contribute to documentation diff --git a/docs/graph-sitter/faq.mdx b/docs/graph-sitter/faq.mdx new file mode 100644 index 000000000..d7f3abfba --- /dev/null +++ b/docs/graph-sitter/faq.mdx @@ -0,0 +1,58 @@ +--- +title: "Frequently Asked Questions" +sidebarTitle: "FAQ" +icon: "square-question" +iconType: "solid" +--- + + + + Codegen currently parses two languages: + - [Python](/api-reference/python) + - [TypeScript](/api-reference/typescript) + + We're actively working on expanding language support based on community needs. + + Learn more about how Codegen handles language specifics in the [Language + Support](/building-with-codegen/language-support) guide. + + + Interested in adding support for your language? [Let us know](https://x.com/codegen) or [contribute](/introduction/community)! + + + + + Pretty much! Codegen is roughly on par with `mypy` and `tsc`. There are always edge cases in static analysis that are provably impossible to get (for example doing `eval()` on a string), but all of Codegen's APIs are intended to be exact unless otherwise specified. Please reach out if you find an edge case and we will do our best to patch it. + + + Yes! Codegen was developed on multmillion-line Python and Typescript codebases + and includes optimizations for handling large-scale transformations. + + For enterprise support, please reach out to [team@codegen.com](mailto:team@codegen.com) + + + + Yes - [by design](/introduction/guiding-principles#python-first-composability). + + Codegen works like any other python package. It works alongside your IDE, version control system, and other development tools. + + + Start by trying out Codegen, joining our [Slack community](https://community.codegen.com), and looking for + issues labeled "good first issue" on [GitHub](https://github.com/codegen-sh/codegen-sdk). We welcome contributions to + documentation, examples, and code improvements. + + + Yes, Codegen is [open source](https://github.com/codegen-sh/codegen-sdk) and free to use under the [Apache 2.0 + license](https://github.com/codegen-sh/codegen-sdk?tab=Apache-2.0-1-ov-file). + You can use it for both personal and commercial projects. + + + The best places to get help are: + 1. Our community [Slack channel](https://community.codegen.com) + 2. [GitHub issues](https://github.com/codegen-sh/codegen-sdk) for bug reports + 3. Reach out to us on [Twitter](https://x.com/codegen) + + diff --git a/docs/introduction/getting-started.mdx b/docs/graph-sitter/getting-started.mdx similarity index 100% rename from docs/introduction/getting-started.mdx rename to docs/graph-sitter/getting-started.mdx diff --git a/docs/graph-sitter/guiding-principles.mdx b/docs/graph-sitter/guiding-principles.mdx new file mode 100644 index 000000000..53201e5c1 --- /dev/null +++ b/docs/graph-sitter/guiding-principles.mdx @@ -0,0 +1,75 @@ +--- +title: "Guiding Principles" +sidebarTitle: "Principles" +icon: "compass" +iconType: "solid" +--- + +Codegen was developed by working backwards from real-world, large-scale codebase migrations. Instead of starting with abstract syntax trees and parser theory, we started with the question: "How do developers actually think about code changes?" + +This practical origin led to four core principles that shape Codegen's design: + +## Intuitive APIs + +Write code that reads like natural language, without worrying about abstract syntax trees or parser internals. Codegen provides high-level APIs that map directly to the transformations developers want to perform: + +```python +# Methods that read like English +function.rename("new_name") # Not ast.update_node(function_node, "name", "new_name") +function.move_to_file("new_file.py") # Not ast.relocate_node(function_node, "new_file.py") + +# Clean, readable properties +if function.is_async: # Not ast.get_node_attribute(function_node, "async") + print(function.name) # Not ast.get_node_name(function_node) + +# Natural iteration patterns +for usage in function.usages: # Not ast.find_references(function_node) + print(f"Used in {usage.file.name}") +``` + +## No Sharp Edges + +Focus on your high-level intent while Codegen handles the intricate details. + +Codegen operations handle the edge cases - it should be hard to break lint. + +```python +# Moving a function? Codegen handles: +function.move_to_file("new_file.py") +# ✓ Updating all import statements +# ✓ Preserving dependencies +# ✓ Maintaining references +# ✓ Fixing relative imports +# ✓ Resolving naming conflicts + +# Renaming a symbol? Codegen manages: +class_def.rename("NewName") +# ✓ Updating all usages +# ✓ Handling string references +# ✓ Preserving docstrings +# ✓ Maintaining inheritance +``` + +## Performance through Pre-Computation + +Codegen frontloads as much as possible to enable fast, efficient transformations. + +It is built with the insight that each codebase only needs to be parsed once per commit. + + + Learn more about parsing the codebase graph in the [How it + Works](/introduction/how-it-works) guide. + + +## Python-First Composability + +Codegen embraces Python's strength as a "glue language" - its ability to seamlessly integrate different tools and APIs. This makes it natural to compose Codegen with your existing toolchain: + +- Build complex transforms by combining simpler operations +- Integrate Codegen with your existing tools (linters, type checkers, test frameworks, AI tools) + + + Python's rich ecosystem makes it ideal for code manipulation tasks. Codegen is + designed to be one tool in your toolbox, not a replacement for your entire + workflow. + diff --git a/docs/graph-sitter/how-it-works.mdx b/docs/graph-sitter/how-it-works.mdx new file mode 100644 index 000000000..8777a5be4 --- /dev/null +++ b/docs/graph-sitter/how-it-works.mdx @@ -0,0 +1,89 @@ +--- +title: "Under the Hood" +sidebarTitle: "How it Works" +icon: "gear" +iconType: "solid" +subtitle: "How Codegen's codebase graph works" +--- + +Codegen performs advanced static analysis to build a rich graph representation of your codebase. This pre-computation step analyzes dependencies, references, types, and control flow to enable fast and reliable code manipulation operations. + + + Codegen is built on top of + [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) and + [rustworkx](https://github.com/Qiskit/rustworkx) and has implemented most + language server features from scratch. + + + Codegen is open source. Check out the [source + code](https://github.com/codegen-sh/codegen-sdk) to learn more! + + +## The Codebase Graph + +At the heart of Codegen is a comprehensive graph representation of your code. When you initialize a [Codebase](/api-reference/core/Codebase), it performs static analysis to construct a rich graph structure connecting code elements: + +```python +# Initialize and analyze the codebase +from codegen import Codebase +codebase = Codebase("./") + +# Access pre-computed relationships +function = codebase.get_symbol("process_data") +print(f"Dependencies: {function.dependencies}") # Instant lookup +print(f"Usages: {function.usages}") # No parsing needed +``` + +### Building the Graph + +Codegen's graph construction happens in two stages: + +1. **AST Parsing**: We use [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) as our foundation for parsing code into Abstract Syntax Trees. Tree-sitter provides fast, reliable parsing across multiple languages. + +2. **Multi-file Graph Construction**: Custom parsing logic, implemented in [rustworkx](https://github.com/Qiskit/rustworkx) and Python, analyzes these ASTs to construct a more sophisticated graph structure. This graph captures relationships between [symbols](/building-with-codegen/symbol-api), [files](/building-with-codegen/files-and-directories), [imports](/building-with-codegen/imports), and more. + +### Performance Through Pre-computation + +Pre-computing a rich index enables Codegen to make certain operations very fast that that are relevant to refactors and code analysis: + +- Finding all usages of a symbol +- Detecting circular dependencies +- Analyzing the dependency graphs +- Tracing call graphs +- Static analysis-based code retrieval for RAG +- ...etc. + + + Pre-parsing the codebase enables constant-time lookups rather than requiring + re-parsing or real-time analysis. + + +## Multi-Language Support + +One of Codegen's core principles is that many programming tasks are fundamentally similar across languages. + +Currently, Codegen supports: + +- [Python](/api-reference/python) +- [TypeScript](/api-reference/typescript) +- [React & JSX](/building-with-codegen/react-and-jsx) + + + Learn about how Codegen handles language specifics in the [Language + Support](/building-with-codegen/language-support) guide. + + +We've started with these ecosystems but designed our architecture to be extensible. The graph-based approach provides a consistent interface across languages while handling language-specific details under the hood. + +## Build with Us + +Codegen is just getting started, and we're excited about the possibilities ahead. We enthusiastically welcome contributions from the community, whether it's: + +- Adding support for new languages +- Implementing new analysis capabilities +- Improving performance +- Expanding the API +- Adding new transformations +- Improving documentation + +Check out our [community guide](/introduction/community) to get involved! diff --git a/docs/graph-sitter/ide-usage.mdx b/docs/graph-sitter/ide-usage.mdx new file mode 100644 index 000000000..3c58d39ab --- /dev/null +++ b/docs/graph-sitter/ide-usage.mdx @@ -0,0 +1,178 @@ +--- +title: "Using Codegen in Your IDE" +sidebarTitle: "IDE Usage" +icon: "window" +iconType: "solid" +--- + +Get up and running with Codegen programs in IDEs like VSCode, Cursor and PyCharm. + +Make sure to [install and initialize](/introduction/installation) Codegen with `codegen init` + +## Configuring your IDE Interpreter + +Codegen creates a custom Python environment in `.codegen/.venv`. Configure your IDE to use this environment for the best development experience. + + + + 1. Install the VSCode Python Extensions for LSP and debugging support. We recommend Python, Pylance and Python Debugger for the best experience. + + 2. Open the Command Palette (Cmd/Ctrl + Shift + P) + 3. Type "Python: Select Interpreter" + + 4. Choose "Enter interpreter path" + 5. Navigate to and select: + ```bash + .codegen/.venv/bin/python + ``` + + Alternatively, create a `.vscode/settings.json`: + ```json + { + "python.defaultInterpreterPath": "${workspaceFolder}/.codegen/.venv/bin/python", + "python.analysis.extraPaths": [ + "${workspaceFolder}/.codegen/.venv/lib/python3.12/site-packages" + ] + } + ``` + + + + 1. Open PyCharm Settings/Preferences + 2. Navigate to "Project > Python Interpreter" + 3. Click the gear icon ⚙️ and select "Add" + 4. Choose "Existing Environment" + 5. Set interpreter path to: + ```bash + .codegen/.venv/bin/python + ``` + + + + + +## MCP Server Setup +This is an optional step but highly recommended if your IDE supports MCP support and you use AI Agents. +The MCP server is a local server that allows your AI Agent to interact with the Codegen specific tools, +it will allow an agent to: +- ask an expert to create a codemod +- improve a codemod +- get setup instructions + +### IDE Configuration +#### Cline +Add this to your cline_mcp_settings.json: +```json +{ + "mcpServers": { + "codegen-cli": { + "command": "uv", + "args": [ + "--directory", + "/codegen-sdk/src/codegen/cli/mcp", + "run", + "server.py" + ] + } + } +} +``` + + +#### Cursor: +Under the `Settings` > `Feature` > `MCP Servers` section, click "Add New MCP Server" and add the following: + +``` +Name: codegen-mcp +Type: Command +Command: uv --directory /codegen-sdk/src/codegen/cli/mcp run server.py +``` + + +## Index Codegen Docs +#### Cursor: +If you use Cursor you'll be able to configure the IDE to index the Codegen docs. To do so go to `Settings` > `Features` > `Docs` +and then click on `Add new docs`. We recommend using this url to index the API reference: +``` +https://docs.codegen.com/api-reference/index +``` + + +## Create a New Codemod + +Generate the boilerplate for a new code manipulation program using [codegen create](/cli/create): + +```bash +codegen create organize-types \ + -d "Move all TypeScript types to \ + into a centralized types.ts file" +``` + + + Passing in `-d --description` will get an LLM expert to compose an initial version for you. This requires a Github account registered on [codegen.sh](https://codegen.sh) + + +This will: +1. Create a new codemod in `.codegen/codemods/organize_types/` +2. Generate a custom `system-prompt.txt` based on your task +3. Set up the basic structure for your program + + +The generated codemod includes type hints and docstrings, making it easy to get IDE autocompletion and documentation. + + +## Iterating with Chat Assistants + +When you do `codegen init`, you will receive a [system prompt optimized for AI consumption](/introduction/work-with-ai) at `.codegen/codegen-system-prompt.txt`. + +If you reference this file in "chat" sessions with Copilot, Cursor, Cody, etc., the assistant will become fluent in Codegen. + + + + Collaborating with Cursor's assistant and the Codegen system prompt + + +In addition, when you [create](/cli/create) a codemod with "-d", Codegen generates an optimized system prompt in `.codegen/codemods/{name}/{name}-system-prompt.txt`. This prompt contains: +- Relevant Codegen API documentation +- Examples of relevant transformations +- Context about your specific task + + +You can also drag and drop the system prompt ([available here](/introduction/work-with-ai))file directly into chat windows like ChatGPT or Claude for standalone help. + + +## Running and Testing Codemods + +```bash +# Run => write changes to disk +codegen run organize-types + +# Reset changes on disk +codegen reset +``` + +You can also run the program directly via `.codegen/.venv/bin/python path/to/codemod.py` or via your editor's debugger + +## Viewing Changes + +We recommend viewing changes in your IDE's native diff editor. + + +## What's Next + + + + See real-world examples of codemods in action. + + + Learn about Codegen's core concepts and features + + diff --git a/docs/graph-sitter/installation.mdx b/docs/graph-sitter/installation.mdx new file mode 100644 index 000000000..41188c64b --- /dev/null +++ b/docs/graph-sitter/installation.mdx @@ -0,0 +1,125 @@ +--- +title: "Installation" +sidebarTitle: "Installation" +icon: "download" +iconType: "solid" +--- + +Install and set up Codegen in your development environment. + +#### We currently support: +- Running Codegen in Python 3.12 - 3.13 (recommended: Python 3.13+) +- macOS and Linux + - macOS is supported + - Linux is supported on x86_64 and aarch64 with glibc 2.34+ + - Windows is supported via WSL. See [here](https://docs.codegen.com/building-with-codegen/codegen-with-wsl) for more details. +- Python, Typescript, Javascript and React codebases + +## Prerequisites + +We recommend using [uv](https://github.com/astral-sh/uv) for installation. If you haven't installed `uv` yet: +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +## Installing Codegen + +```bash +uv tool install codegen --python 3.13 +``` + + + +This makes the `codegen` command available globally in your terminal, while keeping its dependencies isolated. + + +## Quick Start + +Let's walk through a minimal example of using Codegen in a project: + +1. Navigate to your repository: + ```bash + cd path/to/your/project + ``` + +2. Initialize Codegen in your project with [codegen init](/cli/init): + ```bash + codegen init + ``` + + This creates a `.codegen/` directory with: + ```bash + .codegen/ + ├── .venv/ # Python virtual environment (gitignored) + ├── config.toml # Project configuration + ├── codemods/ # Your codemod implementations + ├── jupyter/ # Jupyter notebooks for exploration + └── codegen-system-prompt.txt # AI system prompt + ``` + +3. Create your first codemod with [codegen create](/cli/create): + ```bash + codegen create organize-imports \ + -d "Sort and organize imports according to PEP8" + ``` + + The `-d` flag in `codegen create` generates an AI-powered implementation. This requires a Github account registered on [codegen.sh](https://codegen.sh) + + + + +4. Run your codemod with [codegen run](/cli/run): + ```bash + codegen run organize-imports + ``` + +5. Reset any filesystem changes (excluding `.codegen/*`) with [codegen reset](/cli/reset): + ```bash + codegen reset + ``` + +## Troubleshooting + +Having issues? Here are some common problems and their solutions: + +- **I'm hitting an UV error related to `[[ packages ]]`**: This means you're likely using an outdated version of UV. Try updating to the latest version with: `uv self update`. +- **I'm hitting an error about `No module named 'codegen.sdk.extensions.utils'`**: The compiled cython extensions are out of sync. Update them with `uv sync --reinstall-package codegen`. +- **I'm hitting a `RecursionError: maximum recursion depth exceeded` error while parsing my codebase**: If you are using python 3.12, try upgrading to 3.13. If you are already on 3.13, try upping the recursion limit with `sys.setrecursionlimit(10000)`. + + +For more help, join our [community Slack](/introduction/community) or check the [FAQ](/introduction/faq). + + +## Next Steps + + + + Learn how to use Codegen effectively in VSCode, Cursor, and other IDEs. + + + Follow step-by-step tutorials for common code transformation tasks. + + + Leverage AI assistants like Copilot, Cursor and Devin + + + Learn more about building with Codegen + + + diff --git a/docs/graph-sitter/overview.mdx b/docs/graph-sitter/overview.mdx new file mode 100644 index 000000000..4d428fc71 --- /dev/null +++ b/docs/graph-sitter/overview.mdx @@ -0,0 +1,159 @@ +--- +title: "Codegen" +sidebarTitle: "Overview" +icon: "code" +iconType: "solid" +--- + +[Codegen](https://github.com/codegen-sh/codegen-sdk) is a python library for manipulating codebases. + +It provides a scriptable interface to a powerful, multi-lingual language server built on top of [Tree-sitter](https://tree-sitter.github.io/tree-sitter/). + +```python +from codegen import Codebase + +# Codegen builds a complete graph connecting +# functions, classes, imports and their relationships +codebase = Codebase("./") + +# Work with code without dealing with syntax trees or parsing +for function in codebase.functions: + # Comprehensive static analysis for references, dependencies, etc. + if not function.usages: + # Auto-handles references and imports to maintain correctness + function.remove() + +# Fast, in-memory code index +codebase.commit() +``` + + + +Codegen handles complex refactors while maintaining correctness, enabling a broad set of advanced code manipulation programs. + + +Codegen works with both Python and Typescript/JSX codebases. Learn more about language support [here](/building-with-codegen/language-support). + +## Quick Started + + +Codegen requires Python 3.12 - 3.13 (recommended: Python 3.13+). + + +### Using UV (Recommended) +```bash +uv tool install codegen --python 3.13 +``` + +### Using Pipx + + +Pipx is not officially supported by Codegen, but it should still work. + + +```bash +pipx install codegen +``` + + +For further & more in depth installation instructions, see the [installation guide](/introduction/installation). + + +## What can I do with Codegen? + +Codegen's simple yet powerful APIs enable a range of applications, including: + + + + Create an intelligent agent that can analyze and manipulate your codebase using natural language. + + + Generate interactive visualizations of your codebase's structure, dependencies, and relationships. + + + Create high-quality training data for fine-tuning LLMs on your codebase. + + + Create powerful code transformations to automate large-scale changes. + + + +See below for an example call graph visualization generated with Codegen. + + + + + +View source code on [modal/modal-client](https://github.com/modal-labs/modal-client/blob/cbac0d80dfd98588027ecd21850152776be3ab82/modal/client.py#L70). View codemod on [codegen.sh](https://www.codegen.sh/codemod/66e2e195-ceec-4935-876a-ed4cfc1731c7/public/diff) + + +## Get Started + +import { + COMMUNITY_SLACK_URL, + CODEGEN_SDK_GITHUB_URL, +} from "/snippets/links.mdx"; + + + + Follow our step-by-step tutorial to start manipulating code with Codegen. + + + Learn how to use Codegen for common code transformation tasks. + + + Star us on GitHub and contribute to the project. + + + Get help and connect with the Codegen community. + + + +## Why Codegen? + +Many software engineering tasks - refactors, enforcing patterns, analyzing control flow, etc. - are fundamentally programmatic operations. Yet the tools we use to express these transformations often feel disconnected from how we think about code. + +Codegen was engineered backwards from real-world refactors we performed for enterprises at [Codegen, Inc.](/introduction/about). Instead of starting with theoretical abstractions, we built the set of APIs that map directly to how humans and AI think about code changes: + +- **Natural Mental Model**: Express transformations through high-level operations that match how you reason about code changes, not low-level text or AST manipulation. +- **Clean Business Logic**: Let the engine handle the complexities of imports, references, and cross-file dependencies. +- **Scale with Confidence**: Make sweeping changes across large codebases consistently across Python, TypeScript, JavaScript, and React. + +As AI becomes increasingly sophisticated, we're seeing a fascinating shift: AI agents aren't bottlenecked by their ability to understand code or generate solutions. Instead, they're limited by their ability to efficiently manipulate codebases. The challenge isn't the "brain" - it's the "hands." + +We built Codegen with a key insight: future AI agents will need to ["act via code,"](/blog/act-via-code) building their own sophisticated tools for code manipulation. Rather than generating diffs or making direct text changes, these agents will: + +1. Express transformations as composable programs +2. Build higher-level tools by combining primitive operations +3. Create and maintain their own abstractions for common patterns + +This creates a shared language that both humans and AI can reason about effectively, making code changes more predictable, reviewable, and maintainable. Whether you're a developer writing a complex refactoring script or an AI agent building transformation tools, Codegen provides the foundation for expressing code changes as they should be: through code itself. diff --git a/docs/graph-sitter/work-with-ai.mdx b/docs/graph-sitter/work-with-ai.mdx new file mode 100644 index 000000000..c826aad75 --- /dev/null +++ b/docs/graph-sitter/work-with-ai.mdx @@ -0,0 +1,86 @@ +--- +title: "Working with AI" +sidebarTitle: "AI Integration" +icon: "microchip" +iconType: "solid" +--- + +Codegen is designed to be used with AI assistants. This document describes how to use Codegen with common AI tools, including Copilot, Cursor, Devin and more. + +## System Prompt + +Codegen provides a `.txt` file that you can drag-and-drop into any chat assistant. This is roughly 60k tokens and will enable chat assistants like, ChatGPT, Claude 3.5 etc. to build effectively with Codegen. + +import { + CODEGEN_SYSTEM_PROMPT +} from "/snippets/links.mdx"; + + + Download System Prompt + + +Learn about leveraging this in IDE chat assistants like Cursor [here](/introduction/ide-usage#iterating-with-chat-assistants) + +## Generating System Prompts + +The [Codegen CLI](/cli/about) provides commands to generate `.md` files that can be fed to any AI assistant for more accurate and contextual help. + +When you create a new codemod via [codegen create](/cli/create): + +```bash +codegen create delete-dead-imports . --description "Delete unused imports" +``` + +Codegen automatically generates an optimized ["system prompt"](https://news.ycombinator.com/item?id=37880023) that includes: + +- An introduction to Codegen +- Codegen API documentation +- Examples of relevant transformations + +You can find this generated prompt in the `.codegen/prompts/-system-prompt.md` file. + + + All contents of the `.codegen/prompts` directory are by default ignored the + `.gitignore` file. after running [codegen init](/cli/init) + + +This `.md` file can be used with any AI assistant (Claude, GPT-4, etc.) to get more accurate and contextual help. + +## Example Workflow + + + + Use the [create command](/cli/create) with a detailed description of what you want to accomplish: + ```bash + codegen create modernize-components . --description "Convert class components to functional components with hooks" + ``` + + + Check the AI context that Codegen generated for your transformation: ```bash + cat codegen-sh/codemods/modernize-components/prompt.md ``` + + + + Reference your codemod when asking questions to get contextual help: ``` + @codegen-sh/codemods/modernize-components How should I handle + componentDidMount? ``` + + + + The AI will understand you're working on React modernization and provide relevant suggestions about using useEffect hooks and other modern React patterns. + + + +## Copilot, Cursor and Windsurf (IDEs) + +When using IDE chat assistants, you can leverage Codegen's context by mentioning your codemod in composer mode: + +```bash +@.codegen/codemods/upgrade-react18 @.codegen/prompts/system-prompt.md +``` + +This will ensure that the IDE's native chat model is aware of the APIs and common patterns for Codegen. + +## Devin, OpenHands and Semi-autonomous Code Agents + +Coming soon! diff --git a/docs/introduction/faq.mdx b/docs/introduction/faq.mdx index d7f3abfba..a2accd1de 100644 --- a/docs/introduction/faq.mdx +++ b/docs/introduction/faq.mdx @@ -7,52 +7,50 @@ iconType: "solid" - Codegen currently parses two languages: - - [Python](/api-reference/python) - - [TypeScript](/api-reference/typescript) - - We're actively working on expanding language support based on community needs. - - Learn more about how Codegen handles language specifics in the [Language - Support](/building-with-codegen/language-support) guide. - - - Interested in adding support for your language? [Let us know](https://x.com/codegen) or [contribute](/introduction/community)! - - + The Codegen AI agent leverages modern large language models (LLMs) for code + understanding and generation. This means it can generally handle tasks + involving any programming language, configuration format (like JSON, YAML), + documentation (like Markdown), or other text-based files that current LLMs + are proficient with. If you have specific needs or find limitations with a + particular language or format, please let us know! - - Pretty much! Codegen is roughly on par with `mypy` and `tsc`. There are always edge cases in static analysis that are provably impossible to get (for example doing `eval()` on a string), but all of Codegen's APIs are intended to be exact unless otherwise specified. Please reach out if you find an edge case and we will do our best to patch it. + + The Codegen agent uses large language models to understand and modify code. + While powerful, its understanding isn't based on formal static analysis and + may not always be perfectly exact or catch all edge cases like a traditional + compiler or linter might. It aims for practical correctness based on the + provided context and instructions. - Yes! Codegen was developed on multmillion-line Python and Typescript codebases - and includes optimizations for handling large-scale transformations. + Yes! Codegen's agent is designed to work effectively on large, real-world + codebases. You can provide context and specific instructions to help it + navigate complex projects. - For enterprise support, please reach out to [team@codegen.com](mailto:team@codegen.com) + For enterprise use cases and support, please reach out to + [team@codegen.com](mailto:team@codegen.com) - - Yes - [by design](/introduction/guiding-principles#python-first-composability). - - Codegen works like any other python package. It works alongside your IDE, version control system, and other development tools. + + Yes. The Codegen SDK is a standard Python package (`pip install codegen`). + You can import and use it in your Python scripts, CI/CD pipelines, or any + other development tool that can execute Python code. - Start by trying out Codegen, joining our [Slack community](https://community.codegen.com), and looking for - issues labeled "good first issue" on [GitHub](https://github.com/codegen-sh/codegen-sdk). We welcome contributions to - documentation, examples, and code improvements. - - - Yes, Codegen is [open source](https://github.com/codegen-sh/codegen-sdk) and free to use under the [Apache 2.0 - license](https://github.com/codegen-sh/codegen-sdk?tab=Apache-2.0-1-ov-file). - You can use it for both personal and commercial projects. + Start by trying out the Codegen agent and SDK, joining our [Slack + community](https://community.codegen.com), and reporting any issues or + feedback on [GitHub](https://github.com/codegen-sh/codegen-sdk). We welcome + contributions to documentation, examples, and SDK improvements. - The best places to get help are: - 1. Our community [Slack channel](https://community.codegen.com) - 2. [GitHub issues](https://github.com/codegen-sh/codegen-sdk) for bug reports - 3. Reach out to us on [Twitter](https://x.com/codegen) + The best places to get help are: 1. Our community [Slack + channel](https://community.codegen.com) 2. [GitHub + issues](https://github.com/codegen-sh/codegen-sdk) for bug reports or SDK + feature requests 3. Reach out to us on [Twitter](https://x.com/codegen) diff --git a/docs/introduction/overview.mdx b/docs/introduction/overview.mdx index 4d428fc71..8e945f419 100644 --- a/docs/introduction/overview.mdx +++ b/docs/introduction/overview.mdx @@ -5,112 +5,54 @@ icon: "code" iconType: "solid" --- -[Codegen](https://github.com/codegen-sh/codegen-sdk) is a python library for manipulating codebases. +The [Codegen SDK](https://github.com/codegen-sh/codegen-sdk) enables developers to programmatically interact with [Codegen](https://codegen.com) SWE agents via API. -It provides a scriptable interface to a powerful, multi-lingual language server built on top of [Tree-sitter](https://tree-sitter.github.io/tree-sitter/). + + Go to [developer settings](https://codegen.sh/token) to generate an API token + ```python -from codegen import Codebase +from codegen import Agent -# Codegen builds a complete graph connecting -# functions, classes, imports and their relationships -codebase = Codebase("./") +# Initialize the Agent with your organization ID and API token +agent = Agent(org_id="...", token="...") -# Work with code without dealing with syntax trees or parsing -for function in codebase.functions: - # Comprehensive static analysis for references, dependencies, etc. - if not function.usages: - # Auto-handles references and imports to maintain correctness - function.remove() +# Run an agent with a prompt +task = agent.run(prompt="Leave a review on PR #123") -# Fast, in-memory code index -codebase.commit() -``` - - - -Codegen handles complex refactors while maintaining correctness, enabling a broad set of advanced code manipulation programs. - +# Check the initial status +print(task.status) -Codegen works with both Python and Typescript/JSX codebases. Learn more about language support [here](/building-with-codegen/language-support). +# Refresh the task to get updated status (tasks can take time) +task.refresh() -## Quick Started +if task.status == "completed": + print(task.result) # Result often contains code, summaries, or links +``` -Codegen requires Python 3.12 - 3.13 (recommended: Python 3.13+). + Codegen agents can research code, create PRs, modify Linear tickets, and more. -### Using UV (Recommended) -```bash -uv tool install codegen --python 3.13 -``` - -### Using Pipx +## Installation - -Pipx is not officially supported by Codegen, but it should still work. - +Install the SDK using pip or uv: ```bash -pipx install codegen +pip install codegen +# or +uv pip install codegen ``` - -For further & more in depth installation instructions, see the [installation guide](/introduction/installation). - - -## What can I do with Codegen? +## What can I do with the Codegen SDK? -Codegen's simple yet powerful APIs enable a range of applications, including: +The Codegen SDK is your gateway to programmatically interacting with your AI Software Engineer. You can use it to: - - - Create an intelligent agent that can analyze and manipulate your codebase using natural language. - - - Generate interactive visualizations of your codebase's structure, dependencies, and relationships. - - - Create high-quality training data for fine-tuning LLMs on your codebase. - - - Create powerful code transformations to automate large-scale changes. - - +- **Automate development tasks**: Assign tasks like implementing features, fixing bugs, writing tests, or improving documentation to the agent. +- **Integrate AI into your workflows**: Trigger agent tasks from your CI/CD pipelines, scripts, or other development tools. +- **Provide context and guidance**: Supply the agent with specific instructions, relevant code snippets, or background information to ensure it performs tasks according to your requirements. -See below for an example call graph visualization generated with Codegen. - - - - - -View source code on [modal/modal-client](https://github.com/modal-labs/modal-client/blob/cbac0d80dfd98588027ecd21850152776be3ab82/modal/client.py#L70). View codemod on [codegen.sh](https://www.codegen.sh/codemod/66e2e195-ceec-4935-876a-ed4cfc1731c7/public/diff) - +Essentially, the SDK allows you to leverage Codegen's AI capabilities wherever you can run Python code. ## Get Started @@ -120,12 +62,11 @@ import { } from "/snippets/links.mdx"; - - Follow our step-by-step tutorial to start manipulating code with Codegen. + + Sign up for a free account and get your API token. + + + Get help and connect with the Codegen community. Learn how to use Codegen for common code transformation tasks. @@ -133,27 +74,4 @@ import { Star us on GitHub and contribute to the project. - - Get help and connect with the Codegen community. - - -## Why Codegen? - -Many software engineering tasks - refactors, enforcing patterns, analyzing control flow, etc. - are fundamentally programmatic operations. Yet the tools we use to express these transformations often feel disconnected from how we think about code. - -Codegen was engineered backwards from real-world refactors we performed for enterprises at [Codegen, Inc.](/introduction/about). Instead of starting with theoretical abstractions, we built the set of APIs that map directly to how humans and AI think about code changes: - -- **Natural Mental Model**: Express transformations through high-level operations that match how you reason about code changes, not low-level text or AST manipulation. -- **Clean Business Logic**: Let the engine handle the complexities of imports, references, and cross-file dependencies. -- **Scale with Confidence**: Make sweeping changes across large codebases consistently across Python, TypeScript, JavaScript, and React. - -As AI becomes increasingly sophisticated, we're seeing a fascinating shift: AI agents aren't bottlenecked by their ability to understand code or generate solutions. Instead, they're limited by their ability to efficiently manipulate codebases. The challenge isn't the "brain" - it's the "hands." - -We built Codegen with a key insight: future AI agents will need to ["act via code,"](/blog/act-via-code) building their own sophisticated tools for code manipulation. Rather than generating diffs or making direct text changes, these agents will: - -1. Express transformations as composable programs -2. Build higher-level tools by combining primitive operations -3. Create and maintain their own abstractions for common patterns - -This creates a shared language that both humans and AI can reason about effectively, making code changes more predictable, reviewable, and maintainable. Whether you're a developer writing a complex refactoring script or an AI agent building transformation tools, Codegen provides the foundation for expressing code changes as they should be: through code itself. diff --git a/docs/mint.json b/docs/mint.json index 737c098ee..091a684a7 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -1,396 +1,95 @@ { - "$schema": "https://mintlify.com/schema.json", - "name": "Codegen", - "logo": { - "dark": "https://cdn.prod.website-files.com/67070304751b9b01bf6a161c/679bcf45a3e32761c42b324b_Codegen_Logomark_Dark.svg", - "light": "https://cdn.prod.website-files.com/67070304751b9b01bf6a161c/679bcf45bf55446746125835_Codegen_Logomark_Light.svg" - }, - "modeToggle": { - "default": "dark" - }, - "metadata": { - "og:site_name": "Codegen", - "og:title": "Codegen - Manipulate Code at Scale", - "og:description": "A scriptable interface to a powerful, multi-lingual language server built on top of Tree-sitter.", - "og:url": "https://docs.codegen.com", - "og:locale": "en_US", - "og:logo": "https://i.imgur.com/f4OVOqI.png", - "article:publisher": "Codegen, Inc.", - "twitter:site": "@codegen" - }, - "favicon": "/favicon.svg", - "colors": { - "primary": "#a277ff", - "light": "#a277ff", - "dark": "#a277ff", - "anchors": { - "from": "#61ffca", - "to": "#61ffca" - } - }, - "theme": "prism", - "background": { - "style": "gradient" - }, - "analytics": { - "posthog": { - "apiKey": "phc_GLxaINoQJnuyCyxDmTciQqzdKBYFVDkY7bRBO4bDdso" - } - }, - "feedback": { - "thumbsRating": true - }, - "topbarCtaButton": { - "name": "GitHub", - "url": "https://github.com/codegen-sh/codegen-sdk" - }, - "tabs": [ - { - "name": "API Reference", - "url": "/api-reference" - }, - { - "name": "CLI", - "url": "/cli" - }, - { - "name": "Blog", - "url": "/blog" - }, - { - "name": "Changelog", - "url": "/changelog" - }, - { - "name": "codegen", - "url": "/gen" - } - ], - "navigation": [ - { - "group": "Introduction", - "pages": [ - "introduction/overview", - "introduction/getting-started", - "introduction/installation", - "introduction/ide-usage", - "introduction/work-with-ai", - "introduction/how-it-works", - "introduction/advanced-settings", - "introduction/guiding-principles", - "introduction/community", - "introduction/about", - "introduction/faq" - ] - }, - { - "group": "Tutorials", - "pages": [ - "tutorials/at-a-glance", - "tutorials/build-code-agent", - "tutorials/slack-bot", - "tutorials/github-review-bot", - "tutorials/deep-code-research", - "tutorials/codebase-analytics-dashboard", - "tutorials/training-data", - "tutorials/codebase-visualization", - "tutorials/migrating-apis", - "tutorials/organize-your-codebase", - "tutorials/promise-to-async-await", - "tutorials/modularity", - "tutorials/manage-feature-flags", - "tutorials/deleting-dead-code", - "tutorials/increase-type-coverage", - "tutorials/managing-typescript-exports", - "tutorials/converting-default-exports", - "tutorials/creating-documentation", - "tutorials/react-modernization", - "tutorials/unittest-to-pytest", - "tutorials/sqlalchemy-1.6-to-2.0", - "tutorials/fixing-import-loops-in-pytorch", - "tutorials/python2-to-python3", - "tutorials/flask-to-fastapi", - "tutorials/build-mcp", - "tutorials/neo4j-graph", - "tutorials/attributions" - ] - }, - { - "group": "Building with Codegen", - "pages": [ - "building-with-codegen/at-a-glance", - "building-with-codegen/parsing-codebases", - "building-with-codegen/reusable-codemods", - "building-with-codegen/dot-codegen", - "building-with-codegen/function-decorator", - "building-with-codegen/language-support", - "building-with-codegen/commit-and-reset", - "building-with-codegen/git-operations", - "building-with-codegen/files-and-directories", - "building-with-codegen/the-editable-api", - "building-with-codegen/symbol-api", - "building-with-codegen/class-api", - "building-with-codegen/imports", - "building-with-codegen/exports", - "building-with-codegen/inheritable-behaviors", - "building-with-codegen/statements-and-code-blocks", - "building-with-codegen/dependencies-and-usages", - "building-with-codegen/function-calls-and-callsites", - "building-with-codegen/variable-assignments", - "building-with-codegen/local-variables", - "building-with-codegen/comments-and-docstrings", - "building-with-codegen/external-modules", - "building-with-codegen/type-annotations", - "building-with-codegen/moving-symbols", - "building-with-codegen/collections", - "building-with-codegen/traversing-the-call-graph", - "building-with-codegen/react-and-jsx", - "building-with-codegen/codebase-visualization", - "building-with-codegen/flagging-symbols", - "building-with-codegen/calling-out-to-llms", - "building-with-codegen/semantic-code-search", - "building-with-codegen/reducing-conditions" - ] - }, - { - "group": "CLI", - "pages": [ - "cli/about", - "cli/init", - "cli/notebook", - "cli/create", - "cli/run", - "cli/reset", - "cli/expert" - ] - }, - { - "group": "Changelog", - "pages": [ - "changelog/changelog" - ] - }, - { - "group": "Blog", - "pages": [ - "blog/posts", - "blog/devin", - "blog/act-via-code", - "blog/promise-to-async-await-twilio", - "blog/fixing-import-loops" - ] - }, - { - "group": "codegen", - "pages": [ - "gen/introduction", - "gen/capabilities", - "gen/integrations", - "gen/faq" - ] - }, - { - "group": "API Reference", - "pages": [ - "api-reference/index", - { - "group": "Core", - "icon": "code", - "pages": [ - "api-reference/core/Argument", - "api-reference/core/Assignment", - "api-reference/core/AssignmentStatement", - "api-reference/core/Attribute", - "api-reference/core/AwaitExpression", - "api-reference/core/BinaryExpression", - "api-reference/core/BlockStatement", - "api-reference/core/Boolean", - "api-reference/core/Callable", - "api-reference/core/CatchStatement", - "api-reference/core/ChainedAttribute", - "api-reference/core/Class", - "api-reference/core/CodeBlock", - "api-reference/core/CodeOwner", - "api-reference/core/Codebase", - "api-reference/core/Comment", - "api-reference/core/CommentGroup", - "api-reference/core/ComparisonExpression", - "api-reference/core/Decorator", - "api-reference/core/Dict", - "api-reference/core/Directory", - "api-reference/core/Editable", - "api-reference/core/Export", - "api-reference/core/ExportStatement", - "api-reference/core/Exportable", - "api-reference/core/Expression", - "api-reference/core/ExpressionGroup", - "api-reference/core/ExpressionStatement", - "api-reference/core/ExternalModule", - "api-reference/core/File", - "api-reference/core/FlagKwargs", - "api-reference/core/ForLoopStatement", - "api-reference/core/Function", - "api-reference/core/FunctionCall", - "api-reference/core/GenericType", - "api-reference/core/HasBlock", - "api-reference/core/HasName", - "api-reference/core/HasValue", - "api-reference/core/IfBlockStatement", - "api-reference/core/Import", - "api-reference/core/ImportStatement", - "api-reference/core/ImportType", - "api-reference/core/Importable", - "api-reference/core/Interface", - "api-reference/core/List", - "api-reference/core/MessageType", - "api-reference/core/MultiExpression", - "api-reference/core/MultiLineCollection", - "api-reference/core/Name", - "api-reference/core/NamedType", - "api-reference/core/NoneType", - "api-reference/core/Number", - "api-reference/core/Pair", - "api-reference/core/Parameter", - "api-reference/core/ParenthesizedExpression", - "api-reference/core/Placeholder", - "api-reference/core/PlaceholderType", - "api-reference/core/RaiseStatement", - "api-reference/core/ReturnStatement", - "api-reference/core/SourceFile", - "api-reference/core/Span", - "api-reference/core/Statement", - "api-reference/core/StatementType", - "api-reference/core/String", - "api-reference/core/StubPlaceholder", - "api-reference/core/SubscriptExpression", - "api-reference/core/SwitchCase", - "api-reference/core/SwitchStatement", - "api-reference/core/Symbol", - "api-reference/core/SymbolGroup", - "api-reference/core/SymbolStatement", - "api-reference/core/TernaryExpression", - "api-reference/core/TryCatchStatement", - "api-reference/core/Tuple", - "api-reference/core/TupleType", - "api-reference/core/Type", - "api-reference/core/TypeAlias", - "api-reference/core/TypePlaceholder", - "api-reference/core/Typeable", - "api-reference/core/UnaryExpression", - "api-reference/core/UnionType", - "api-reference/core/Unpack", - "api-reference/core/Unwrappable", - "api-reference/core/Usable", - "api-reference/core/Usage", - "api-reference/core/UsageKind", - "api-reference/core/UsageType", - "api-reference/core/Value", - "api-reference/core/WhileStatement", - "api-reference/core/WithStatement" - ] - }, - { - "group": "Python", - "icon": "python", - "pages": [ - "api-reference/python/PyAssignment", - "api-reference/python/PyAssignmentStatement", - "api-reference/python/PyAttribute", - "api-reference/python/PyBlockStatement", - "api-reference/python/PyBreakStatement", - "api-reference/python/PyCatchStatement", - "api-reference/python/PyChainedAttribute", - "api-reference/python/PyClass", - "api-reference/python/PyCodeBlock", - "api-reference/python/PyComment", - "api-reference/python/PyCommentGroup", - "api-reference/python/PyCommentType", - "api-reference/python/PyConditionalExpression", - "api-reference/python/PyDecorator", - "api-reference/python/PyFile", - "api-reference/python/PyForLoopStatement", - "api-reference/python/PyFunction", - "api-reference/python/PyGenericType", - "api-reference/python/PyHasBlock", - "api-reference/python/PyIfBlockStatement", - "api-reference/python/PyImport", - "api-reference/python/PyImportStatement", - "api-reference/python/PyMatchCase", - "api-reference/python/PyMatchStatement", - "api-reference/python/PyNamedType", - "api-reference/python/PyParameter", - "api-reference/python/PyPassStatement", - "api-reference/python/PyReturnTypePlaceholder", - "api-reference/python/PyString", - "api-reference/python/PySymbol", - "api-reference/python/PyTryCatchStatement", - "api-reference/python/PyUnionType", - "api-reference/python/PyWhileStatement" - ] - }, - { - "group": "Typescript", - "icon": "js", - "pages": [ - "api-reference/typescript/JSXElement", - "api-reference/typescript/JSXExpression", - "api-reference/typescript/JSXProp", - "api-reference/typescript/TSArrayType", - "api-reference/typescript/TSAssignment", - "api-reference/typescript/TSAssignmentStatement", - "api-reference/typescript/TSAttribute", - "api-reference/typescript/TSBlockStatement", - "api-reference/typescript/TSCatchStatement", - "api-reference/typescript/TSChainedAttribute", - "api-reference/typescript/TSClass", - "api-reference/typescript/TSCodeBlock", - "api-reference/typescript/TSComment", - "api-reference/typescript/TSCommentGroup", - "api-reference/typescript/TSCommentType", - "api-reference/typescript/TSConditionalType", - "api-reference/typescript/TSConfig", - "api-reference/typescript/TSDecorator", - "api-reference/typescript/TSDict", - "api-reference/typescript/TSEnum", - "api-reference/typescript/TSExport", - "api-reference/typescript/TSExpressionType", - "api-reference/typescript/TSFile", - "api-reference/typescript/TSForLoopStatement", - "api-reference/typescript/TSFunction", - "api-reference/typescript/TSFunctionType", - "api-reference/typescript/TSGenericType", - "api-reference/typescript/TSHasBlock", - "api-reference/typescript/TSIfBlockStatement", - "api-reference/typescript/TSImport", - "api-reference/typescript/TSImportStatement", - "api-reference/typescript/TSInterface", - "api-reference/typescript/TSLabeledStatement", - "api-reference/typescript/TSLookupType", - "api-reference/typescript/TSNamedType", - "api-reference/typescript/TSNamespace", - "api-reference/typescript/TSObjectType", - "api-reference/typescript/TSPair", - "api-reference/typescript/TSParameter", - "api-reference/typescript/TSQueryType", - "api-reference/typescript/TSReadonlyType", - "api-reference/typescript/TSReturnTypePlaceholder", - "api-reference/typescript/TSString", - "api-reference/typescript/TSSwitchCase", - "api-reference/typescript/TSSwitchStatement", - "api-reference/typescript/TSSymbol", - "api-reference/typescript/TSTernaryExpression", - "api-reference/typescript/TSTryCatchStatement", - "api-reference/typescript/TSTypeAlias", - "api-reference/typescript/TSUndefinedType", - "api-reference/typescript/TSUnionType", - "api-reference/typescript/TSWhileStatement" - ] - } - ] - } - ], - "footerSocials": { - "x": "https://x.com/codegen", - "linkedin": "https://linkedin.com/company/codegen-dot-com" - } -} \ No newline at end of file + "$schema": "https://mintlify.com/schema.json", + "name": "Codegen", + "logo": { + "dark": "https://cdn.prod.website-files.com/67070304751b9b01bf6a161c/679bcf45a3e32761c42b324b_Codegen_Logomark_Dark.svg", + "light": "https://cdn.prod.website-files.com/67070304751b9b01bf6a161c/679bcf45bf55446746125835_Codegen_Logomark_Light.svg" + }, + "modeToggle": { + "default": "dark" + }, + "metadata": { + "og:site_name": "Codegen", + "og:title": "Codegen - The SWE that Never Sleeps", + "og:description": "Code agents accessible via API, Slack, Linear, Github, and more.", + "og:url": "https://docs.codegen.com", + "og:locale": "en_US", + "og:logo": "https://i.imgur.com/f4OVOqI.png", + "article:publisher": "Codegen, Inc.", + "twitter:site": "@codegen" + }, + "favicon": "/favicon.svg", + "colors": { + "primary": "#a277ff", + "light": "#a277ff", + "dark": "#a277ff", + "anchors": { + "from": "#61ffca", + "to": "#61ffca" + } + }, + "theme": "prism", + "background": { + "style": "gradient" + }, + "analytics": { + "posthog": { + "apiKey": "phc_GLxaINoQJnuyCyxDmTciQqzdKBYFVDkY7bRBO4bDdso" + } + }, + "feedback": { + "thumbsRating": true + }, + "topbarCtaButton": { + "name": "GitHub", + "url": "https://github.com/codegen-sh/codegen-sdk" + }, + "tabs": [ + { + "name": "API Reference", + "openapi": "https://codegen-sh--rest-api.modal.run/api/openapi.json", + "url": "/api-reference" + }, + { + "name": "Blog", + "url": "/blog" + }, + { + "name": "Changelog", + "url": "/changelog" + }, + { + "name": "Graph-sitter", + "url": "/graph-sitter" + } + ], + "navigation": [ + { + "group": "Introduction", + "pages": [ + "introduction/overview", + "introduction/community", + "introduction/about", + "introduction/faq" + ] + }, + { + "group": "Changelog", + "pages": ["changelog/changelog"] + }, + { + "group": "Blog", + "pages": [ + "blog/posts", + "blog/devin", + "blog/act-via-code", + "blog/promise-to-async-await-twilio", + "blog/fixing-import-loops" + ] + } + ], + "footerSocials": { + "x": "https://x.com/codegen", + "linkedin": "https://linkedin.com/company/codegen-dot-com" + } +} diff --git a/pyproject.toml b/pyproject.toml index 2b22a207b..26ede8797 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -72,6 +72,7 @@ dependencies = [ "datasets", "colorlog>=6.9.0", "codegen-sdk-pink>=0.1.0", + "dateutil", ] license = { text = "Apache-2.0" } diff --git a/src/codegen/sdk/README.md b/src/codegen/sdk/README.md index 7aefbe289..f9e94756b 100644 --- a/src/codegen/sdk/README.md +++ b/src/codegen/sdk/README.md @@ -1,8 +1,117 @@ -# Codegen SDK +
-A codegen module that contains the core Codebase graph parsing and manipulation logic. +

+ + + +

-### Dependencies +

+ Scriptable interface to a powerful, multi-lingual language server. +

-- [codegen.git](https://github.com/codegen-sh/codegen-sdk/tree/develop/src/codegen/git) -- [codegen.shared](https://github.com/codegen-sh/codegen-sdk/tree/develop/src/codegen/shared) +
+ +[![PyPI](https://img.shields.io/badge/PyPi-codegen-gray?style=flat-square&color=blue)](https://pypi.org/project/codegen/) +[![Documentation](https://img.shields.io/badge/Docs-docs.codegen.com-purple?style=flat-square)](https://docs.codegen.com) +[![Slack Community](https://img.shields.io/badge/Slack-Join-4A154B?logo=slack&style=flat-square)](https://community.codegen.com) +[![License](https://img.shields.io/badge/Code%20License-Apache%202.0-gray?&color=gray)](https://github.com/codegen-sh/codegen-sdk/tree/develop?tab=Apache-2.0-1-ov-file) +[![Follow on X](https://img.shields.io/twitter/follow/codegen?style=social)](https://x.com/codegen) + +
+ +
+ +[Codegen](https://docs.codegen.com) is a python library for manipulating codebases. + +```python +from codegen import Codebase + +# Codegen builds a complete graph connecting +# functions, classes, imports and their relationships +codebase = Codebase("./") + +# Work with code without dealing with syntax trees or parsing +for function in codebase.functions: + # Comprehensive static analysis for references, dependencies, etc. + if not function.usages: + # Auto-handles references and imports to maintain correctness + function.move_to_file("deprecated.py") +``` + +Write code that transforms code. Codegen combines the parsing power of [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) with the graph algorithms of [rustworkx](https://github.com/Qiskit/rustworkx) to enable scriptable, multi-language code manipulation at scale. + +## Installation and Usage + +We support + +- Running Codegen in Python 3.12 - 3.13 (recommended: Python 3.13+) +- macOS and Linux + - macOS is supported + - Linux is supported on x86_64 and aarch64 with glibc 2.34+ + - Windows is supported via WSL. See [here](https://docs.codegen.com/building-with-codegen/codegen-with-wsl) for more details. +- Python, Typescript, Javascript and React codebases + +``` +# Install inside existing project +uv pip install codegen + +# Install global CLI +uv tool install codegen --python 3.13 + +# Create a codemod for a given repo +cd path/to/repo +codegen init +codegen create test-function + +# Run the codemod +codegen run test-function + +# Create an isolated venv with codegen => open jupyter +codegen notebook +``` + +## Usage + +See [Getting Started](https://docs.codegen.com/introduction/getting-started) for a full tutorial. + +``` +from codegen import Codebase +``` + +## Troubleshooting + +Having issues? Here are some common problems and their solutions: + +- **I'm hitting an UV error related to `[[ packages ]]`**: This means you're likely using an outdated version of UV. Try updating to the latest version with: `uv self update`. +- **I'm hitting an error about `No module named 'codegen.sdk.extensions.utils'`**: The compiled cython extensions are out of sync. Update them with `uv sync --reinstall-package codegen`. +- **I'm hitting a `RecursionError: maximum recursion depth exceeded` error while parsing my codebase**: If you are using python 3.12, try upgrading to 3.13. If you are already on 3.13, try upping the recursion limit with `sys.setrecursionlimit(10000)`. + +If you run into additional issues not listed here, please [join our slack community](https://community.codegen.com) and we'll help you out! + +## Resources + +- [Docs](https://docs.codegen.com) +- [Getting Started](https://docs.codegen.com/introduction/getting-started) +- [Contributing](CONTRIBUTING.md) +- [Contact Us](https://codegen.com/contact) + +## Why Codegen? + +Software development is fundamentally programmatic. Refactoring a codebase, enforcing patterns, or analyzing control flow - these are all operations that can (and should) be expressed as programs themselves. + +We built Codegen backwards from real-world refactors performed on enterprise codebases. Instead of starting with theoretical abstractions, we focused on creating APIs that match how developers actually think about code changes: + +- **Natural mental model**: Write transforms that read like your thought process - "move this function", "rename this variable", "add this parameter". No more wrestling with ASTs or manual import management. + +- **Battle-tested on complex codebases**: Handle Python, TypeScript, and React codebases with millions of lines of code. + +- **Built for advanced intelligences**: As AI developers become more sophisticated, they need expressive yet precise tools to manipulate code. Codegen provides a programmatic interface that both humans and AI can use to express complex transformations through code itself. + +## Contributing + +Please see our [Contributing Guide](CONTRIBUTING.md) for instructions on how to set up the development environment and submit contributions. + +## Enterprise + +For more information on enterprise engagements, please [contact us](https://codegen.com/contact) or [request a demo](https://codegen.com/request-demo).