Introduce standalone C parser for RBS with arena allocation

[st0012]

RBS C Parser Library Refactoring

This PR refactors the RBS parser and related components into a standalone C library that no longer depends on the Ruby runtime. This architectural change enables direct integration with static analysis tools like Sorbet while potentially improving performance.

Sorbet's RBS support already runs on this new architecture and we haven't discovered any major issues around it.

This work was a collaborative effort by

• @amomchilov
• @Morriar
• @st0012

Key Improvements

• Ruby-Independent Implementation: Extracted parser from ext folder into a standalone C library with a clean API, which can now be embedded in non-Ruby tools without Ruby runtime dependency (e.g. Sorbet, JRuby)
• Enhanced Memory Management: Implemented arena allocator to efficiently manage parser object lifecycles
• Improved Architecture: Clear separation between public API (headers) and implementation
• Performance: Potential performance gains from custom memory management and reduced overhead

Enhanced Memory Management

Arena allocator handles all memory for parser objects, including parser itself, lexer, constant pool, strings...etc. When the parser is freed by calling rbs_parser_free, the allocator will free all the objects it allocated. This eliminates the need to manually free individual objects and reduces the risk of memory leaks.

Component Architecture

graph TD
    RubyClient[Ruby Client] --> RubyAPI[Ruby API]
    CClient[C Client] --> CAPI[C API]

    RubyAPI --> CExtension[C Extension]
    CExtension --> CLibrary
    CAPI --> CLibrary

    subgraph CLibrary[C Library]
        subgraph Parser1[Parser Instance 1]
            direction TB
            ConstantPool1[Constant Pool]
            Lexer1[Lexer]
            ArenaAllocator1[Arena Allocator]
        end

        subgraph Parser2[Parser Instance 2]
            direction TB
            ConstantPool2[Constant Pool]
            Lexer2[Lexer]
            ArenaAllocator2[Arena Allocator]
        end
    end

    subgraph "Public API"
        RubyAPI
        CAPI
    end

    %% Parser1 --> ConstantPool1
    %% Parser1 --> Lexer1
    %% Parser1 --> ArenaAllocator1
    %% Parser2 --> ConstantPool2
    %% Parser2 --> Lexer2
    %% Parser2 --> ArenaAllocator2

Loading

[soutaro]

Thank you all for this contribution! I really appreciate all of you. 👏
The code generally looks great, but to confirm everything works I would like to implement another parsing function by myself. I will try it tomorrow. 🙏

[soutaro]

I tried to add a new inline annotation at #2443, and confirmed that I feel I can work on this codebase. 👍

[soutaro]

Opened ruby/ruby#13237 to test if the new C code works with Ruby CI compilers. (I should have done the test before merging... 💦 )