docstring-metadata.md

Docstring Metadata Convention

Consent Scheme source can carry documentation metadata as ordinary R7RS data in procedure bodies. The convention supports simple string docstrings and richer property records without adding reader syntax or changing standard Scheme evaluation.

Comments remain source-only contributor notes. A standard R7RS reader does not return comments as datums, so comments are not visible to runtime reflection, reference generation, logs, yields, or compiled metadata unless a separate source tool reads the original text.

R7RS Compatibility

The convention uses literal body expressions that R7RS-small already accepts. A literal expression in a non-final body position is evaluated and discarded by ordinary Scheme semantics. Consent Scheme treats selected non-final literals as metadata while preserving the same runtime result.

Documentation metadata is recognized only in the leading non-tail metadata position of a body:

1. Internal definitions come first, as R7RS requires.
2. Zero or more leading metadata literals may follow those definitions.
3. At least one non-metadata body expression must remain after the metadata.
4. The first non-metadata expression ends metadata recognition for that body.

A string or rich property record in final position is an ordinary return value, not metadata. This rule avoids changing the meaning of procedures such as (lambda () "value").

(define (fact n)
  "Return the factorial of exact non-negative integer N."
  (if (= n 0)
      1
      (* n (fact (- n 1)))))

Internal definitions still precede documentation metadata:

(define (twice x)
  (define factor 2)
  "Return X multiplied by the local factor."
  (* x factor))

Metadata after executable body expressions is ignored as metadata and remains ordinary Scheme code:

(define (not-a-docstring x)
  (display x)
  "This string is an ordinary expression, not metadata."
  x)

Simple String Form

A simple string in metadata position is shorthand for the documentation field. Adjacent simple strings in metadata position form one documentation string, with one space inserted at each string boundary. Use an explicit \n inside a string when the documentation should contain a real line or paragraph break. The reader annotates the shortcut into the same rich field record shape that an explicit #((documentation . "...")) property record would produce.

(define (sum xs)
  "Return the arithmetic sum of XS."
  "XS must be a list of numbers."
  (let loop ((rest xs) (total 0))
    (if (null? rest)
        total
        (loop (cdr rest) (+ total (car rest))))))

The documentation field for sum is:

"Return the arithmetic sum of XS. XS must be a list of numbers."

The live reflection surface exposes the same simple string through (documentation subject) in (agent reflect). subject can be a binding symbol, binding name string, or procedure value:

(import (scheme base) (agent reflect))

(define (sum xs)
  "Return the arithmetic sum of XS."
  (let loop ((rest xs) (total 0))
    (if (null? rest)
        total
        (loop (cdr rest) (+ total (car rest))))))

(documentation 'sum)
;; =>
(documentation-metadata
  (subject (binding sum))
  (kind procedure)
  (library #f)
  (source #f)
  (origin (body-literal string))
  (fields
    ((arguments (xs))
     (documentation "Return the arithmetic sum of XS."))))

Procedures with no body-literal documentation still expose their signature metadata:

(define (identity x)
  x)

(documentation 'identity)
;; =>
(documentation-metadata
  (subject (binding identity))
  (kind procedure)
  (library #f)
  (source #f)
  (origin (signature))
  (fields ((arguments (x)))))

Retention Options

Evaluation accepts a docstring-retention option for callers that need to trade reflection detail for lower runtime retention cost. The Emacs Lisp host uses the plist key :docstring-retention; the portable Scheme host uses the option alist key docstring-retention.

• full is the default. It retains generated arguments, simple string docstrings, and rich property records.
• simple keeps generated arguments and simple string docstrings, but drops rich property fields after using them only to recognize the metadata prefix.
• none, Emacs Lisp nil, and Scheme #f drop all body-derived procedure documentation metadata, including generated arguments.

Recognized leading non-final docstring literals are removed from stored compound procedure bodies after metadata extraction when a non-metadata body expression remains. This avoids retaining and re-evaluating source documentation literals as procedure body data. A final string or rich vector literal remains ordinary Scheme code and is not removed.

The none mode intentionally does not promise generated signature metadata. R7RS specifies procedure calling behavior and lexical binding semantics, but it does not require procedure values or later compiled representations to retain the source text of formal parameter names for reflection.

Rich Property Records

Rich documentation metadata uses a literal vector of pairs in the same leading non-tail metadata position. This follows Guile's procedure-property style as an influence, but the fields and reflection shape below are Consent Scheme public behavior.

For exported runtime procedures, every expanded parameter and return descriptor includes both (type ...) and a non-empty (description ...). Neither field substitutes for the other: the type supports machine-readable reflection, while the description explains the value's role in the contract. Every exported procedure also includes a non-empty effects field made only of symbols. pure states the absence of observable effects; omission is not an implicit purity claim.

(define (open-agent-log path)
  "Open PATH as an Consent Scheme log input port."
  #((parameters
     (path (type string)
      (description "Path to a readable log file.")))
    (returns (type input-port)
     (description "An input port."))
    (effects file-read))
  (open-input-file path))

The simple string form and rich property form may appear together:

(define (normalize-name name)
  "Return NAME in canonical Consent Scheme identifier form."
  #((parameters
     (name (type (or string symbol))
      (description "A string or symbol.")))
    (returns (type symbol)
     (description "A symbol.")))
  (if (symbol? name)
      name
      (string->symbol name)))

When reflected through (documentation subject), the string contributes the documentation field and rich properties preserve their Scheme-readable values. In reflected metadata, every field entry has the record-field shape (field-name value ...); therefore (returns ((type ...) ...)) means the single value of the returns field is the return descriptor:

(documentation 'normalize-name)
;; =>
(documentation-metadata
  (subject (binding normalize-name))
  (kind procedure)
  (library #f)
  (source #f)
  (origin (body-literal string vector))
  (fields
    ((arguments (name))
     (documentation "Return NAME in canonical Consent Scheme identifier form.")
     (parameters
      ((name
        (type (or string symbol))
        (description "A string or symbol."))))
     (returns
      ((type symbol)
       (description "A symbol."))))))

Applies To

The body convention applies independently to every procedure body:

• procedure shorthand define, such as (define (name args ...) body ...)
• lambda expressions
• each case-lambda clause body
• top-level or internal bindings whose initializer is a lambda or case-lambda expression

For a binding such as (define name followed by (lambda (...) ...) on the next line, metadata belongs to the procedure value and may also be associated with the binding name by the frontend or reference generator. If the same procedure value is stored in multiple bindings, binding-specific documentation remains a separate metadata subject from procedure-value documentation. This describes a supported metadata position, not the preferred project source style. Checked-in Scheme should follow Scheme Style Guidelines, which prefer procedure definition syntax and keep unavoidable lambda or case-lambda initializers on the line after the binding name.

Primitive bindings are not read as ordinary procedure bodies. Kernel primitives, standard host-effecting bindings, Agent primitive libraries, and host capability primitives therefore use the primitive manifest as their runtime documentation source. Public primitive manifest entries should carry explicit documentation metadata with origin (primitive-manifest metadata) when the manifest supplies rich fields, or (primitive-manifest string) for string-only manifest documentation. Reflected string-only entries report (origin (primitive-manifest string)). Tests guard that surface. When an implementation-only or generated manifest entry lacks explicit documentation, the bootstrap may derive a documentation field from the registered implementation procedure's own docstring. Reflected fallback metadata reports (origin (implementation-procedure string)) instead of (origin (body-literal string)) so tools can distinguish source body docstrings from host or bootstrap implementation docs.

This convention does not make simple string docstrings for these surfaces:

• define-syntax and macro exports
• define-record-type and record fields
• define-library forms
• re-exported or renamed bindings

Those surfaces need explicit binding, syntax, record, library, or export metadata records so static reference tools can describe the exported API without pretending a transformer procedure body documents the macro it creates. Later metadata work may add such subject-specific records while still using ordinary R7RS datums.

Metadata Records

Runtime reflection, static reference generation, logs, yields, and compiled runtimes should expose one Scheme-readable record shape. The fields member is a list of record fields shaped as (field-name value ...), so the reflected returns entry wraps the return descriptor as its single value:

(documentation-metadata
  (subject (binding fact))
  (kind procedure)
  (library (example math))
  (source (file "example/math.sld") (line 12) (column 3))
  (origin (body-literal string))
  (fields
    ((arguments (n))
     (documentation "Return the factorial of exact non-negative integer N.")
     (parameters
      ((n
        (type exact-non-negative-integer)
        (description "Exact non-negative integer."))))
     (returns
      ((type exact-integer)
       (description "Exact integer.")))
     (effects (pure)))))

Manifest-backed primitive documentation uses the same record shape:

(documentation '+)
;; =>
(documentation-metadata
  (subject (binding +))
  (kind procedure)
  (library (scheme base))
  (source kernel)
  (origin (primitive-manifest metadata))
  (fields
    ((documentation
      "Return the sum of all numeric arguments, or 0 when called with no arguments.")
     (parameters
      ((numbers
        (type (list-of number))
        (description "Numeric addends to sum."))))
     (returns
      ((type number)
       (description "The numeric sum.")))
     (effects (pure)))))

Field values are ordinary Scheme-readable data. The initial field set is:

• arguments: the procedure formals as Scheme-readable data using the procedure's symbolic bindings; proper, dotted, variadic, and empty formals reflect as (x y), (x . rest), rest, and ()
• documentation: string documentation for humans and agents
• summary: short string suitable for indexes
• parameters: association list from parameter symbol to a descriptor
• returns: descriptor for the return value or values
• effects: list of effect symbols, such as (pure) or (file-read)
• examples: list of source/result example records
• see-also: list of related binding, library, issue, or document references
• since: version datum such as (consent-version 0 15 2)
• deprecated: #f or a string explaining the replacement
• stability: symbol such as experimental, stable, or internal

Implementations may preserve unknown fields as Scheme-readable data for tools that understand them, but public documentation should prefer the field names above until a later issue extends the convention.

Parameter and return descriptors normally use proper lists of descriptor entries. The initial descriptor fields are:

• type: Scheme-readable type form; omitted type metadata defaults to (type any) so metadata-free and partially documented definitions keep their ordinary untyped meaning
• description: string prose for the value; a non-empty list of strings is accepted and joined with spaces, which lets source wrap descriptions without changing the reflected prose

For convenience, a parameter value or returns value may be just a string or a non-empty list of strings; that shorthand is normalized to a descriptor with (type any) and a description. Use that shorthand when the value is intentionally opaque, generic, or polymorphic at this library edge. When a narrower contract-shaped type is known, prefer the expanded descriptor and spell the type explicitly. Project library lint treats string and string-list shorthand as intentional any, but still rejects expanded public parameter and return descriptors that omit (type ...). The lint also rejects shorthand or expanded (type any) descriptors whose prose names an obvious primitive type such as string, symbol, list, vector, procedure, port, or boolean; spell the narrower type in those cases.

For expanded descriptors, prefer the compact layout that keeps (type ...) on the same line as the parameter name or returns head when the line fits within the soft line limit:

#((parameters
   (field (type symbol)
    (description "Symbol naming the field.")))
  (returns (type pair)
   (description "The matching field pair."))
  (effects pure))

Keep longer type forms on their own line, and prefer a plain description string when it fits. Use a list of description strings only when the prose itself needs wrapping.

The first type vocabulary is intentionally contract-shaped:

• (type any) is the top type and the default for omitted type metadata
• atomic Scheme value names, such as boolean, symbol, string, number, exact-integer, pair, list, vector, and procedure
• compound forms (or type ...), (list-of type), (vector-of type), (pair car-type cdr-type), (procedure (arg-type ...) return-type), and (values type ...)
• zero returned values are written (type (values)); multiple values are written (type (values type ...))
• custom type names are preserved as symbols for documentation, static tools, and future contract lowering; implementations may treat names they do not recognize as opaque extension points

Prefer custom type names that align with predicates: remove the trailing ? from the predicate name, so eval-context? implies (type eval-context) and consent-session-manager? implies (type consent-session-manager). When a value is only a Scheme-readable tagged datum and the library does not define a matching predicate, prefer the structural type that the current vocabulary can honestly state, such as (type list). For example, R7RS library names are documented as (type (list-of (or symbol exact-integer))); the current vocabulary does not yet express the nonnegative integer refinement.

The form (forall ...) is reserved for future polymorphic metadata, but the first pass does not implement relationship-aware checking.

Boundary Contract Checking

Typed metadata is advisory by default: ordinary evaluation keeps the same untyped runtime behavior when a call passes a value that does not match a documented descriptor. Callers that want fail-closed boundary checks can enable shallow checking with the Emacs Lisp option :boundary-contract-checking t or the portable Scheme option (boundary-contract-checking . #t).

The first checking mode lowers the documented first-order vocabulary at compound procedure call and return boundaries. It checks any, literal #f, atomic value names such as string, symbol, number, exact-integer, pair, list, vector, procedure, ports, bytevectors, characters, and EOF objects, plus shallow compound forms (or ...), (list-of ...), (vector-of ...), (pair car-type cdr-type), (procedure ...), and (values ...). Procedure types are shallow in this pass: the value must be callable, but the checker does not yet wrap higher-order argument or return contracts. Unknown custom type names, unknown compound forms, and reserved forms such as (forall ...) remain advisory extension points.

Contract failures render through the normal evaluation-result surface with a condition type of boundary-contract and a Scheme-readable contract-failure datum in the event stream and condition irritants. The datum records the boundary (procedure-call or procedure-return), blame (caller or callee), the parameter and position when applicable, the expected type form, and the observed value shape instead of exposing the full value.

Checking requires rich metadata. When checking is enabled but docstring retention has stripped rich fields with simple or none, evaluation reports boundary-contract-unavailable instead of silently running unchecked. Keep docstring-retention at full for runs that expect boundary enforcement.

Merge and Malformed Rules

The metadata prefix is processed in source order.

• The generated arguments field is derived from the procedure formals before body-literal metadata is merged.
• Adjacent simple strings are joined with space separators.
• A simple string is equivalent to a documentation field.
• Multiple documentation string values from simple strings and rich records are joined in source order with space separators.
• examples and see-also values append in source order when each value is a list.
• parameters values merge by parameter name; duplicate parameter names are malformed.
• Every parameters key must be present in the generated arguments datum; documenting an unbound parameter name is malformed.
• returns may appear at most once.
• Descriptor shorthand strings and descriptor description string lists are joined with spaces. Use explicit \n inside a string for a real line or paragraph break.
• Other duplicate scalar fields are malformed instead of silently replacing an earlier value.

A malformed rich metadata literal does not change evaluation semantics. Metadata-aware passes keep executing the program according to ordinary R7RS rules, attach no partial fields from the malformed literal, and report a malformed-documentation-metadata diagnostic with source information when that information is available. Valid earlier metadata for the same subject remains valid.

Source locations are best-effort metadata. When source information is available, records should identify both the documented subject and the metadata literal span by file, line, and column. When source information is unavailable, use #f or omit the source field rather than inventing a location.

Implementation Status

Simple string docstrings and rich property records are implemented for the Emacs Lisp bootstrap and the portable R7RS path. A leading non-final metadata prefix after internal definitions attaches a normalized field record to compound procedures and can be queried through (documentation subject) from (agent reflect). Compound procedures also receive generated arguments metadata from their lambda formals, so simple docstrings and rich property records share one field record with the signature metadata. Procedure shorthand define, explicit lambda, top-level bindings whose initializer is a lambda, and internal bindings with lambda initializers share the same body-literal extraction rule. Callers may select full, simple, or none docstring retention when evaluating source; recognized metadata literals are stripped from stored procedure bodies when they are not final return values.

Primitive bindings can also be queried through the same reflection procedure. Explicit manifest documentation is required for public primitive manifest entries and wins over implementation fallback. The fallback remains available for implementation-only or generated primitive hooks where the host can provide a procedure docstring. The portable R7RS path uses manifest metadata for public primitive help because standard R7RS does not provide a procedure-docstring reflection API for implementation procedures.

The current (scheme case-lambda) library is a portable macro that lowers each clause through an internal lambda, so ordinary evaluation still preserves the body string semantics, but the runtime does not yet expose durable clause-level documentation metadata for a case-lambda procedure value. That representation work is left to a later reflection/metadata slice.

• #300 defines the public convention.
• #301 implements simple string docstrings for the initial runtime and reflection slice.
• #302 adopts simple docstrings in checked-in libraries after extraction works.
• #303 implements rich documentation property records.
• #344 adds manifest-backed primitive documentation with implementation procedure fallback.
• #304 preserves documentation metadata across compiled and reference runtimes.
• #338 supplies syntax datum source metadata that can improve doc metadata source locations.
• #325 adds evaluator docstring retention modes for CI performance work.
• #598 enforces the rich property record on every exported runtime procedure, inverting the documentation gate (consent-scheme-documentation-test-public-rich-docstrings) from an opt-in allowlist to fail-closed coverage over all runtime scheme/ source with an empty exclusion list.
• #604 adds typed parameter and return descriptors, string-fragment prose shorthands, space-joined docstring fragments, and the project lint requirement that public runtime procedures spell out type metadata.
• #606 lowers typed metadata into opt-in shallow boundary contracts, including Scheme-readable failure datums and fail-closed reporting when rich metadata is stripped.