An experimental search service for ecosyste.ms, built in Go. search.ecosyste.ms provides a web interface and JSON API for selected projects from packages.ecosyste.ms.
Bleve provides full-text indexing and ecosystem filtering through its Scorch backend. The HTTP server and HTML templates use Go's standard library. CSS and JavaScript are served as embedded assets, without a frontend build step. Dependency versions are recorded in go.mod.
The server opens a prebuilt index read-only and uses a separate catalogue for descriptions, categories and public identifiers. Package details are read from the catalogue for returned results. Downloads and indexing run through a separate build command; serving requires no database service or search cluster.
Search retrieves up to 500 BM25 candidates and combines their text and popularity ranks using reciprocal rank fusion (RRF), with equal weights and k=60. Popularity uses the highest member-package dependent-repository percentile within its ecosystem. Exact-name matches come first, followed by the remaining ranked results. If no candidate has popularity data, search keeps BM25 ordering and scores. Descriptions use English stemming; names and identifiers are not stemmed. The web interface displays up to 100 results, with category suggestions drawn from those selected results. There is no pagination, blank-query browsing or exhaustive facet counting.
The packages service's corpus exporter supplies selected package records and project assignments. This is a selected corpus, not an exhaustive index of every ecosyste.ms package. The snapshot command requires those assignments; it validates them but does not generate or revise the grouping. Coverage, metadata and assignments can change between exports. Repository-based groups can include unrelated or spam packages, and component descriptions can be misleading at project level.
| Archive | Contents | Use |
|---|---|---|
| Publisher corpus | Package records, projects, assignments and checksummed manifests | Input to the snapshot command |
Search bundle (search.tar.gz) |
Bleve index, serving catalogue and integrity manifests | Extract and run the server |
The server verifies the index/catalogue pairing before serving. /api/v1/health reports the snapshot version, creation date and source checksums. Project IDs are snapshot-local and may change between daily builds when upstream assignments change. Use PURLs or registry/package identifiers for references across snapshots.
A public search bundle contains the 2 November 2026 snapshot: 249,891 projects from 471,954 packages, approximately 65 MB compressed. Its SHA256 is 44bdc7c771763a1fdb5770dad4a2462cacf3481caf7c973584ad60bb4a540e8d. This is a fixed snapshot URL; there is no automatic latest lookup. Tests use synthetic publisher fixtures and run without a production snapshot.
Install Go at the version required by go.mod or newer, and Make. Extracting a bundle requires tar with gzip support. JavaScript tests require Node.js with node --test. Local builds and tests do not require Docker, a packages checkout or a database.
From the repository root, build the binary and extract a prebuilt search bundle into a new directory. Replace /path/to/search.tar.gz with the bundle you obtained:
make build
mkdir -p data
mkdir data/local
tar -xzf /path/to/search.tar.gz -C data/local
make serve SNAPSHOT=data/localOpen http://localhost:8100. Stop the server with Ctrl+C. The extracted directory should contain index/, serving-catalog.ndjson, manifest.json and index-checksums.json. Keep these files together and extract new snapshots into separate directories.
To use the compiled binary or choose another address:
./bin/search explore -index data/local/index -catalog data/local -addr 127.0.0.1:8100If you have a publisher archive rather than a prebuilt bundle, set CORPUS_URL to its HTTP(S) URL and CORPUS_SHA256 to the checksum supplied by the publisher, then run:
make build
./bin/search snapshot -url "$CORPUS_URL" -sha256 "$CORPUS_SHA256" -output data/release
make serve SNAPSHOT=data/release/snapshotThe URL can be a public or signed S3-compatible storage URL. The output directory must not already exist. Preparation validates checksums, memberships and public identifiers, then writes snapshot/, a downloadable search.tar.gz and receipt.json. A failed build removes its staging directory and leaves existing releases unchanged. Building needs more memory than serving.
HTML, CSS, JavaScript and the OpenAPI specification are embedded in the executable. Restart make serve, or rebuild and restart the binary, after changing them.
Search and project pages have query-specific or project-specific titles, descriptions and social metadata. Canonical URLs omit expansion parameters. Search results and snapshot-local project pages use noindex, follow; the homepage remains indexable.
The API base URL is https://search.ecosyste.ms/api/v1. The OpenAPI specification is also included in the repository. The API and specification allow cross-origin browser requests without credentials, using Access-Control-Allow-Origin: *.
curl 'https://search.ecosyste.ms/api/v1/search?q=numpy&ecosystem=pypi&limit=100'Search requires q, with these optional parameters:
| Parameter | Behaviour |
|---|---|
limit |
10 (default) or 100 |
ecosystem |
One case-insensitive value, such as npm, pypi or rubygems; omitted or empty means unfiltered |
suggestions |
true (default) or false |
Ecosystem filtering happens before selecting candidates and does not add to relevance scores. Unknown ecosystems return no results. A project can span several ecosystems, so its category suggestions can include other ecosystems. Registry filtering is not supported.
Each search hit contains its name, description, scores and public identity data. The public object includes up to ten package identities with PURLs, public links and bulk lookup requests for those packages. Exact package-name or PURL matches are shown first, followed by a representative from each remaining registry, then other packages in snapshot order. Exact matches and registry representatives prefer registries with more package memberships in the loaded snapshot; these are corpus counts, not global registry sizes. Ecosystem labels also appear in descending membership order. This changes the displayed identities, not project ranking. package_count gives the full membership count; packages_truncated indicates a shortened list. Snapshots retain all memberships. No separate detail lookup is required. /api/v1/health is retained for deployment checks.
The application sets a one-hour cache lifetime for successful search responses and HTML pages. On search.ecosyste.ms, Cloudflare overrides the browser lifetime to one day; shared caches retain the one-hour lifetime. Search errors and health checks use Cache-Control: no-store. Cache keys must include the full query string, including filters and result limits. Cached responses can show the previous snapshot until they expire.
An abbreviated hit from a search for @types/node:
{
"id": "package:1333965",
"name": "@types/node",
"description": "TypeScript definitions for node",
"identity": {
"label": "npm · @types/node · npmjs.org",
"purl": "pkg:npm/%40types/node",
"url": "https://repos.ecosyste.ms/hosts/GitHub/repositories/definitelytyped%2Fdefinitelytyped",
"package_count": 1
}
}The example omits scores and the public object. Multi-package projects list their PURLs in public.packages rather than selecting one PURL for the project. The application does not enforce request-rate limits; limits for the hosted service have not yet been specified.
make testThis runs the Go race tests and JavaScript tests. HTTP tests need loopback listener access; the updater tests run only on Linux and require jq. The saved publisher fixtures require no Ruby installation, packages checkout or database. Fixture provenance and the regeneration script are recorded beside the fixtures.
To compare two builds from identical publisher input:
SEARCH_REBUILD_ARCHIVE=/absolute/path/to/corpus.tar.gz \
go test ./internal/cli -run '^TestSnapshotRebuildRanking$' -count=1 -timeout=15m -vThis builds two temporary snapshots through the CLI and compares 66 queries through the HTTP handler. It rejects score changes and reordered unequal-score results, but allows equal-score ties to reorder or cross the top-100 cutoff. Temporary snapshots are removed after the test. The regular suite uses a small fixture when the environment variable is unset.
For a code change, compare against a running server using the same snapshot:
SEARCH_SNAPSHOT=/absolute/path/to/snapshot SEARCH_BASELINE_URL=http://127.0.0.1:8100 \
go test ./internal/cli -run TestExistingSnapshotRegression -vThat check compares API responses and HTML as well as rankings. The code is split between cmd/search (entry point) and internal/cli, internal/project, internal/snapshot, internal/search, internal/web and internal/storage. Web templates and assets are in internal/web/assets.
The Docker image contains the application and web assets; snapshots are mounted read-only. For data updates, scripts/update-snapshot can build from a publisher archive or activate a prebuilt search bundle with --bundle. It selects the running application image, validates the new snapshot, switches the active directory and restarts the server. The previous snapshot is retained for rollback. Application deployments are separate from data updates. See the Docker and Dokku deployment guide.
AppSignal monitoring uses OpenTelemetry. Set APPSIGNAL_APP_NAME, APPSIGNAL_PUSH_API_KEY and APPSIGNAL_COLLECTOR_ENDPOINT to enable it. APPSIGNAL_APP_ENV defaults to production; the deployment revision comes from APP_REVISION or Dokku's GIT_REV. With none of the three required settings present, monitoring is disabled. Incomplete configuration logs a warning and the server continues without monitoring.
HTML and API requests report timings, internal errors and panics. Query strings are excluded from request attributes; health checks and static assets are not traced. Reports are sent in the background and flushed on shutdown. No monitoring agent or writable volume is required in the application container.
Licensed under the GNU Affero General Public License v3.0.