Working with Dataloaders, Observable/Framework's Superpower

[stevexvm]

Hi everyone,

As someone who is new to Observable Framework, I wanted to open a discussion to share my early experiences working with dataloaders and to invite comments, corrections, and suggestions for best practices. I would be grateful for any guidance from those who have spent more time with the ecosystem.

I have found both the Framework documentation and Allison Horst’s video walkthroughs very helpful:
Write your first data loader in an Observable Framework project
A practical introduction to Observable Framework’s data loaders: tips from the Observable team

To me, dataloaders feel like another major innovation in the spirit of D3.js. They are powerful, expressive, and broadly useful, and I can easily imagine them becoming a standard topic in data science courses. They solve several longstanding challenges that arise when building data-driven (web) applications.

Setting up Framework and working through the provided examples was straightforward, as long as I stayed within the expected patterns. Things became a bit more delicate once I began experimenting beyond the basics, where “coloring outside the lines” required more care.

What I hope to capture here are my observations, along with a few issues I encountered and the solutions that worked for me as I moved from using Framework “as is” to extending it and integrating it more deeply into my own development workflow. In other words, I am interested in going beyond the standard approach of building Framework apps with:

npx @observablehq/framework@latest create

and exploring how to adopt, then adapt the environment more flexibly.

My notes to share:

1. Using Bun instead of npm has worked very smoothly. I have not encountered any issues, and the performance improvements are exactly what you would expect, everything feels noticeably faster.
2. Much of the Framework code follows what I think of as the “Bostock style”: concise, elegant, and brilliantly composed. However, this can make it a bit more challenging to step through in a debugger, especially when trying to observe intermediate return values or follow nested function calls.
3. Preview mode and the development server have become essential tools for me. For example, I wanted to add a second sidebar on the right, adjust the position of the sidebar toggle button, and make a few related UI changes. Without Preview, the development and debugging cycle would have been significantly slower.
4. The observablehq.config.ts file is a true linchpin for everything that follows. In the Framework documentation, it deserves prominent placement, perhaps immediately after “Getting Started,” rather than at the end of the Reference section. As one example, I did not want to use “src” (or "docs" as is in the repo) as the "root" directory for my static site. It was not until my third reading of the configuration reference that I noticed the root property could be renamed. I now use “app,” and next time it may well be “site.”
5. The Framework source is well designed, but challenging to navigate. The main difficulty I encountered was simply the number of files involved. This is perhaps where AI-assisted code navigation really shines. My own approach was to introduce an additional layer of directory structure to make the codebase easier to reason about and more modular. For example, I added separate directories for components, markup, renderers, and types. The first three support an ongoing effort to refactor the page layout so it is more customizable, for instance, to enable a second sidebar, and easier to extend. My longer-term goal is to reach a point where I can then integrate AstroJS components into Framework, giving me the flexibility to use either Markdown or Astro for pages. This may be overly ambitious, but an interesting direction I want to explore as I find AstroJS a pleasure to work with.
6. Adding an extra layer of directory structure also makes it feasible to extend Framework from a single-application static site generator into a multi-application static portal generator. This is the direction I hope to take: working with multiple datasets, each with its own index page and each contained within its own directory to keep everything organized and easy to maintain. In this setup, adding a new dataset and visualization becomes straightforward. I can prototype the visualization in Observable, and once it is ready, I can simply “slip” the project into Framework as another self-contained module, another portal card on the main landing page.
7. Debugging requires more than placing a simple browser breakpoint, and I want to describe what worked for me when stepping through JS/TS code that spans both browser and server. I use VS Code, which includes a very helpful tool called the JavaScript Debug Terminal (JDT). When I launch the Framework Preview server from a JDT, I am able to set breakpoints in the server-side JavaScript code and watch how it interacts with the client-side code. I am not sure whether the same applies to Python-based dataloaders or those written in other languages. VS Code does offer a Python debug terminal as well, but I do not know whether breakpoints will be respected when running from within a JDT (my guess, probably not, but would love to be proven wrong as the age of AI seems to belong to Python).
8. This matters particularly for dataloader development, since dataloaders run once at build/startup or whenever they are modified. The tutorials suggest several techniques for debugging JavaScript/JSON dataloaders, but each has trade-offs. For example, one clever approach is to run the dataloader as a standalone Node.js script. However, when running in Preview mode, the dataloader’s behavior is tied to Framework’s internal runtime, its request context, environment variables, and data paths. Which means that running it independently does not always reproduce the exact conditions under which Framework executes it.
9. Taking the US Electricity Grid example, EIA the dataloader us-states.json.js has a hard-coded filepath:

const us = simplify(presimplify(JSON.parse(readFileSync("src/data/us-counties-10m.json", "utf-8"))), 0.1);

which is fine if you stay with Framework's project structure conventions. But I want to build a multi-application portal, and in which case each dataset sits within its own directory, like so:

src/data/us-counties-10m.json -> src/pages/eia/src/data/us-counties-10m.json

I also wanted to find a reliable way to reference files relative to the current source file’s directory. In Node.js, this is normally done with __dirname together with path.join(). Using __dirname, I was able to debug my dataloader as an independent Node.js script, confirm its behavior, and give myself more flexibility when loading local data files.
10. However, in Preview mode the dataloader (for example, us-states.json.js) runs as an ESM module, where __dirname does not exist. To work around this, the code needs to extract the directory path relative to the current script using something like the following pattern:

import { fileURLToPath } from 'node:url';
import { readFileSync } from "node:fs";
import path from 'node:path';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

11. This is where combining browser breakpoints with VS Code’s debugging engine becomes invaluable. By running Framework’s Preview server inside a JavaScript Debug Terminal (JDT), I can pause execution in the browser, inspect the current application state, and then step directly into the server-side dataloader (JS/TS) code, line by line, to observe exactly how Node.js is handling it.

That is enough for now. Thank you for taking the time to read through these notes, and thank you in advance for any comments or corrections. I also want to express my appreciation for everyone contributing to Framework's Discussions, Issues, and Pull Requests. I think it is a wonderful project, and community support is especially important while the Observable team is busy advancing Canvases, Notebooks, and the upcoming Desktop application builder.