Skip to content

Import Safari browsing history - #694

Merged
asciimoo merged 4 commits into
asciimoo:masterfrom
PerpetualBeta:safari-history-import
Aug 30, 2026
Merged

Import Safari browsing history#694
asciimoo merged 4 commits into
asciimoo:masterfrom
PerpetualBeta:safari-history-import

Conversation

@PerpetualBeta

@PerpetualBeta PerpetualBeta commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Safari is the default browser on macOS and the only major one hister import browser could not
read. This adds it.

Not related to #49 / #46. Those are about running the extension as a Safari Web Extension for
live indexing, and the CORS preflight that blocks it. This is the bulk import path and touches
nothing they touch — I mention it only because "Safari" in a title invites the assumption.

Why the change has this shape

The import turned out to need very little from a browser. browserImportURLQuery builds the whole
thing as:

SELECT DISTINCT url FROM <table> WHERE (url LIKE 'http://%' OR url LIKE 'https://%')

plus optional visit_count and timestamp clauses. It reads the URL and nothing else — no title, no
visit dates — because hister fetches and indexes each page itself. So supporting a browser is
mostly a question of what shape its history is in.

Safari's is the one shape that does not fit: URLs live in history_items, one row per visit in
history_visits. There is no single table name that answers the query.

Rather than branch inside the query builder, browserHistoryTimestampSchema becomes
browserHistorySource, which can also carry a FROM clause. Browsers with a flat schema leave it
empty and use their table name exactly as before — including the existing pass-through that lets a
caller name a table this code has never heard of. Safari supplies a derived table that joins the
two and reduces visits to the most recent per URL, so it presents the same one-row-per-URL shape
everything else already has, visit_count included so --min-visit keeps working.

Its epoch offset is negative: Safari counts seconds from 2001-01-01 rather than 1970, so a Unix
timestamp is 978,307,200 seconds further along than the same moment in its terms. The existing
(unix + offset) * units arithmetic handles that unchanged.

The Full Disk Access diagnostic

~/Library/Safari/History.db is protected by TCC, so hister cannot read it unless the terminal or
application running it has been granted Full Disk Access. sql.Open is lazy, so without the grant
the failure surfaces later as the driver's "unable to open database file" — which on macOS reads as
a file-permission problem and is not one. No chmod fixes it. Opening the file up front turns that
into an error that names the setting.

Happy to split this out if you would rather keep the PR to one concern. It is here because it
is the first thing a macOS user will hit, and the error it replaces sends people in the wrong
direction.

Testing

go build ./..., go test ./... and golangci-lint run ./cmd/... are all clean.

Four tests added, the most important being TestBrowserImportURLQueryFlatSchemasUnchanged — a
regression guard asserting that introducing a FROM expression does not alter one byte of the query
the other browsers produce, unknown table names included. That is the actual risk this refactor
carries.

TestPrepareBrowserImportsReadsSafariHistory runs end to end against a Safari-shaped database with
one visit either side of a --start-date cut-off, so the date filter is exercised through the
derived table rather than only asserted about.

Notes for review

  • visit_time is stored as REAL in Safari, and the test fixture declares it that way. The
    comparison against an integer timestamp works because SQLite compares numerically across storage
    classes, but it is worth knowing rather than discovering.
  • The macOS detection path and the Full Disk Access branch are not covered by tests: the first
    needs GOOS == "darwin" and a real home directory, the second an unreadable file. getDBPaths
    has no existing coverage either, so this is consistent with the file rather than a new gap — but
    say if you would like it addressed.
  • The derived table is a string literal in what was previously a table of constants. The tidier
    alternative gives every browser an explicit FROM clause naming its own table, at the cost of
    touching all of them. I took the smaller diff; happy to switch if you prefer the other trade.

Safari is the default browser on macOS and the only major one hister could
not import from. The gap was small but structural: every supported browser
keeps the URL and its last-visit timestamp in one flat table, so the import
query is built as `SELECT DISTINCT url FROM <table>`. Safari splits them —
URLs live in history_items, one row per visit in history_visits — so no
single table name can describe it.

Rather than special-case Safari inside the query builder, the timestamp
schema map becomes a source descriptor that can also carry a FROM clause.
Browsers with a flat schema leave it empty and are unaffected: their queries
are byte for byte what they were, including the pass-through for a table name
this code has never seen. Safari supplies a derived table that joins the two
and reduces the visits to the most recent per URL, giving the same one-row-
per-URL shape everything else already has, visit_count included so
--min-visit keeps working.

Its epoch offset is negative because Safari counts from 2001-01-01 rather
than 1970, and the existing (unix + offset) * units arithmetic handles that
without change.

Reading ~/Library/Safari/History.db needs Full Disk Access. sql.Open is lazy,
so without it the failure surfaced much later as the driver's "unable to open
database file", which on macOS reads as a file-permission problem and is not
one — no chmod will fix it. Opening the file up front turns that into an
answer that names the setting.
@asciimoo

Copy link
Copy Markdown
Owner

Thank you for your contribution. The code looks good but I don't have a mac to try it out. @ad3lre could you please test it?

@ad3lre

ad3lre commented Aug 29, 2026

Copy link
Copy Markdown

On it.

@ad3lre

ad3lre commented Aug 29, 2026

Copy link
Copy Markdown

Tested on my Mac with a real Safari History.db and a full e2e run (local server + crawl + search).

Works: hister import browser safari auto-detects, reads URLs, --start-date / --min-visit filter correctly, tests pass. E2e: 3 URLs from a Safari-shaped fixture → all crawled and searchable (example.com, github.com/asciimoo/hister, wikipedia.org).

Please fix before merge: hister import browser ~/Library/Safari/History.db (path only) still maps to table History and fails (no such table: History). safari or auto-detect work but path-only should too, or it should be documented.

@PerpetualBeta

@PerpetualBeta

Copy link
Copy Markdown
Contributor Author

Will do.

Reported by @ad3lre on this PR: `hister import browser ~/Library/Safari/History.db`
failed with "no such table: History".

Safari and Ladybird both name their database History.db, so the filename check
in importHistoryFile mapped Safari's to Ladybird's table. Chrome's is called
History with no extension, distinguished from those two only by the absence of a
suffix — the whole approach was already close to its limit and adding a second
browser using an existing filename pushed it over.

What a database IS cannot be settled by what it is called. It now opens the file
and looks at which tables are present. Safari is checked first, being the only
one identified by a pair of tables, so a match there is unambiguous.

This also fixes the pre-existing ambiguity between Ladybird and anything else
sharing its filename, and gives a clearer failure for an unrecognised file than
"couldn't auto detect table" did.
@PerpetualBeta

Copy link
Copy Markdown
Contributor Author

Fixed in 9bba0a3 — thanks for testing it properly, that was a real hole.

The filename check couldn't work: Safari and Ladybird both use History.db, and Chrome's is History with no extension, so the three were separated only by suffix presence. Adding Safari pushed it over.

It now identifies the database by the tables it contains rather than by its name, checking Safari first since it's the only one keyed on a pair of tables. That also resolves the Ladybird ambiguity that predated this PR.

Tests cover all four schemas including the path-only Safari case you hit, and I've confirmed it against a real ~/Library/Safari/History.db.

@asciimoo

Copy link
Copy Markdown
Owner

Thank you for both the testing and the fixes.

I have a few notes:

  • As I see Safari is missing from getBrowserType. The selection screen will label Safari as unknown, and entering safari when asked which histories to exclude will not work.
  • The Full Disk Access hint is inconsistent. Path only imports fail before the hint runs, while unrelated macOS permission errors can incorrectly show it. A solution could be running the check before schema detection and limiting the hint to Safari.
  • Please extend the import documentation (website/src/content/docs/import.md)

Three points raised by @asciimoo.

getBrowserType did not know about Safari, so the selection screen labelled it
"unknown" and typing "safari" at the exclusion prompt matched nothing.

The Full Disk Access hint was wrong in both directions. It ran too late: a
path-only import reaches detectHistoryTable first, so the driver's "unable to
open database file" arrived ahead of any explanation. The check now happens
before the file is opened. It also ran too widely — any macOS permission error
produced it, including on a Chrome profile, where it would send somebody to
change a system setting that was never the problem. It is now limited to paths
inside Safari's protected directory, which is the only place the advice is true.

Documentation covers Safari's location, the Full Disk Access requirement, and
the fact that a path-only import identifies a database by the tables it contains
rather than its filename, since Safari and Ladybird share one.
@PerpetualBeta

Copy link
Copy Markdown
Contributor Author

Thanks — all three addressed in 0cba8a0.

Oh wow, I totally missed this, good catch. getBrowserType — added. It was labelling Safari unknown on the selection screen and safari matched nothing at the exclusion prompt, exactly as you described.

The Full Disk Access hint — you were right on both counts, and the fix is the one you suggested. It now runs before the file is opened, inside detectHistoryTable, and it is limited to paths inside ~/Library/Safari/: a permission error on a Chrome profile is an ordinary permission error, and pointing someone at a system settings pane that was never the problem is worse than saying nothing.

Docsimport.md now covers Safari's location, the Full Disk Access requirement, and the fact that a path-only import identifies a database by the tables it contains rather than by its filename, since Safari and Ladybird share one.

One deliberate omission: I had written a line pointing at hister companion safari, then removed it. That command is #695 and does not exist on this branch, so the documentation would have described something unavailable if this merged alone. It is documented over there instead, and neither PR now links to an anchor the other creates, so they can land in either order.

Tests cover the new browser type and the path scoping.


**Reading it requires Full Disk Access** for the terminal or application running Hister, granted under System Settings > Privacy & Security > Full Disk Access. Without it the import fails with a permission error naming the setting.

Safari splits its history across two tables, storing one row per visit, and counts timestamps from 2001-01-01 rather than 1970. Hister handles all of this internally; `--start-date` and `--min-visit` behave exactly as they do for any other browser.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that this information would be useful for our users.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted, removed.


Safari splits its history across two tables, storing one row per visit, and counts timestamps from 2001-01-01 rather than 1970. Hister handles all of this internally; `--start-date` and `--min-visit` behave exactly as they do for any other browser.

Safari has no Hister browser extension, so an import is a snapshot rather than a running feed: re-run it to pick up pages visited since.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the imports behave as described here regardless having an extension or not . I'd remove this sentence.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted, removed.

Both cuts requested in review, and both were me explaining more than a user
needs.

The schema description — two tables, one row per visit, timestamps from 2001 —
is how the import works rather than how to use it. Nobody running the command
needs to know, and the point of handling it internally is that they do not.

"An import is a snapshot rather than a running feed" is true of every import,
not Safari's, so putting it under a Safari heading implies a difference that
does not exist.

What is left is the part that is genuinely specific: where the database is, that
there are no profiles to choose between, and that reading it needs Full Disk
Access.

@asciimoo asciimoo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@asciimoo
asciimoo merged commit 5980b96 into asciimoo:master Aug 30, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants