Import Safari browsing history - #694
Conversation
Safari is the default browser on macOS and the only major one hister could not import from. The gap was small but structural: every supported browser keeps the URL and its last-visit timestamp in one flat table, so the import query is built as `SELECT DISTINCT url FROM <table>`. Safari splits them — URLs live in history_items, one row per visit in history_visits — so no single table name can describe it. Rather than special-case Safari inside the query builder, the timestamp schema map becomes a source descriptor that can also carry a FROM clause. Browsers with a flat schema leave it empty and are unaffected: their queries are byte for byte what they were, including the pass-through for a table name this code has never seen. Safari supplies a derived table that joins the two and reduces the visits to the most recent per URL, giving the same one-row- per-URL shape everything else already has, visit_count included so --min-visit keeps working. Its epoch offset is negative because Safari counts from 2001-01-01 rather than 1970, and the existing (unix + offset) * units arithmetic handles that without change. Reading ~/Library/Safari/History.db needs Full Disk Access. sql.Open is lazy, so without it the failure surfaced much later as the driver's "unable to open database file", which on macOS reads as a file-permission problem and is not one — no chmod will fix it. Opening the file up front turns that into an answer that names the setting.
|
Thank you for your contribution. The code looks good but I don't have a mac to try it out. @ad3lre could you please test it? |
|
On it. |
|
Tested on my Mac with a real Safari History.db and a full e2e run (local server + crawl + search). Works: hister import browser safari auto-detects, reads URLs, --start-date / --min-visit filter correctly, tests pass. E2e: 3 URLs from a Safari-shaped fixture → all crawled and searchable (example.com, github.com/asciimoo/hister, wikipedia.org). Please fix before merge: hister import browser ~/Library/Safari/History.db (path only) still maps to table History and fails (no such table: History). safari or auto-detect work but path-only should too, or it should be documented. |
|
Will do. |
Reported by @ad3lre on this PR: `hister import browser ~/Library/Safari/History.db` failed with "no such table: History". Safari and Ladybird both name their database History.db, so the filename check in importHistoryFile mapped Safari's to Ladybird's table. Chrome's is called History with no extension, distinguished from those two only by the absence of a suffix — the whole approach was already close to its limit and adding a second browser using an existing filename pushed it over. What a database IS cannot be settled by what it is called. It now opens the file and looks at which tables are present. Safari is checked first, being the only one identified by a pair of tables, so a match there is unambiguous. This also fixes the pre-existing ambiguity between Ladybird and anything else sharing its filename, and gives a clearer failure for an unrecognised file than "couldn't auto detect table" did.
|
Fixed in 9bba0a3 — thanks for testing it properly, that was a real hole. The filename check couldn't work: Safari and Ladybird both use It now identifies the database by the tables it contains rather than by its name, checking Safari first since it's the only one keyed on a pair of tables. That also resolves the Ladybird ambiguity that predated this PR. Tests cover all four schemas including the path-only Safari case you hit, and I've confirmed it against a real |
|
Thank you for both the testing and the fixes. I have a few notes:
|
Three points raised by @asciimoo. getBrowserType did not know about Safari, so the selection screen labelled it "unknown" and typing "safari" at the exclusion prompt matched nothing. The Full Disk Access hint was wrong in both directions. It ran too late: a path-only import reaches detectHistoryTable first, so the driver's "unable to open database file" arrived ahead of any explanation. The check now happens before the file is opened. It also ran too widely — any macOS permission error produced it, including on a Chrome profile, where it would send somebody to change a system setting that was never the problem. It is now limited to paths inside Safari's protected directory, which is the only place the advice is true. Documentation covers Safari's location, the Full Disk Access requirement, and the fact that a path-only import identifies a database by the tables it contains rather than its filename, since Safari and Ladybird share one.
|
Thanks — all three addressed in 0cba8a0. Oh wow, I totally missed this, good catch. The Full Disk Access hint — you were right on both counts, and the fix is the one you suggested. It now runs before the file is opened, inside Docs — One deliberate omission: I had written a line pointing at Tests cover the new browser type and the path scoping. |
|
|
||
| **Reading it requires Full Disk Access** for the terminal or application running Hister, granted under System Settings > Privacy & Security > Full Disk Access. Without it the import fails with a permission error naming the setting. | ||
|
|
||
| Safari splits its history across two tables, storing one row per visit, and counts timestamps from 2001-01-01 rather than 1970. Hister handles all of this internally; `--start-date` and `--min-visit` behave exactly as they do for any other browser. |
There was a problem hiding this comment.
I don't think that this information would be useful for our users.
There was a problem hiding this comment.
Noted, removed.
|
|
||
| Safari splits its history across two tables, storing one row per visit, and counts timestamps from 2001-01-01 rather than 1970. Hister handles all of this internally; `--start-date` and `--min-visit` behave exactly as they do for any other browser. | ||
|
|
||
| Safari has no Hister browser extension, so an import is a snapshot rather than a running feed: re-run it to pick up pages visited since. |
There was a problem hiding this comment.
All the imports behave as described here regardless having an extension or not . I'd remove this sentence.
There was a problem hiding this comment.
Noted, removed.
Both cuts requested in review, and both were me explaining more than a user needs. The schema description — two tables, one row per visit, timestamps from 2001 — is how the import works rather than how to use it. Nobody running the command needs to know, and the point of handling it internally is that they do not. "An import is a snapshot rather than a running feed" is true of every import, not Safari's, so putting it under a Safari heading implies a difference that does not exist. What is left is the part that is genuinely specific: where the database is, that there are no profiles to choose between, and that reading it needs Full Disk Access.
Safari is the default browser on macOS and the only major one
hister import browsercould notread. This adds it.
Not related to #49 / #46. Those are about running the extension as a Safari Web Extension for
live indexing, and the CORS preflight that blocks it. This is the bulk import path and touches
nothing they touch — I mention it only because "Safari" in a title invites the assumption.
Why the change has this shape
The import turned out to need very little from a browser.
browserImportURLQuerybuilds the wholething as:
plus optional
visit_countand timestamp clauses. It reads the URL and nothing else — no title, novisit dates — because hister fetches and indexes each page itself. So supporting a browser is
mostly a question of what shape its history is in.
Safari's is the one shape that does not fit: URLs live in
history_items, one row per visit inhistory_visits. There is no single table name that answers the query.Rather than branch inside the query builder,
browserHistoryTimestampSchemabecomesbrowserHistorySource, which can also carry a FROM clause. Browsers with a flat schema leave itempty and use their table name exactly as before — including the existing pass-through that lets a
caller name a table this code has never heard of. Safari supplies a derived table that joins the
two and reduces visits to the most recent per URL, so it presents the same one-row-per-URL shape
everything else already has,
visit_countincluded so--min-visitkeeps working.Its epoch offset is negative: Safari counts seconds from 2001-01-01 rather than 1970, so a Unix
timestamp is 978,307,200 seconds further along than the same moment in its terms. The existing
(unix + offset) * unitsarithmetic handles that unchanged.The Full Disk Access diagnostic
~/Library/Safari/History.dbis protected by TCC, so hister cannot read it unless the terminal orapplication running it has been granted Full Disk Access.
sql.Openis lazy, so without the grantthe failure surfaces later as the driver's "unable to open database file" — which on macOS reads as
a file-permission problem and is not one. No
chmodfixes it. Opening the file up front turns thatinto an error that names the setting.
Happy to split this out if you would rather keep the PR to one concern. It is here because it
is the first thing a macOS user will hit, and the error it replaces sends people in the wrong
direction.
Testing
go build ./...,go test ./...andgolangci-lint run ./cmd/...are all clean.Four tests added, the most important being
TestBrowserImportURLQueryFlatSchemasUnchanged— aregression guard asserting that introducing a FROM expression does not alter one byte of the query
the other browsers produce, unknown table names included. That is the actual risk this refactor
carries.
TestPrepareBrowserImportsReadsSafariHistoryruns end to end against a Safari-shaped database withone visit either side of a
--start-datecut-off, so the date filter is exercised through thederived table rather than only asserted about.
Notes for review
visit_timeis stored asREALin Safari, and the test fixture declares it that way. Thecomparison against an integer timestamp works because SQLite compares numerically across storage
classes, but it is worth knowing rather than discovering.
needs
GOOS == "darwin"and a real home directory, the second an unreadable file.getDBPathshas no existing coverage either, so this is consistent with the file rather than a new gap — but
say if you would like it addressed.
alternative gives every browser an explicit FROM clause naming its own table, at the cost of
touching all of them. I took the smaller diff; happy to switch if you prefer the other trade.