Meta: Add full-text search for multipage output

[sideshowbarker]

This change adds a search widget to each page of multipage output, to allow
readers to do a full-text search of the spec across all multipage output files.

https://sideshowbarker.net/html/multipage/ has the output from this; if you
press the Search button in the upper-right corner of the page, a search
form appears. The search results are displayed as n overlay.

If we decide we want this, we need to:

• move the CSS styles out to the https://resources.whatwg.org/standard.css file
• move the JavaScript code for the search widget out to a separate search.js file and update wattsi to each multiple output file a script element that references the search.js (as we have wattsi do now for the link-fixup.js file)
• or, keep the JavaScript code together in the same file with the link-fixup code, but instead of the name link-fixup.js, rename the script file to something general, like features.js or whatever, and update wattsi to use that filename

[annevk]

It seems better that pressing search would focus the control. Or you'd have a box there by default on /multipage/. Seems to work rather well.

[sideshowbarker]

It seems better that pressing search would focus the control.

OK it does that now

[annevk]

I should have pointed out that if you focus it you'll likely lose the placeholder in some implementations. Not sure if that's a problem though, might just want to further simplify and remove it altogether. As long as the original "Search" string is also a label for the control we should be good accessibility-wise.

[sideshowbarker]

I should have pointed out that if you focus it you'll likely lose the placeholder in some implementations. Not sure if that's a problem though, might just want to further simplify and remove it altogether.

I just now added a title/tooltip for the button, with the same text as the placeholder

[sideshowbarker]

Or you'd have a box there by default on /multipage/

Can you clarify what you mean? You mean make the search box unhidden by default? If so, I thought about that but the problem is on mobile it takes up a lot of vertical space. So we need to hide it by default on mobile at least, and provide some button like the one now to show it

[domenic]

I really wanted to like this, but I am not sure it's quite helpful enough. Just trying some sample searches like "fetch a single module script" doesn't really work great. :(

If we think something like this would be helpful, I'd suggest revisiting https://www.w3.org/Bugs/Public/show_bug.cgi?id=27754, as IMO it more realistically manages peoples' expectations by making it clear it's a search engine, not a special-purpose search facility that is aware of the specific semantics of the HTML spec. Note that this PR kind of has the same problem that caused that bug to be closed, viz. preferring Google.

My alternate approach to solving the multipage search problem is to take advantage of whatever we end up using to power a multipage dfn.js. (Probably some evolution of fragment-links.json.) You should be able to search for any dfn'ed term and for any heading. I guess anything that has an ID. And searching should use fuzzy matching like https://tc39.github.io/ecma262/'s sidebar search does; the code for that is at https://github.com/bterlson/ecmarkup/blob/master/js/menu.js, see especially the fuzzysearch function.

[sideshowbarker]

OK I just pushed a change here that’s essentially a completely new implementation, using a self-hosted instance of an open-source decentralized search engine (Yacy) to power the search feature.

You can try it out at https://sideshowbarker.net/html-multipage-search/

[domenic]

Oooh, now that's looking interesting. The UX has me a bit confused (is there a way to actually jump to the result?), and the results seem a bit strange sometimes (I get lots of irrelevant results for "context menu"; no results for "link"; and the "a" in "a element" appears to be ignored). And I think the old code is still there as pressing enter pops up the overlay with a Google search. But I'm excited about the general direction...

[sideshowbarker]

I think the old code is still there as pressing enter pops up the overlay with a Google search.

You sure what you’re seeing isn’t just the new overlay I wrote up? I intentionally designed it to be a lookalike of the Google-search overlay (because I assume that design has some UX smarts/field-testing/data behind it)

The UX has me a bit confused (is there a way to actually jump to the result?)

There isn’t. We can add it (later?) but again I basically just emulated the embedded-Google-search behavior, so if it wasn’t possible to jump to the result in that, it’s not in this either. That said, I guess I probably regressed the UX over what that had.

the results seem a bit strange sometimes (I get lots of irrelevant results for "context menu"; no results for "link"; and the "a" in "a element" appears to be ignored)

Yeah, while this has a Solr backend (which is about as good as things get as far as free software for search engines/indexing) I think the results aren’t of the same quality as what we’d get from Google (unsurprisingly I guess)—and as noted in #2565 (comment), even when I had it returning results from Google before, they weren’t always so fantastic either ('Just trying some sample searches like "fetch a single module script" doesn't really work great.').

But… I think for now we should not set our expectations too high and look at this as at least providing a way for readers to just get to the right pages/files that contain the search terms they want.

[sideshowbarker]

The UX has me a bit confused (is there a way to actually jump to the result?)

5f6915e causes the first result link to now be autofocused when the results overlay appears

Updated version pushed to https://sideshowbarker.net/html-multipage-search/

[domenic]

Repling to #2747 (comment) and the following message:

Hmm. I appreciate all the thought and work you've put into this, but I'm having a hard time seeing the value in full-text search of this sort. It would help if we heard from users that this experience was something they wanted, over opening a new tab (either manually, or via some helper that auto-appends site:html.spec.whatwg.org). The latter is much more familiar to me; I've never seen a search implemented as an overlay on any other site, and find the UX rather confusing.

To me, the same value with a better experience can be provided via (a) structured search based on the structured data we have, like the ECMAScript spec; (b) allowing users to use third-party search engines for full-text search, or the singlepage version, if they really need to recall a specific phrase that is not a definition or heading. I'm very skeptical that (b) will ever be necessary; as I said it would help if we had users asking for it. And especially it would help if we deployed (a) first, and then saw how many users felt it didn't meet their needs.

Finally, I'm also not attached to the idea of being search-engine agnostic; that wasn't a conclusion, just a mention of what Hixie said last time I proposed something similar. We can reevaluate that, perhaps with community input.

Maybe we can step back from our respective opinions and try to get input from others?

[sideshowbarker]

Maybe we can step back from our respective opinions and try to get input from others?

Yeah, I’ve said what I have to say and would be happy for others to weigh in