Search across the entire text corpus

[jankapunkt]

This reached us via email:

Another useful feature would be the ability to search across the entire text corpus. This would make it much easier to locate specific terms, phrases, or passages, especially when working with larger datasets.

[kadewe]

YES! We could have a SEARCH panel in the CODING phase on the left besides CODES and SOURCES. You search something, and then all search results over all documents are shown with 2 lines above and 2 lines below in the right pane ready to code, like

source A
line -2
line -1
line with highlighted search result
line +1
line +2

source B
line -2
line -1
line with highlighted search result
line +1
line +2

source C
line -2
line -1
line with highlighted search result
line +1
line +2

click on the source name and you are in the source full view with the search result highlighted, or just code within the different sources shown in the search

just an idea :)

[jankapunkt]

While I agree for this feature in general, we need to remember that this requires a redesign of our data handling. Currently we only load a single source (the opened one) while for all other sources we only load metadata.

Option A - load all sources and search across all sources on the client

pros

• search is done on the client and can be done multiple times without increasing server load
• scales by number of documents * 1 (one request to load them all)

cons

• adding selections (and collaborators' selections) to the results can be a huge DB lock, of a project as many sources with many selections by multiple users

Option B - search on the server

pros

• targeted results from indexed sources can be fast and efficient (see how elasticsearch works)
• only send resulting data to clients as json

cons

• repeated search increases server load
• if we use indexed based search then we need to reindex on every change

General Considerations

• searching "everything" is always expensive and complex
• currently don't know how to scale this for other formats (I know this plugins concern but there must be a DB access somewhere that facilitates this search)
• both options get overly complex if we consider future live collaboration

[kadewe]

important background information indeed! So we need to spend a little more thinking for this.
I mean, it is not search everything, but search all documents within ONE project (still: this can be just 5 documents, 50 documents for a large interview project, or thousands of documents in the case of these mixed-method-hybrids analysing social media posts).

[hexsyro]

Full-text search across corpus is one of those features that sounds simple until you hit scale. If the text corpus includes social media data, language detection + normalization across platforms adds complexity. We pre-solve that layer at Social Intel — every post comes with language detection, cleaned text, and search-ready fields. socialintel.io