List corpuses

[blakerosenthal]

Addresses #492

[blakerosenthal]

@pmeier Is this about right? I like the idea of raising NotImplemented on source_storage.list_corpuses() in case the user decides to ignore corpuses altogether.

Also right now the /corpuses endpoint returns the list of corpus names directly from the source_storages. I have two questions about this:

1. Should the api call distinguish which source storage the corpus name belongs to? E.g. ["LanceDB/<corpus_name>", "ChromaDB/<corpus_name>"]? Maybe actually returning a JSON of {"LanceDB": [<corpus_name>], "ChromaDB": [<corpus_name>]} would be better?
2. For the built in source storages, the default corpus name is just the embedding ID. Maybe list_corpuses() should translate this to a prettier string like "default" for the sake of the UI.

Finally, should I add the actual UI changes as part of this issue, or will that be a separate issue?

[pmeier]

I like the idea of raising NotImplemented on source_storage.list_corpuses() in case the user decides to ignore corpuses altogether.

On second thought, we should probably raise a descriptive RagnaException instead. We need to tell users what they have to do if they encounter this error and then exit gracefully.

1. Should the api call distinguish which source storage the corpus name belongs to? E.g. ["LanceDB/<corpus_name>", "ChromaDB/<corpus_name>"]? Maybe actually returning a JSON of {"LanceDB": [<corpus_name>], "ChromaDB": [<corpus_name>]} would be better?

You are right, that needs to happen. I would go for the JSON approach where the key is the .display_name().

2. For the built in source storages, the default corpus name is just the embedding ID. Maybe list_corpuses() should translate this to a prettier string like "default" for the sake of the UI.

Yes, that needs to happen on the source storage itself. I'm also inclined to remove the feature of using the embedding ID as name for now. There is no use and it makes it more complicated. So feel free to just remove the embedding ID and just use "default".

[pmeier]

Finally, should I add the actual UI changes as part of this issue, or will that be a separate issue?

Up to you. When using two PRs we can merge this PR soon and don't have to wait for #484 to be merged.

[pmeier]

Looking good. I'll review properly when the open questions you had are addressed.

[pmeier]

@blakerosenthal I fixed the corpus support for the demo source storage in #498. Meaning, you can roll back all changes there besides the list_corpuses implementation.

[blakerosenthal]

Up to you. When using two PRs we can merge this PR soon and don't have to wait for #484 to be merged.

Okay, let's keep them separate then

[pmeier]

Thanks Blake, almost LGTM to me!

[pmeier]

Thanks Blake. FYI: I pushed a small cleanup commit.