diff --git a/docs/docs/extraction/audio.md b/docs/docs/extraction/audio.md index a7a41cb89..4be7ee8ac 100644 --- a/docs/docs/extraction/audio.md +++ b/docs/docs/extraction/audio.md @@ -9,7 +9,7 @@ to extract speech from audio files. !!! note - NVIDIA Ingest and nv-ingest is now known as the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. Currently, you can extract speech from the following file types: diff --git a/docs/docs/extraction/content-metadata.md b/docs/docs/extraction/content-metadata.md index a329d7b80..78057ab22 100644 --- a/docs/docs/extraction/content-metadata.md +++ b/docs/docs/extraction/content-metadata.md @@ -10,7 +10,7 @@ Metadata can be extracted from a source or content, or generated by using models !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. diff --git a/docs/docs/extraction/contributing.md b/docs/docs/extraction/contributing.md index d05123e16..b39dcb636 100644 --- a/docs/docs/extraction/contributing.md +++ b/docs/docs/extraction/contributing.md @@ -1,4 +1,4 @@ -# Contributing to NeMo Retriever +# Contributing to NeMo Retriever Library -External contributions to NeMo Retriever will be welcome soon, and they are greatly appreciated! -For more information, refer to [Contributing to NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever/blob/main/CONTRIBUTING.md). +External contributions to NeMo Retriever Library will be welcome soon, and they are greatly appreciated! +For more information, refer to [Contributing to NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/main/CONTRIBUTING.md). diff --git a/docs/docs/extraction/custom-metadata.md b/docs/docs/extraction/custom-metadata.md index 96d81760d..98c654d58 100644 --- a/docs/docs/extraction/custom-metadata.md +++ b/docs/docs/extraction/custom-metadata.md @@ -56,7 +56,7 @@ meta_df.to_csv(file_path) ### Example: Add Custom Metadata During Ingestion The following example adds custom metadata during ingestion. -For more information about the `Ingestor` class, see [Use the NeMo Retriever Python API](python-api-reference.md). +For more information about the `Ingestor` class, see [Use the NeMo Retriever Library Python API](python-api-reference.md). For more information about the `vdb_upload` method, see [Upload Data](data-store.md). ```python diff --git a/docs/docs/extraction/data-store.md b/docs/docs/extraction/data-store.md index 61946b30e..990ee4f3d 100644 --- a/docs/docs/extraction/data-store.md +++ b/docs/docs/extraction/data-store.md @@ -4,7 +4,7 @@ Use this documentation to learn how [NeMo Retriever Library](overview.md) handle !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Overview diff --git a/docs/docs/extraction/environment-config.md b/docs/docs/extraction/environment-config.md index a59837583..422db0add 100644 --- a/docs/docs/extraction/environment-config.md +++ b/docs/docs/extraction/environment-config.md @@ -5,7 +5,7 @@ You can specify these in your .env file or directly in your environment. !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## General Environment Variables diff --git a/docs/docs/extraction/faq.md b/docs/docs/extraction/faq.md index ae5dcb7d1..f709d603c 100644 --- a/docs/docs/extraction/faq.md +++ b/docs/docs/extraction/faq.md @@ -4,7 +4,7 @@ This documentation contains the Frequently Asked Questions (FAQ) for [NeMo Retri !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. @@ -57,18 +57,12 @@ You can set those directly in `docker-compose.yaml`, or in an [environment varia ### Library Mode -For production environments, you should use the provided Helm charts. For [library mode](quickstart-library-mode.md), you should set the environment variable `NVIDIA_API_KEY`. This is because the NeMo Retriever containers and the NeMo Retriever services running inside them do not have access to the environment variables on the host machine where you run the `docker compose` command. Setting the variables in the `.env` file ensures that they are passed into the containers and available to the services that need them. +For production environments, you should use the provided Helm charts. For [library mode](quickstart-library-mode.md), you should set the environment variable `NVIDIA_API_KEY`. This happens because the NeMo Retriever Library containers—and the services running inside them—don’t have access to the environment variables of the host machine where the `docker compose` command is executed. Setting the variables in the `.env` file ensures that they are passed into the containers and available to the services that need them. For advanced scenarios, you might want to use library mode with self-hosted NIM instances. You can set custom endpoints for each NIM. For examples of `*_ENDPOINT` variables, refer to [NeMo-Retriever/docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docker-compose.yaml). - - - - - - ## What parameters or settings can I adjust to optimize extraction from my documents or data? See the [Profile Information](quickstart-guide.md#profile-information) section diff --git a/docs/docs/extraction/nemoretriever-parse.md b/docs/docs/extraction/nemoretriever-parse.md index 5dc655c64..60408ec4b 100644 --- a/docs/docs/extraction/nemoretriever-parse.md +++ b/docs/docs/extraction/nemoretriever-parse.md @@ -13,7 +13,7 @@ to run [NeMo Retriever Library](overview.md) with nemotron-parse. !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Limitations @@ -32,7 +32,7 @@ Use the following procedure to run the NIM locally. Due to limitations in available VRAM controls in the current release of nemotron-parse, it must run on a [dedicated additional GPU](support-matrix.md). Edit docker-compose.yaml to set nemotron-parse's device_id to a dedicated GPU: device_ids: ["1"] or higher. -1. Start the NeMo Retriever services with the `nemotron-parse` profile. This profile includes the necessary components for extracting text and metadata from images. Use the following command. +1. Start the NeMo Retriever Library services with the `nemotron-parse` profile. This profile includes the necessary components for extracting text and metadata from images. Use the following command. - The --profile nemotron-parse flag ensures that vision-language retrieval services are launched. For more information, refer to [Profile Information](quickstart-guide.md#profile-information). @@ -40,11 +40,11 @@ Use the following procedure to run the NIM locally. docker compose --profile nemotron-parse up ``` -2. After the services are running, you can interact with NeMo Retriever by using Python. +2. After the services are running, you can interact with NeMo Retriever Library by using Python. - The `Ingestor` object initializes the ingestion process. - The `files` method specifies the input files to process. - - The `extract` method tells NeMo Retriever to use `nemotron-parse` for extracting text and metadata from images. + - The `extract` method tells NeMo Retriever Library to use `nemotron-parse` for extracting text and metadata from images. - The `document_type` parameter is optional, because `Ingestor` should detect the file type automatically. ```python @@ -60,12 +60,12 @@ Use the following procedure to run the NIM locally. !!! tip - For more Python examples, refer to [NeMo Retriever: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). + For more Python examples, refer to [NeMo Retriever Library: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). ## Using NVCF Endpoints for Cloud-Based Inference -Instead of running NeMo Retriever locally, you can use NVCF to perform inference by using remote endpoints. +Instead of running NeMo Retriever Library locally, you can use NVCF to perform inference by using remote endpoints. 1. Set the authentication token in the `.env` file. @@ -85,7 +85,7 @@ Instead of running NeMo Retriever locally, you can use NVCF to perform inference - The `Ingestor` object initializes the ingestion process. - The `files` method specifies the input files to process. - - The `extract` method tells NeMo Retriever to use `nemotron-parse` for extracting text and metadata from images. + - The `extract` method tells NeMo Retriever Library to use `nemotron-parse` for extracting text and metadata from images. - The `document_type` parameter is optional, because `Ingestor` should detect the file type automatically. ```python @@ -101,7 +101,7 @@ Instead of running NeMo Retriever locally, you can use NVCF to perform inference !!! tip - For more Python examples, refer to [NeMo Retriever: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). + For more Python examples, refer to [NeMo Retriever Library: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). ## Run the Ray batch pipeline with `nemotron-parse` @@ -130,4 +130,4 @@ Replace `/path/to/pdfs` with the path to your input directory (for example, `/ho - [Support Matrix](support-matrix.md) - [Troubleshoot NeMo Retriever Library](troubleshoot.md) -- [Use the NeMo Retriever Python API](python-api-reference.md) +- [Use the NeMo Retriever Library Python API](python-api-reference.md) diff --git a/docs/docs/extraction/nimclient.md b/docs/docs/extraction/nimclient.md index 170a2e279..cc1c402f2 100644 --- a/docs/docs/extraction/nimclient.md +++ b/docs/docs/extraction/nimclient.md @@ -5,7 +5,7 @@ This documentation demonstrates how to create custom NIM integrations for use in !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. The NimClient architecture consists of two main components: diff --git a/docs/docs/extraction/notebooks.md b/docs/docs/extraction/notebooks.md index 9cff8c6f2..c9a0ca480 100644 --- a/docs/docs/extraction/notebooks.md +++ b/docs/docs/extraction/notebooks.md @@ -4,7 +4,7 @@ To get started using [NeMo Retriever Library](overview.md), you can try one of t !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Dataset Downloads for Benchmarking @@ -15,8 +15,8 @@ If you plan to run benchmarking or evaluation tests, you must download the [Benc To get started with the basics, try one of the following notebooks: -- [NeMo Retriever: CLI Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/cli_client_usage.ipynb) -- [NeMo Retriever: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb) +- [NeMo Retriever Library: CLI Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/cli_client_usage.ipynb) +- [NeMo Retriever Library: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb) - [How to add metadata to your documents and filter searches](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/metadata_and_filtered_search.ipynb) - [How to reindex a collection](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/reindex_example.ipynb) @@ -25,7 +25,7 @@ For more advanced scenarios, try one of the following notebooks: - [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb) - [Try Enterprise RAG Blueprint](https://github.com/NVIDIA/NeMo-Retriever/blob/main/deploy/pdf-blueprint.ipynb) -- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever and Milvus](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb) +- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever Library and Milvus](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb) - [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) - [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) diff --git a/docs/docs/extraction/overview.md b/docs/docs/extraction/overview.md index da38686c8..263204ddc 100644 --- a/docs/docs/extraction/overview.md +++ b/docs/docs/extraction/overview.md @@ -6,7 +6,7 @@ to find, contextualize, and extract text, tables, charts and infographics that y !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. NeMo Retriever Library enables parallelization of splitting documents into pages where artifacts are classified (such as text, tables, charts, and infographics), extracted, and further contextualized through optical character recognition (OCR) into a well defined JSON schema. From there, NeMo Retriever Library can optionally manage computation of embeddings for the extracted content, diff --git a/docs/docs/extraction/prerequisites.md b/docs/docs/extraction/prerequisites.md index 6fb4b9cb4..902c499c8 100644 --- a/docs/docs/extraction/prerequisites.md +++ b/docs/docs/extraction/prerequisites.md @@ -4,7 +4,7 @@ Before you begin using [NeMo Retriever Library](overview.md), ensure the followi !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. diff --git a/docs/docs/extraction/python-api-reference.md b/docs/docs/extraction/python-api-reference.md index 04aee370b..03bb138ca 100644 --- a/docs/docs/extraction/python-api-reference.md +++ b/docs/docs/extraction/python-api-reference.md @@ -4,7 +4,7 @@ The [NeMo Retriever Library](overview.md) Python API provides a simple and flexi !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. !!! tip diff --git a/docs/docs/extraction/quickstart-guide.md b/docs/docs/extraction/quickstart-guide.md index 01ee18a41..97bcfb578 100644 --- a/docs/docs/extraction/quickstart-guide.md +++ b/docs/docs/extraction/quickstart-guide.md @@ -112,7 +112,7 @@ Because many service URIs default to localhost, running inside the `nemo-retriev ## Step 3: Ingest Documents -You can submit jobs programmatically in Python or using the [NeMo Retriever CLI](cli-reference.md). +You can submit jobs programmatically in Python or using the [NeMo Retriever Library CLI](cli-reference.md). The following examples demonstrate how to extract text, charts, tables, and images: @@ -126,7 +126,7 @@ The following examples demonstrate how to extract text, charts, tables, and imag !!! tip - For more Python examples, refer to [NeMo Retriever: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). + For more Python examples, refer to [NeMo Retriever Library: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). ```python @@ -386,7 +386,7 @@ python src/util/image_viewer.py --file_path ./processed_docs/image/multimodal_te !!! tip - Beyond inspecting the results, you can read them into things like [llama-index](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) or [langchain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) retrieval pipelines. Also, checkout our [Enterprise RAG Blueprint on build.nvidia.com](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) to query over document content pre-extracted with NeMo Retriever. + Beyond inspecting the results, you can read them into things like [llama-index](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) or [langchain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) retrieval pipelines. Also, checkout our [Enterprise RAG Blueprint on build.nvidia.com](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) to query over document content pre-extracted with NeMo Retriever Library. @@ -459,7 +459,7 @@ docker compose \ ## Specify MIG slices for NIM models -When you deploy NeMo Retriever with NIM models on MIG‑enabled GPUs, MIG device slices are requested and scheduled through the `values.yaml` file for the corresponding NIM microservice. For IBM Content-Aware Storage (CAS) deployments, this allows NeMo Retriever NIM pods to land only on nodes that expose the desired MIG profiles [raw.githubusercontent](https://raw.githubusercontent.com/NVIDIA/NeMo-Retriever/main/helm/README.md%E2%80%8B).​ +When you deploy NeMo Retriever Library with NIM models on MIG‑enabled GPUs, MIG device slices are requested and scheduled through the `values.yaml` file for the corresponding NIM microservice. For IBM Content-Aware Storage (CAS) deployments, this allows NeMo Retriever Library NIM pods to land only on nodes that expose the desired MIG profiles [raw.githubusercontent](https://raw.githubusercontent.com/NVIDIA/NeMo-Retriever/main/helm/README.md%E2%80%8B).​ To target a specific MIG profile—for example, a 3g.20gb slice on an A100, which is a hardware-partitioned virtual GPU instance that gives your workload a fixed mid-sized share of the A100’s compute plus 20 GB of dedicated GPU memory and behaves like a smaller independent GPU—for a given NIM, configure the `resources` and `nodeSelector` under that NIM’s values path in `values.yaml`. @@ -482,7 +482,7 @@ Key points: * Use the appropriate NIM‑specific values path (for example, `nemo_retriever.nvidiaNim.nemoretrieverPageElements.resources`) rather than the generic `nemo_retriever.nim` placeholder. * Set `resources.requests` and `resources.limits` to the desired MIG resource name (for example, `nvidia.com/mig-3g.20gb`). * Use `nodeSelector` (or tolerations/affinity, if you prefer) to target nodes labeled with the corresponding MIG‑enabled GPU product (for example, `nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb`). -This syntax and structure can be repeated for each NIM model used by CAS, ensuring that each NeMo Retriever NIM pod is mapped to the correct MIG slice type and scheduled onto compatible nodes. +This syntax and structure can be repeated for each NIM model used by CAS, ensuring that each NeMo Retriever Library NIM pod is mapped to the correct MIG slice type and scheduled onto compatible nodes. !!! important diff --git a/docs/docs/extraction/quickstart-library-mode.md b/docs/docs/extraction/quickstart-library-mode.md index a6e921825..e65b4fac1 100644 --- a/docs/docs/extraction/quickstart-library-mode.md +++ b/docs/docs/extraction/quickstart-library-mode.md @@ -4,7 +4,7 @@ !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. In addition, you can use library mode, which is intended for the following cases: @@ -65,7 +65,7 @@ You can submit jobs programmatically by using Python. !!! tip - For more Python examples, refer to [NeMo Retriever: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). + For more Python examples, refer to [NeMo Retriever Library: Python Client Quick Start Guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/client/client_examples/examples/python_client_usage.ipynb). If you have a very high number of CPUs, and see the process hang without progress, diff --git a/docs/docs/extraction/releasenotes-nv-ingest.md b/docs/docs/extraction/releasenotes-nv-ingest.md index bc3319959..9e767c04c 100644 --- a/docs/docs/extraction/releasenotes-nv-ingest.md +++ b/docs/docs/extraction/releasenotes-nv-ingest.md @@ -4,7 +4,7 @@ This documentation contains the release notes for [NeMo Retriever Library](overv !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. @@ -12,7 +12,7 @@ This documentation contains the release notes for [NeMo Retriever Library](overv The NeMo Retriever Library 26.01 release adds new hardware and software support, and other improvements. -To upgrade the Helm Charts for this version, refer to [NeMo Retriever Helm Charts](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/helm/README.md). +To upgrade the Helm Charts for this version, refer to [NeMo Retriever Library Helm Charts](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/helm/README.md). ### Highlights @@ -20,7 +20,7 @@ To upgrade the Helm Charts for this version, refer to [NeMo Retriever Helm Chart This release contains the following key changes: - Added functional support for [H200 NVL](https://www.nvidia.com/en-us/data-center/h200/). For details, refer to [Support Matrix](support-matrix.md). -- All Helm deployments for Kubernetes now use [NVIDIA NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html). For details, refer to [NeMo Retriever Helm Charts](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/helm/README.md). +- All Helm deployments for Kubernetes now use [NVIDIA NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html). For details, refer to [NeMo Retriever Library Helm Charts](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/helm/README.md). - Updated RIVA NIM to version 1.4.0. For details, refer to [Extract Speech](audio.md). - Updated VLM NIM to [nemotron-nano-12b-v2-vl](https://build.nvidia.com/nvidia/nemotron-nano-12b-v2-vl/modelcard). For details, refer to [Extract Captions from Images](python-api-reference.md#extract-captions-from-images). - Added VLM caption prompt customization parameters, including reasoning control. For details, refer to [Caption Images and Control Reasoning](python-api-reference.md#caption-images-and-control-reasoning). @@ -33,7 +33,7 @@ This release contains the following key changes: - Large PDFs are now automatically split into chunks and processed in parallel, delivering faster ingestion for long documents. For details, refer to [PDF Pre-Splitting](v2-api-guide.md). - Issues maintaining extraction quality while processing very large files are now resolved with the V2 API. For details, refer to [V2 API Guide](v2-api-guide.md). - Updated the embedding task to support embedding on custom content fields like the results of summarization functions. For details, refer to [Use the Python API](python-api-reference.md). -- User-defined function summarization is now using `nemotron-mini-4b-instruct` which provides significant speed improvements. For details, refer to [User-defined Functions](user-defined-functions.md) and [NeMo Retriever UDF Examples](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/examples/udfs/README.md). +- User-defined function summarization is now using `nemotron-mini-4b-instruct` which provides significant speed improvements. For details, refer to [User-defined Functions](user-defined-functions.md) and [NeMo Retriever Library UDF Examples](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/examples/udfs/README.md). - In the `Ingestor.extract` method, the defaults for `extract_text` and `extract_images` are now set to `true` for consistency with `extract_tables` and `extract_charts`. For details, refer to [Use the Python API](python-api-reference.md). - The `table-structure` profile is no longer available. The table-structure profile is now part of the default profile. For details, refer to [Profile Information](quickstart-guide.md#profile-information). - New documentation [Why Throughput Is Dataset-Dependent](throughput-is-dataset-dependent.md). diff --git a/docs/docs/extraction/scaling-modes.md b/docs/docs/extraction/scaling-modes.md index 5669af9f7..d1546b724 100644 --- a/docs/docs/extraction/scaling-modes.md +++ b/docs/docs/extraction/scaling-modes.md @@ -7,7 +7,7 @@ This guide covers how resource scaling modes work across stages in [NeMo Retriev !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. diff --git a/docs/docs/extraction/support-matrix.md b/docs/docs/extraction/support-matrix.md index 4ae66e28a..e66bd4046 100644 --- a/docs/docs/extraction/support-matrix.md +++ b/docs/docs/extraction/support-matrix.md @@ -4,7 +4,7 @@ Before you begin using [NeMo Retriever Library](overview.md), ensure that you ha !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Core and Advanced Pipeline Features diff --git a/docs/docs/extraction/telemetry.md b/docs/docs/extraction/telemetry.md index dbf62682a..5c050452f 100644 --- a/docs/docs/extraction/telemetry.md +++ b/docs/docs/extraction/telemetry.md @@ -4,7 +4,7 @@ You can view telemetry data for [NeMo Retriever Library](overview.md). !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## OpenTelemetry diff --git a/docs/docs/extraction/troubleshoot.md b/docs/docs/extraction/troubleshoot.md index ab99f51fe..1e97448e8 100644 --- a/docs/docs/extraction/troubleshoot.md +++ b/docs/docs/extraction/troubleshoot.md @@ -4,7 +4,7 @@ Use this documentation to troubleshoot issues that arise when you use [NeMo Retr !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Can't process long, non-language text strings diff --git a/docs/docs/extraction/user-defined-functions.md b/docs/docs/extraction/user-defined-functions.md index 81f67f8f4..d188afab7 100644 --- a/docs/docs/extraction/user-defined-functions.md +++ b/docs/docs/extraction/user-defined-functions.md @@ -5,7 +5,7 @@ This guide covers how to write, validate, and submit UDFs using both the CLI and !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. @@ -121,7 +121,7 @@ results = ingestor.files("/path/to/document.pdf") \ ### Understanding IngestControlMessage (ICM) -The `IngestControlMessage` is the primary data structure that flows through the NeMo Retriever pipeline. Your UDF receives an ICM and must return a (potentially modified) ICM. +The `IngestControlMessage` is the primary data structure that flows through the NeMo Retriever Library pipeline. Your UDF receives an ICM and must return a (potentially modified) ICM. #### Key ICM Methods @@ -379,7 +379,7 @@ def my_udf(control_message: IngestControlMessage) -> IngestControlMessage: ### UDF Function Specification Formats -NeMo Retriever supports four different formats for specifying UDF functions: +NeMo Retriever Library supports four different formats for specifying UDF functions: ### 1. Inline Function String Define your function directly as a string: @@ -456,7 +456,7 @@ ingestor.udf(udf_function="my_package.processors.text_utils:enhance_metadata") ## Integrating with NVIDIA NIMs -NVIDIA Inference Microservices (NIMs) provide powerful AI capabilities that can be seamlessly integrated into your UDFs. The `NimClient` class offers a unified interface for connecting to and using NIMs within the NeMo Retriever pipeline. +NVIDIA Inference Microservices (NIMs) provide powerful AI capabilities that can be seamlessly integrated into your UDFs. The `NimClient` class offers a unified interface for connecting to and using NIMs within the NeMo Retriever Library pipeline. ### Quick NIM Integration @@ -521,7 +521,7 @@ export NGC_API_KEY="your-ngc-api-key" ### Available NIM Interfaces -NeMo Retriever provides several pre-built model interfaces: +NeMo Retriever Library provides several pre-built model interfaces: - **VLMModelInterface**: Vision-Language Models for image analysis and captioning - **EmbeddingModelInterface**: Text embedding generation @@ -542,7 +542,7 @@ See the comprehensive [**NimClient Usage Guide**](nimclient_usage.md). ### Error Handling -The NeMo Retriever system automatically catches all exceptions that occur within UDF execution. If your UDF fails for any reason, the system will: +The NeMo Retriever Library system automatically catches all exceptions that occur within UDF execution. If your UDF fails for any reason, the system will: 1. Annotate the job with appropriate error information 2. Mark the job as failed @@ -553,7 +553,7 @@ You do not need to implement extensive error handling within your UDF - focus on ### Performance Considerations -UDFs execute within the NeMo Retriever pipeline and can significantly impact overall system performance and stability. Understanding these considerations is crucial for maintaining optimal pipeline throughput and reliability. +UDFs execute within the NeMo Retriever Library pipeline and can significantly impact overall system performance and stability. Understanding these considerations is crucial for maintaining optimal pipeline throughput and reliability. #### Pipeline Impact @@ -941,6 +941,6 @@ def debug_udf(control_message: IngestControlMessage) -> IngestControlMessage: ## Related Topics -- [NeMo Retriever UDF Examples](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/examples/udfs/README.md) +- [NeMo Retriever Library UDF Examples](https://github.com/NVIDIA/NeMo-Retriever/blob/release/26.1.2/examples/udfs/README.md) - [User-Defined Stages for NeMo Retriever Library](user-defined-stages.md) - [NimClient Usage](nimclient.md) diff --git a/docs/docs/extraction/user-defined-stages.md b/docs/docs/extraction/user-defined-stages.md index 8acb0af9e..a20e17673 100644 --- a/docs/docs/extraction/user-defined-stages.md +++ b/docs/docs/extraction/user-defined-stages.md @@ -8,7 +8,7 @@ and operate on a well-defined DataFrame payload and metadata structure. !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. To add user-defined stages to your pipeline, you need the following: @@ -21,7 +21,7 @@ To add user-defined stages to your pipeline, you need the following: - **A DataFrame payload** — The `control_message.payload` field must be a [pandas.DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). For more information, refer to [Create a DataFrame Payload](#create-a-dataframe-payload). -- **Valid metadata** — The `metadata` field must conform to the [NeMo Retriever metadata schema](content-metadata.md). For more information, refer to [Update and Validate Metadata](#update-and-validate-metadata). +- **Valid metadata** — The `metadata` field must conform to the [NeMo Retriever Library metadata schema](content-metadata.md). For more information, refer to [Update and Validate Metadata](#update-and-validate-metadata). @@ -160,7 +160,7 @@ When the pipeline runs it does the following: ## Update and Validate Metadata The `metadata` column in each row is a dictionary (JSON object), -and must conform to the [NeMo Retriever metadata schema](content-metadata.md). +and must conform to the [NeMo Retriever Library metadata schema](content-metadata.md). After you change any metadata, you can validate it by using the `validate_metadata` function as demonstrated in the following code example. diff --git a/docs/docs/extraction/vlm-embed.md b/docs/docs/extraction/vlm-embed.md index 76084054f..331379ab3 100644 --- a/docs/docs/extraction/vlm-embed.md +++ b/docs/docs/extraction/vlm-embed.md @@ -9,7 +9,7 @@ The model supports three embedding modalities: `text`, `image`, and `text_image` !!! note - This library is the NeMo Retriever Library. + NVIDIA Ingest (nv-ingest) has been renamed to the NeMo Retriever Library. ## Configure and Run the Multimodal NIM @@ -189,5 +189,5 @@ results = ingestor.ingest() - [Support Matrix](support-matrix.md) - [Troubleshoot NeMo Retriever Library](troubleshoot.md) -- [Use the NeMo Retriever Python API](python-api-reference.md) +- [Use the NeMo Retriever Library Python API](python-api-reference.md) - [Extract Captions from Images](python-api-reference.md#extract-captions-from-images) diff --git a/docs/docs/index.md b/docs/docs/index.md index b40c7a568..2e867b2d0 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -1,13 +1,13 @@ -# What is NVIDIA NeMo Retriever? +# What is NVIDIA NeMo Retriever Library? -NVIDIA NeMo Retriever is a collection of microservices +NVIDIA NeMo Retriever Library is a collection of microservices for building and scaling multimodal data extraction, embedding, and reranking pipelines with high accuracy and maximum data privacy – built with NVIDIA NIM. -NeMo Retriever, part of the [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/) software suite for managing the AI agent lifecycle, +NeMo Retriever Library, part of the [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/) software suite for managing the AI agent lifecycle, ensures data privacy and seamlessly connects to proprietary data wherever it resides, empowering secure, enterprise-grade retrieval. -NeMo Retriever provides the following: +NeMo Retriever Library provides the following: - **Multimodal Data Extraction** — Quickly extract documents at scale that include text, tables, charts, and infographics. - **Embedding + Indexing** — Embed all extracted text from text chunks and images, and then insert into LanceDB (default) or Milvus — accelerated with NVIDIA cuVS. @@ -19,18 +19,18 @@ NeMo Retriever provides the following: ## Enterprise-Ready Features -NVIDIA NeMo Retriever comes with enterprise-ready features, including the following: +NVIDIA NeMo Retriever Library comes with enterprise-ready features, including the following: -- **High Accuracy** — NeMo Retriever exhibits a high level of accuracy when retrieving across various modalities through enterprise documents. -- **High Throughput** — NeMo Retriever is capable of extracting, embedding, indexing and retrieving across hundreds of thousands of documents at scale with high throughput. -- **Decomposable/Customizable** — NeMo Retriever consists of modules that can be separately used and deployed in your own environment. -- **Enterprise-Grade Security** — NeMo Retriever NIMs come with security features such as the use of [safetensors](https://huggingface.co/docs/safetensors/index), continuous patching of CVEs, and more. +- **High Accuracy** — NeMo Retriever Library exhibits a high level of accuracy when retrieving across various modalities through enterprise documents. +- **High Throughput** — NeMo Retriever Library is capable of extracting, embedding, indexing and retrieving across hundreds of thousands of documents at scale with high throughput. +- **Decomposable/Customizable** — NeMo Retriever Library consists of modules that can be separately used and deployed in your own environment. +- **Enterprise-Grade Security** — NeMo Retriever Library NIMs come with security features such as the use of [safetensors](https://huggingface.co/docs/safetensors/index), continuous patching of CVEs, and more. ## Applications -The following are some applications that use NVIDIA NeMo Retriever: +The following are some applications that use NVIDIA NeMo Retriever Library: - [AI Virtual Assistant for Customer Service](https://github.com/NVIDIA-AI-Blueprints/ai-virtual-assistant) (NVIDIA AI Blueprint) - [Build an Enterprise RAG pipeline](https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline/blueprintcard) (NVIDIA AI Blueprint) @@ -43,7 +43,7 @@ The following are some applications that use NVIDIA NeMo Retriever: ## Related Topics -- [NeMo Retriever Text Embedding NIM](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/overview.html) -- [NeMo Retriever Text Reranking NIM](https://docs.nvidia.com/nim/nemo-retriever/text-reranking/latest/overview.html) +- [NeMo Retriever Library Text Embedding NIM](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/overview.html) +- [NeMo Retriever Library Text Reranking NIM](https://docs.nvidia.com/nim/nemo-retriever/text-reranking/latest/overview.html) - [NVIDIA NIM for Object Detection](https://docs.nvidia.com/nim/ingestion/object-detection/latest/overview.html) - [NVIDIA NIM for Image OCR](https://docs.nvidia.com/nim/ingestion/table-extraction/latest/overview.html) diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 678ae4cfa..36cf26dd2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -58,7 +58,7 @@ nav: - NeMo Retriever: - Overview: - Overview: index.md - - NeMo Retriever Extraction: + - NeMo Retriever Library: - Overview: extraction/overview.md - Release Notes: extraction/releasenotes-nv-ingest.md - Get Started: