v24.10.00

Morpheus 24.10.00 (01 Nov 2024)

🚨 Breaking Changes

• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv

🐛 Bug Fixes

• Pin boto3 and s3fs to compatible versions to resolve access denied errors (#2039) @dagardner-nv
• Fix phishing Python API example to match CLI example (#2037) @dagardner-nv
• Model updates and cleanup following upgrade to to triton 24.09 (#2036) @AnuradhaKaruppiah
• Copy data files needed by root_cause_analysis to examples/data (#2032) @AnuradhaKaruppiah
• Fix for duplicate row IDs in log_parsing output (#2031) @dagardner-nv
• Fix log_parsing example pipeline null output issue (#2024) @yczhang-nv
• Fixup file paths in the modular digital fingerprinting documentation. (#2016) @AnuradhaKaruppiah
• Fix DeserializeStage to ensure output messages correctly contain the correct rows for each batch (#2015) @dagardner-nv
• Fix offset calculation when taking a slice of a SlicedMessageMeta (#2006) @dagardner-nv
• Fix CUDF's Column.from_column_view by copying it and adjusting. (#2004) @cwharris
• Fix up file paths in the DFP README (#2003) @AnuradhaKaruppiah
• Fix AttributeError: 'int' object has no attribute 'item' (#1995) @dagardner-nv
• Fix utilities submodule commit (#1987) @cwharris
• Update val-run-all.sh to run cpp pipeline only (#1986) @yczhang-nv
• Fix onnx-to-trt utility (#1984) @dagardner-nv
• Update Utilities submodule and fix compilation with latest build of MRC (#1981) @mdemoret-nv
• Fix missing dependency in DFP Grafana example (#1977) @efajardo-nv
• Populate all the LFS data needed for running examples within the release container (#1976) @AnuradhaKaruppiah
• Ensure timestamps are copied in LLMEngineStage (#1975) @dagardner-nv
• Install sentence-transformers via pip to avoid CPU-torch conda dependencies (#1974) @efajardo-nv
• Add **kwargs back to NVFoundationLLMClient.generate_batch() and generate_batch_async() (#1967) @ashsong-nv
• Benchmark updates/fixes (#1958) @efajardo-nv
• Improve test performance (#1953) @dagardner-nv
• Adopt updated utilities fix in-place Python installs (#1952) @dagardner-nv
• Update cuda version for docker containers (#1941) @dagardner-nv
• Multiple fixes related to SharedProcessPool & MultiProcessingStage (#1940) @yczhang-nv
• Fix dask error in DFP Integrated training pipeline (#1931) @dagardner-nv
• Remove SharedProcessPool.terminate() related tests to avoid stack traces and blocking remote-ci (#1929) @yczhang-nv
• Provide a timeout to the queue.get call in HttpServerSourceStage to avoid spinlocking (#1928) @dagardner-nv
• Ensure that pytest is able to run without optional dependencies (#1927) @dagardner-nv
• Better handle exceptions generated in the LLMEngine to not show the stoul error (#1922) @mdemoret-nv
• Fixing the docker build when Morpheus is a submodule (#1914) @mdemoret-nv
• Build morpheus_llm by default (#1911) @AnuradhaKaruppiah
• Fix conda path for missing llm packages (#1907) @dagardner-nv
• Update WriteToVectorDBStage to re-raise errors from the underlying database (#1905) @dagardner-nv
• Avoid memory leak warnings from pypdfium2 (#1902) @dagardner-nv
• Remove redundant copy of the load_labels_file method (#1901) @dagardner-nv
• Fix Can't find 'action.yml' CI error (#1896) @dagardner-nv
• Fix DFP integrated training Azure pipeline (#1894) @yczhang-nv
• Drop 'CI Pipeline / Check' dependency from the 'package-core' job (#1885) @AnuradhaKaruppiah
• Python source stages now optionally receive a reference to mrc.Subscription (#1881) @dagardner-nv
• Fix Unregistered type : mrc::pymrc::coro::BoostFibersMainPyAwaitable error (#1869) @dagardner-nv
• Revert PR_1736 "Always run the PR builder step even if others are cancelled" (#1860) @AnuradhaKaruppiah
• ensure columns are strings before concatenation (#1857) @cwharris
• Update Kafka DL script to 2.13-3.8.0 (#1856) @cwharris
• Update isort settings file path in fix_all.sh (#1855) @yczhang-nv
• Move isort settings into pyproject.toml (#1854) @dagardner-nv
• Update location of morpheus setup and data files in VS settings (#1843) @AnuradhaKaruppiah
• Fix isort config marking _utils as known first party (#1842) @dagardner-nv
• Fix usage of the C++ impl of write_df_to_file (#1840) @dagardner-nv
• Fix shutdown on Ctrl+C for Python source stages (#1839) @dagardner-nv
• Improved type-hints for stage and source decorators (#1831) @dagardner-nv
• Add tests to confirm that a mis-configured MultiPortModulesStage will raise an exception rather than segfaulting (#1829) @dagardner-nv
• Ensure proper initialization of CMAKE_INSTALL_PREFIX if needed (#1815) @dagardner-nv
• Fix merging of CLI args and Yaml configs in vdb_upload example (#1813) @dagardner-nv
• Fix log parsing undefined variable and duplicate sequence id errors (#1809) @dagardner-nv
• Remove obsolete version string from compose yamls (#1808) @dagardner-nv
• Ensure the release container does not contain any unintended files (#1807) @dagardner-nv
• Update ci/release/update-version.sh to include missed files (#1801) @dagardner-nv

📖 Documentation

• Add known issue for dask shutdown (#2027) @cwharris
• Set the version in the conda packages docs (#2017) @AnuradhaKaruppiah
• Fix mis-leading deserialize stage comments (#2009) @dagardner-nv
• Update Morpheus docs to use cuda 12.5 (#2008) @yczhang-nv
• Fix minor issues with LLM example documentation (#1992) @dagardner-nv
• Incorporate review comments in the conda packages documentation (#1982) @AnuradhaKaruppiah
• Add CPU-only documentation (#1969) @dagardner-nv
• Document each of the Conda environment files (#1932) @dagardner-nv
• Update documentation to reflect CPU-only execution mode (#1924) @dagardner-nv
• Remove TODO statements from documentation (#1879) @dagardner-nv
• Remove automock for merlin as we no longer have/use merlin (#1830) @dagardner-nv
• Add documentation checks to CI (#1821) @dagardner-nv
• Fix documentation links to work in both source repo and documentation builds (#1814) @dagardner-nv
• Update documentation for vdb_upload to use realistic source data with the --file_source flag (#1800) @dagardner-nv

🚀 New Features

• Install morpheus-dfp conda package in the DFP container (#1971) @AnuradhaKaruppiah
• Morpheus docs update post compartmentalization (#1964) @AnuradhaKaruppiah
• Adding implementation of Router Nodes (#1963) @mdemoret-nv
• Include requirements files in the morpheus packages (#1957) @AnuradhaKaruppiah
• Unit tests for the namespace update script (#1954) @AnuradhaKaruppiah
• Script for updating the namespace due to compartmentalization changes (#1946) @AnuradhaKaruppiah
• Move tests/common to tests/morpheus/common (#1942) @AnuradhaKaruppiah
• Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing (#1933) @AnuradhaKaruppiah
• Remove debug log in HttpServerSourceStage when the queue is empty (#1921) @dagardner-nv
• Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) (#1897) @AnuradhaKaruppiah
• Move vector db stages to morpheus-llm (#1889) @AnuradhaKaruppiah
• Scripts for building and uploading the morpheus-core conda package (#1883) @AnuradhaKaruppiah
• Implement MultiProcessingStage (#1878) @yczhang-nv
• Update to RAPIDS 24.10 (#1874) @cwharris
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• [morpheus-refactor] Move morpheus source to python/morpheus (#1836) @AnuradhaKaruppiah
• Support for ControlMessage as an output type for HttpServerSourceStage and HttpClientSourceStage (#1834) @dagardner-nv
• Remove NVTabular (#1825) @cwharris
• Create a Docker image for Morpheus models (#1804) @dagardner-nv
• Add unique column to output of the log_parsing pipeline (#1795) @dagardner-nv

🛠️ Improvements

• Update to Triton Inference Server container version 24.09 (#2001) @dagardner-nv
• Remove temporary DFP todo list (#1998) @AnuradhaKaruppiah
• devcontainer: replace VAULT_HOST with AWS_ROLE_ARN (#1962) @jjacobelli
• Reduce the number of warnings emitted (#1947) @dagardner-nv
• Set lower CPU usage for test_shared_process_pool.py to avoid slowing down the test (#1935) @yczhang-nv
• Remove unused pymysql dependency from DFP mlflow container (#1930) @dagardner-nv
• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Run pre-commit on all files, not just python (#1880) @cwharris
• Prefer len(os.sched_getaffinity(0)) over os.cpu_count() (#1866) @cwharris
• Remove cloudtrail debug log from autoencoder source stage (#1865) @cwharris
• Run yapf, flake8, isort as part of pre-commit (#1859) @cwharris
• Warn when Config's pipeline_batch_size < model_max_batch_size (#1858) @cwharris
• Breakout morpheus_llm (#1853) @AnuradhaKaruppiah
• Install built documentation into release container (#1806) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv
• Batch incoming DOCA raw packet data (#1731) @dagardner-nv

New Contributors

• @AnuradhaKaruppiah made their first contribution in #1836

Full Changelog: v24.06.01...v24.10.00

[NIGHTLY] v25.02.00

🔗 Links

• Development Branch

• Compare with main branch

  There are no changes for v0.25.02.0 yet.

v24.06.01

Morpheus 24.06.01 (23 Aug 2024)

🛠️ Improvements

• Replace pdf parsing libs (#1861) @dagardner-nv

v24.06.00

Morpheus 24.06.00 (03 Jul 2024)

🚨 Breaking Changes

• Introduce multi-endpoint servers and health check endpoints to HttpServerSourceStage (#1734) @jadu-nv
• Update devcontainer to use latest build utils (#1658) @cwharris
• Update CI to install DOCA and build Morpheus DOCA components. (#1622) @cwharris
• Support non-json serializable objects in LLMContext (#1589) @dagardner-nv

🐛 Bug Fixes

• Fix LLM Agents Kafka pipeline (#1793) @dagardner-nv
• Add fetch data command in build_container_release.sh (#1787) @ifengw-nv
• Add cuda and cudf to link targets for C++ examples (#1777) @dagardner-nv
• Release container fixes (#1766) @dagardner-nv
• Remove setting of prog_name, this implies that an executable named morpheus_llm exists (#1759) @dagardner-nv
• Provide a default set of questions for the standalone RAG pipeline (#1758) @dagardner-nv
• Disable shared memory by default, and fix --stop_after flag for vdb_upload example (#1755) @dagardner-nv
• Fix conda errors in release container (#1750) @dagardner-nv
• Fixes for C++ impl for DeserializeStage and add missing get_info overloads to SlicedMessageMeta (#1749) @dagardner-nv
• Add GIT_CLONE_PROTECTION_ACTIVE env config to fix build script (#1748) @jadu-nv
• Fix triton multi threading when using the C++ stage (#1739) @mdemoret-nv
• resolve rapids-dependency-file-generator warning (#1735) @jameslamb
• Updating all uses of the secrets.PROJECT_MANAGEMENT_PAT to use a registered Github App (#1730) @mdemoret-nv
• DOCA: fix optional deps + remove PreallocatorMixin from source stage (#1729) @e-ago
• Remove pyarrow_hotfix import from __init__.py (#1692) @efajardo-nv
• Support the filter_null parameter in the C++ impl of the FileSourceStage (#1689) @dagardner-nv
• Enable C++ mode for abp_pcap_detection example (#1687) @dagardner-nv
• Strip HTML & XML tags from RSS feed input (#1670) @dagardner-nv
• Truncate strings exceeding max_length when inserting to Milvus (#1665) @dagardner-nv
• Fix a typo in the devcontainer base image (#1638) @cwharris
• Fix tests to detect issue #1626 (#1629) @dagardner-nv
• Fix cupy_to_tensor to also infer uint8 and int8 dtypes (#1621) @efajardo-nv
• Add struct column support to cudf_helpers (#1538) @efajardo-nv

📖 Documentation

• Cleanup docs so that each as a single H1 title (#1794) @dagardner-nv
• Mark the LLM Agents Kafka pipeline as broken (#1792) @dagardner-nv
• Document supported environments for each example (#1786) @dagardner-nv
• Removes unused environment variables from Morpheus build docs (#1784) @yczhang-nv
• Remove documentation for yaml config files in vdb_upload pipeline until #1752 is resolved (#1778) @dagardner-nv
• Remove out of date instructions from contributing.md (#1774) @dagardner-nv
• Add troubleshooting reference for unsuccessful documentation builds (#1768) @ifengw-nv
• Remove pre-built container section from getting_started.md (#1764) @yczhang-nv
• Clarify Documentation: Run fetch_data.py Outside Docker Container (#1762) @ifengw-nv
• Add function return documentation for LLMService (#1721) @acaklovic-nv
• Fix description for cache_mode option of DFP Rolling Window module (#1707) @efajardo-nv
• Update root-cause-analysis-model-card.md (#1684) @HesAnEasyCoder
• Update abp-model-card.md (#1683) @HesAnEasyCoder
• Update dfp-model-card.md (#1682) @HesAnEasyCoder
• Update gnn-fsi-model-card.md (#1681) @HesAnEasyCoder
• Update phishing-model-card.md (#1680) @HesAnEasyCoder
• Update examples to execute from the root of the repo (#1674) @dagardner-nv
• Update dfp-model-card.md (#1644) @HesAnEasyCoder

🚀 New Features

• Add ransomware model to devcontainer (#1785) @yczhang-nv
• Introduce multi-endpoint servers and health check endpoints to HttpServerSourceStage (#1734) @jadu-nv
• Support ControlMessage for Preprocess and PostProcess stages (#1623) @yczhang-nv
• Update CI to install DOCA and build Morpheus DOCA components. (#1622) @cwharris
• DOCA stage split: source + convert (#1617) @e-ago
• ControlMessage support in TritonInferenceStage and PreallocatorMixin (#1610) @cwharris

🛠️ Improvements

• Merge Agent Morpheus changes (#1760) @dagardner-nv
• Fix RabbitMQ output path (#1756) @yczhang-nv
• Misc improvements for sid_visualization example (#1751) @dagardner-nv
• Auditing the dependencies and syncing dependencies.yaml with meta.yaml (#1743) @mdemoret-nv
• Always run the PR builder step even if others are cancelled (#1736) @mdemoret-nv
• Enable Python install by default in compile.sh (#1724) @dagardner-nv
• Generate deprecation warning for MultiMessage (#1719) @yczhang-nv
• Improve the logging tests and add support for resetting the logger (#1716) @mdemoret-nv
• Allow passing metadata to LangChainAgentNode._run_single (#1710) @dagardner-nv
• Support passing a custom parser to HttpServerSourceStage and HttpClientSourceStage stages (#1705) @dagardner-nv
• Use EnvConfigValue for passing env-configured arguments to services (#1704) @cwharris
• Remove unused MLflow client arg from DFP inference implementations (#1700) @efajardo-nv
• Add group by column stage (#1699) @dagardner-nv
• Fix non-deterministic output of gnn sampler (#1677) @tzemicheal
• Ensuring consistent use of the export macro MORPHEUS_EXPORT (#1672) @aserGarcia
• Update devcontainer to use latest build utils (#1658) @cwharris
• Update ControlMessage to hold arbitrary Python objects & update MessageMeta to copy & slice (#1637) @yczhang-nv
• Use conda env create --yes instead of --force (#1636) @efajardo-nv
• Misc CI improvements (#1618) @dagardner-nv
• Support non-json serializable objects in LLMContext (#1589) @dagardner-nv

[NIGHTLY] v24.10.00

🔗 Links

• Development Branch
• Compare with main branch

🚨 Breaking Changes

• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv

🐛 Bug Fixes

• Various Lint fixes (#2041) @dagardner-nv
• Pin boto3 and s3fs to compatible versions to resolve access denied errors (#2039) @dagardner-nv
• Fix phishing Python API example to match CLI example (#2037) @dagardner-nv
• Model updates and cleanup following upgrade to to triton 24.09 (#2036) @AnuradhaKaruppiah
• Copy data files needed by root_cause_analysis to examples/data (#2032) @AnuradhaKaruppiah
• Fix for duplicate row IDs in log_parsing output (#2031) @dagardner-nv
• Fix log_parsing example pipeline null output issue (#2024) @yczhang-nv
• Fixup file paths in the modular digital fingerprinting documentation. (#2016) @AnuradhaKaruppiah
• Fix DeserializeStage to ensure output messages correctly contain the correct rows for each batch (#2015) @dagardner-nv
• Fix offset calculation when taking a slice of a SlicedMessageMeta (#2006) @dagardner-nv
• Fix CUDF's Column.from_column_view by copying it and adjusting. (#2004) @cwharris
• Fix up file paths in the DFP README (#2003) @AnuradhaKaruppiah
• Fix AttributeError: 'int' object has no attribute 'item' (#1995) @dagardner-nv
• Fix utilities submodule commit (#1987) @cwharris
• Update val-run-all.sh to run cpp pipeline only (#1986) @yczhang-nv
• Fix onnx-to-trt utility (#1984) @dagardner-nv
• Update Utilities submodule and fix compilation with latest build of MRC (#1981) @mdemoret-nv
• Fix missing dependency in DFP Grafana example (#1977) @efajardo-nv
• Populate all the LFS data needed for running examples within the release container (#1976) @AnuradhaKaruppiah
• Ensure timestamps are copied in LLMEngineStage (#1975) @dagardner-nv
• Install sentence-transformers via pip to avoid CPU-torch conda dependencies (#1974) @efajardo-nv
• Add **kwargs back to NVFoundationLLMClient.generate_batch() and generate_batch_async() (#1967) @ashsong-nv
• Benchmark updates/fixes (#1958) @efajardo-nv
• Improve test performance (#1953) @dagardner-nv
• Adopt updated utilities fix in-place Python installs (#1952) @dagardner-nv
• Update cuda version for docker containers (#1941) @dagardner-nv
• Multiple fixes related to SharedProcessPool & MultiProcessingStage (#1940) @yczhang-nv
• Fix dask error in DFP Integrated training pipeline (#1931) @dagardner-nv
• Remove SharedProcessPool.terminate() related tests to avoid stack traces and blocking remote-ci (#1929) @yczhang-nv
• Provide a timeout to the queue.get call in HttpServerSourceStage to avoid spinlocking (#1928) @dagardner-nv
• Ensure that pytest is able to run without optional dependencies (#1927) @dagardner-nv
• Better handle exceptions generated in the LLMEngine to not show the stoul error (#1922) @mdemoret-nv
• Fixing the docker build when Morpheus is a submodule (#1914) @mdemoret-nv
• Build morpheus_llm by default (#1911) @AnuradhaKaruppiah
• Fix conda path for missing llm packages (#1907) @dagardner-nv
• Update WriteToVectorDBStage to re-raise errors from the underlying database (#1905) @dagardner-nv
• Avoid memory leak warnings from pypdfium2 (#1902) @dagardner-nv
• Remove redundant copy of the load_labels_file method (#1901) @dagardner-nv
• Fix Can't find 'action.yml' CI error (#1896) @dagardner-nv
• Fix DFP integrated training Azure pipeline (#1894) @yczhang-nv
• Drop 'CI Pipeline / Check' dependency from the 'package-core' job (#1885) @AnuradhaKaruppiah
• Python source stages now optionally receive a reference to mrc.Subscription (#1881) @dagardner-nv
• Fix Unregistered type : mrc::pymrc::coro::BoostFibersMainPyAwaitable error (#1869) @dagardner-nv
• Revert PR_1736 "Always run the PR builder step even if others are cancelled" (#1860) @AnuradhaKaruppiah
• ensure columns are strings before concatenation (#1857) @cwharris
• Update Kafka DL script to 2.13-3.8.0 (#1856) @cwharris
• Update isort settings file path in fix_all.sh (#1855) @yczhang-nv
• Move isort settings into pyproject.toml (#1854) @dagardner-nv
• Update location of morpheus setup and data files in VS settings (#1843) @AnuradhaKaruppiah
• Fix isort config marking _utils as known first party (#1842) @dagardner-nv
• Fix usage of the C++ impl of write_df_to_file (#1840) @dagardner-nv
• Fix shutdown on Ctrl+C for Python source stages (#1839) @dagardner-nv
• Improved type-hints for stage and source decorators (#1831) @dagardner-nv
• Add tests to confirm that a mis-configured MultiPortModulesStage will raise an exception rather than segfaulting (#1829) @dagardner-nv
• Ensure proper initialization of CMAKE_INSTALL_PREFIX if needed (#1815) @dagardner-nv
• Fix merging of CLI args and Yaml configs in vdb_upload example (#1813) @dagardner-nv
• Fix log parsing undefined variable and duplicate sequence id errors (#1809) @dagardner-nv
• Remove obsolete version string from compose yamls (#1808) @dagardner-nv
• Ensure the release container does not contain any unintended files (#1807) @dagardner-nv
• Update ci/release/update-version.sh to include missed files (#1801) @dagardner-nv

📖 Documentation

• Add known issue for dask shutdown (#2027) @cwharris
• Set the version in the conda packages docs (#2017) @AnuradhaKaruppiah
• Fix mis-leading deserialize stage comments (#2009) @dagardner-nv
• Update Morpheus docs to use cuda 12.5 (#2008) @yczhang-nv
• Fix minor issues with LLM example documentation (#1992) @dagardner-nv
• Incorporate review comments in the conda packages documentation (#1982) @AnuradhaKaruppiah
• Add CPU-only documentation (#1969) @dagardner-nv
• Document each of the Conda environment files (#1932) @dagardner-nv
• Update documentation to reflect CPU-only execution mode (#1924) @dagardner-nv
• Remove TODO statements from documentation (#1879) @dagardner-nv
• Remove automock for merlin as we no longer have/use merlin (#1830) @dagardner-nv
• Add documentation checks to CI (#1821) @dagardner-nv
• Fix documentation links to work in both source repo and documentation builds (#1814) @dagardner-nv
• Update documentation for vdb_upload to use realistic source data with the --file_source flag (#1800) @dagardner-nv

🚀 New Features

• Install morpheus-dfp conda package in the DFP container (#1971) @AnuradhaKaruppiah
• Morpheus docs update post compartmentalization (#1964) @AnuradhaKaruppiah
• Adding implementation of Router Nodes (#1963) @mdemoret-nv
• Include requirements files in the morpheus packages (#1957) @AnuradhaKaruppiah
• Unit tests for the namespace update script (#1954) @AnuradhaKaruppiah
• Script for updating the namespace due to compartmentalization changes (#1946) @AnuradhaKaruppiah
• Move tests/common to tests/morpheus/common (#1942) @AnuradhaKaruppiah
• Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing (#1933) @AnuradhaKaruppiah
• Remove debug log in HttpServerSourceStage when the queue is empty (#1921) @dagardner-nv
• Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) (#1897) @AnuradhaKaruppiah
• Move vector db stages to morpheus-llm (#1889) @AnuradhaKaruppiah
• Scripts for building and uploading the morpheus-core conda package (#1883) @AnuradhaKaruppiah
• Implement MultiProcessingStage (#1878) @yczhang-nv
• Update to RAPIDS 24.10 (#1874) @cwharris
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• [morpheus-refactor] Move morpheus source to python/morpheus (#1836) @AnuradhaKaruppiah
• Support for ControlMessage as an output type for HttpServerSourceStage and HttpClientSourceStage (#1834) @dagardner-nv
• Remove NVTabular (#1825) @cwharris
• Create a Docker image for Morpheus models (#1804) @dagardner-nv
• Add unique column to output of the log_parsing pipeline (#1795) @dagardner-nv

🛠️ Improvements

• Update to Triton Inference Server container version 24.09 (#2001) @dagardner-nv
• Remove temporary DFP todo list (#1998) @AnuradhaKaruppiah
• devcontainer: replace VAULT_HOST with AWS_ROLE_ARN (#1962) @jjacobelli
• Reduce the number of warnings emitted (#1947) @dagardner-nv
• Set lower CPU usage for test_shared_process_pool.py to avoid slowing down the test (#1935) @yczhang-nv
• Remove unused pymysql dependency from DFP mlflow container (#1930) @dagardner-nv
• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Run pre-commit on all files, not just python (#1880) @cwharris
• Prefer len(os.sched_getaffinity(0)) over os.cpu_count() (#1866) @cwharris
• Remove cloudtrail debug log from autoencoder source stage (#1865) @cwharris
• Run yapf, flake8, isort as part of pre-commit (#1859) @cwharris
• Warn when Config's pipeline_batch_size < model_max_batch_size (#1858) @cwharris
• Breakout morpheus_llm (#1853) @AnuradhaKaruppiah
• Install built documentation into release container (#1806) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv
• Batch incoming DOCA raw packet data (#1731) @dagardner-nv

v24.03.02

🐛 Bug Fixes

• Don't set pe_count for the C++ impl of the TritonInferenceStage (#1640) @dagardner-nv
• Fix vdb_upload runtime error (#1643) @dagardner-nv

📖 Documentation

• Document current known issues in 24.03.02 (#1656) @dagardner-nv
• Fix documentation for building examples (#1659) @dagardner-nv
• Fix type-o in documentation (#1662) @dagardner-nv
• Fix mis-spelling of examples (#1664) @dagardner-nv

⚠️ Known Issues

• TrainAEStage fails with a Segmentation fault (#1641)
• vdb_upload example pipeline triggers an internal error in Triton (#1649)
• vdb_upload example pipeline error on inserting large strings (#1650)
• vdb_upload example pipeline only works with C++ mode disabled (#1651)

v24.03.01

🚨 Breaking Changes

• Move MemoryDescriptor to the morpheus namespace (#1602) @dagardner-nv

🐛 Bug Fixes

• Switch to kafka 3.5.2 (#1612) @dagardner-nv
• Update mlflow to avoid CVE-2024-27132 and CVE-2024-27133 (#1609) @dagardner-nv
• Fix for databricks_cli import error (#1604) @dagardner-nv
• Move MemoryDescriptor to the morpheus namespace (#1602) @dagardner-nv

v24.03.00

🚨 Breaking Changes

• Updating nlohman_json to 3.11 to match MRC (#1596) @mdemoret-nv
• Add retry logic and proxy support to the NeMo LLM Service (#1544) @mdemoret-nv
• Upgrade openai version to 1.13 and langchain to version 0.1.9 (#1529) @mdemoret-nv
• Make start_async() available to source stages (#1523) @efajardo-nv
• RAPIDS 24.02 Upgrade (#1468) @cwharris
• Decouple TritonInferenceStage from pipeline mode (#1402) @dagardner-nv

🐛 Bug Fixes

• Serialize datetime objects into the module config (#1592) @dagardner-nv
• Remove the defaults channel from dependencies.yml (#1584) @mdemoret-nv
• Fix iso_date_regex_pattern config in file_batcher module and allow override (#1580) @efajardo-nv
• Update DFP MLflow ModelManager to handle model retrieval using file URI (#1578) @efajardo-nv
• Fix configure_logging in DFP benchmarks (#1553) @efajardo-nv
• Catch langchain agent errors (#1539) @dagardner-nv
• Adding missing dependency on pydantic (#1535) @yuchenz427
• Fix memory leak in the mutable dataframe checkout/checkin code (#1534) @dagardner-nv
• Fix pathlib.Path support for FileSourceStage (#1531) @dagardner-nv
• Make start_async() available to source stages (#1523) @efajardo-nv
• Update CI Containers (#1521) @cwharris
• Fix intermittent segfault on interpreter shutdown (#1513) @dagardner-nv
• Adopt updated builds of CI runners (#1503) @dagardner-nv
• Update mlflow plugin version for deployments fix (#1499) @pdmack
• Add runtime environment output to fix building the release container (#1496) @cwharris
• Fix logging of sleep time (#1493) @dagardner-nv
• Pin pytest to <8 (#1485) @dagardner-nv
• Improve pipeline stop logic to ensure join is called exactly once for all stages (#1479) @efajardo-nv
• Fix expected JSON config file extension in logger (#1471) @efajardo-nv
• Fix Loss Function to Improve Model Convergence for AutoEncoder (#1460) @hsin-c
• GNN fraud detection notebook fix (#1450) @efajardo-nv
• Eliminate Redundant Fetches in RSS Controller (#1442) @bsuryadevara
• Updating the workspace settings to remove deprecated python options (#1440) @mdemoret-nv
• Improve camouflage startup issues (#1436) @dagardner-nv
• Fixes to modular DFP examples and benchmarks (#1429) @efajardo-nv

📖 Documentation

• Update minimum compute requirements to Volta (#1594) @dagardner-nv
• Fix broken link in getting started with Morpheus doc (#1494) @edknv
• Update abp-model-card.md (#1439) @drobison00
• Update gnn-fsi-model-card.md (#1438) @drobison00
• Update phishing-model-card.md (#1437) @drobison00
• Document incompatible mlflow models issue (#1434) @dagardner-nv

🚀 New Features

• Adding retry logic to the TritonInferenceStage to allow recovering from errors (#1548) @cwharris
• Create a base mixin class for ingress & egress stages (#1473) @dagardner-nv
• RAPIDS 24.02 Upgrade (#1468) @cwharris
• Install headers & morpheus-config.cmake (#1448) @dagardner-nv

🛠️ Improvements

• Updating nlohman_json to 3.11 to match MRC (#1596) @mdemoret-nv
• DOCA 2.6 from public repo (#1588) @e-ago
• Support ControlMessage for PreProcessNLPStage PreProcessFILStage AddScoreStageBase (#1573) @yuchenz427
• Update MLflow in Production DFP example to use Python 3.10 (#1572) @efajardo-nv
• Fix environment yaml paths (#1551) @efajardo-nv
• Add retry logic and proxy support to the NeMo LLM Service (#1544) @mdemoret-nv
• Update to match new MRC function sig for AsyncioRunnable::on_data (#1541) @dagardner-nv
• Expose max_retries parameter to OpenAIChatService & OpenAIChatClient (#1536) @dagardner-nv
• Upgrade openai version to 1.13 and langchain to version 0.1.9 (#1529) @mdemoret-nv
• Update ops-bot.yaml (#1528) @AyodeAwe
• Add the ability to attach Tensor objects and timestamps to ControlMessage (#1511) @drobison00
• Fix or silence warnings emitted during tests (#1501) @dagardner-nv
• Support ControlMessage output in the C++ impl of DeserializeStage (#1478) @dagardner-nv
• DOCA Source Stage improvements (#1475) @e-ago
• Update copyright headers for 2024 (#1474) @efajardo-nv
• Add conda builds to CI (#1466) @dagardner-nv
• Grafana log monitoring and error alerting example (#1463) @efajardo-nv
• Misc Conda Improvements (#1462) @dagardner-nv
• Simplification of the streaming RAG ingest example to improve usability (#1454) @drobison00
• Replace GPUtil with pynvml for benchmark reports (#1451) @efajardo-nv
• Misc test improvements (#1447) @dagardner-nv
• Add a --manual_seed flag to the CLI (#1445) @dagardner-nv
• Optionally skip ci based on a label in the pr (#1444) @dagardner-nv
• Refactor verification of optional dependencies (#1443) @dagardner-nv
• Use dependencies.yaml as source-of-truth for environment files. (#1441) @cwharris
• Add mocked test & benchmark for LLM agents pipeline (#1424) @dagardner-nv
• Add benchmarks for stand-alone RAG & vdb upload pipelines (#1421) @dagardner-nv
• Add benchmark for completion pipeline (#1414) @dagardner-nv
• Decouple TritonInferenceStage from pipeline mode (#1402) @dagardner-nv

Full Changelog: v23.11.01...v24.03.00

v23.11.01

🐛 Bug Fixes

• Convert models/ransomware-models/ransomw-model-short-rf-20220126.sav to LFS (#1408) @mdemoret-nv

📖 Documentation

• Cloud deployment guide fixes (#1406) @dagardner-nv

Full Changelog: v23.11.00...v23.11.01

v23.11.00

🚨 Breaking Changes

• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv

🐛 Bug Fixes

• CVE-2023-47248 Mitigation (#1399) @mdemoret-nv
• Fixing the hammah and phishing validation pipelines (#1398) @mdemoret-nv
• Fix the SID Viz workflow shutdown process with the new pipeline shutdown process (#1392) @mdemoret-nv
• Fix race condition in the C++ impl for the pre-process fil stage (#1390) @dagardner-nv
• Fixing the conda-build with DOCA enabled and upgrading to CMake 3.25 (#1386) @mdemoret-nv
• Add missing milvus marker (#1385) @dagardner-nv
• Register DataBricksDeltaLakeSourceStage with the CLI (#1384) @dagardner-nv
• Guard optional dependencies in try/except blocks (#1382) @dagardner-nv
• Fix test_vdb_upload_pipe (#1381) @dagardner-nv
• DFP container updates (#1347) @efajardo-nv
• Removed Mutex Related Milvus Tests (#1325) @bsuryadevara
• Pin cuda-python to 11.8.2 as a work around for 11.8.3 incompatibility. (#1320) @drobison00
• Forward-merge branch-23.07 to branch-23.11 [resolved conflicts] (#1246) @dagardner-nv
• Fix tests to support mlflow v2.7 (#1220) @dagardner-nv
• Update DFP training model_kwargs (#1216) @efajardo-nv
• Fix Kafka offset checking test (#1212) @dagardner-nv
• Add review_requested as a trigger & increased timeouts for camouflage (#1200) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv
• Update feature length for test_abp_fil_e2e benchmark (#1188) @dagardner-nv
• Make manual_seed an autouse fixture for gnn_fraud_detection_pipeline tests (#1165) @dagardner-nv
• Update Downloader utility class to use static dask cluster (#1161) @efajardo-nv
• Update to handle GitHub CLI not installed (#1157) @efajardo-nv
• Update TimeSeries stage to also work with Production DFP (#1121) @efajardo-nv
• Fix issue where DFPFileToDataFrameStage logs messages about S3 even when S3 is not in use (#1120) @dagardner-nv
• Fix Tests broken by Test Reorganization (#1118) @cwharris
• Break circular reference issue causing a memory leak (#1115) @dagardner-nv
• Fix intermittent failures in test_dfencoder_distributed_e2e test (#1113) @dagardner-nv
• Resolve forward merger conflices for branch-23.11 (#1092) @mdemoret-nv
• Fix local CI building from a specific commit (#1083) @dagardner-nv

📖 Documentation

• Grafana example readme update (#1393) @efajardo-nv
• Align model card requirements (#1388) @drobison00
• Docs update to indicate use of conda-merge to generate install files (#1387) @drobison00
• Stage documentation improvements (#1362) @dagardner-nv
• Documentation patch for Examples (#1357) @pranavm7
• Update developer documentation to reflect new compute_schema changes (#1341) @dagardner-nv
• Create LICENSE.psycopg2 (#1295) @exactlyallan
• Fix documentation for morpheus.loaders.sql_loader (#1264) @dagardner-nv
• Phishing example fix (#1215) @efajardo-nv
• ABP PCAP detection readme update (#1205) @efajardo-nv
• Command line examples for module-based DFP pipelines (#1154) @efajardo-nv
• Update DFP E2E Benchmarks README to use dev container (#1125) @efajardo-nv
• Less intrusive doc builds (#1060) @dagardner-nv

🚀 New Features

• Add source & stage decorators (#1364) @dagardner-nv
• Add a Vector Database Service to allow stages to read and write to VDBs (#1225) @bsuryadevara
• CI test stage no longer depends on build stage (#1219) @dagardner-nv
• Updates for MRC/Morpheus to build in the same RAPIDS devcontainer environment (#1171) @cwharris
• KafkaSourceStage OAuth Callback Support (#1169) @cwharris
• GitHub Project Automation and Infra Updates (#1168) @jarmak-nv
• Elasticsearch Sink Module (#1163) @bsuryadevara
• RSS Source Stage for Reading RSS Feeds (#1149) @bsuryadevara
• Add parser_kwargs to FileSourceStage to support json files (#1137) @cwharris
• Add a --viz_direction flag to CLI (#1119) @dagardner-nv
• Adds support to read and write to Databricks delta tables (#630) @pthalasta

🛠️ Improvements

• LLM C++ test and doc updates (#1379) @efajardo-nv
• Merge fea-sherlock feature branch into branch-23.11 (#1359) @drobison00
• Make dfp_azure_pipeline inference output file configurable. (#1290) @drobison00
• Loosen nodejs version restriction (#1262) @dagardner-nv
• Use conda environment yaml's for training-tuning-scripts (#1256) @efajardo-nv
• Cherry pick to pull in august DFP enhancements (#1248) @drobison00
• [DRAFT] Add model and experiment template 'click' options to dfp example pipelines, and make model names Databricks compatible. (#1245) @drobison00
• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Adopt updated camouflage-server & fix test_dfp_mlflow_model_writer (#1195) @dagardner-nv
• Add development deps to dependencies.yaml (#1193) @cwharris
• Update to clang-16 & boost-1.82 (#1186) @dagardner-nv
• Scope Zookeeper & Kafka fixtures to session (#1160) @dagardner-nv
• Use copy-pr-bot (#1159) @ajschmidt8
• Update dfp_training stage to support ControlMessages or MultiDFPMessages (#1155) @drobison00
• Prefer conda package over pip dependencies (#1135) @cwharris
• Add tasks and metadata properties to python ControlMessage (#1134) @cwharris
• Eliminate redundant code blocks in modules and stages (#1123) @bsuryadevara
• update devcontainer base to 23.10 (#1116) @cwharris
• Slimmed down CI runners and published artifact urls (#1112) @dagardner-nv
• Updating tests to force .pyi files to be committed into the repo (#1111) @mdemoret-nv
• ABP nvsmi sample data generation (#1108) @efajardo-nv
• Reorganize C++ Tests (#1095) @cwharris
• Improve gitutils.py by using the Github CLI when available (#1088) @mdemoret-nv
• Fixing linting errors which could not be resolved in 23.07 (#1082) @mdemoret-nv
• Move testing utilities into tests/_utils (#1065) @dagardner-nv
• Update Versions for v23.11.00 (#1059) @mdemoret-nv
• Update GNN stellargraph with DGL (#1032) @tzemicheal
• Implement rest data loader (#986) @yuchenz427
• Adding HTTP sources & sinks (#977) @dagardner-nv

New Contributors

• @pthalasta made their first contribution in #630
• @exactlyallan made their first contribution in #1295
• @pranavm7 made their first contribution in #1357

Full Changelog: v23.07.03...v23.11.00