-
Notifications
You must be signed in to change notification settings - Fork 12
feat(test): Adding cross runtime tests suite for PocketPaw, LangChain, CrewAI #232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+135
−0
Closed
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
ffad2f0
This commit adds langchain integration test
Akshat-Raj 4c2f9a7
This commit adds a test for CrewAI
Akshat-Raj 51e872b
Adding tests for PocketPaw
Akshat-Raj 2465fa8
This commit adds tests for pocketpaw
Akshat-Raj b61d0ef
This commit adds a readme.md file
Akshat-Raj 39acf33
This commit adds a readme.md file
Akshat-Raj 011fe31
This commit fixes formatting issues
Akshat-Raj 17ee1eb
feat(test): Adding cross runtime tests suite for PocketPaw, LangChain…
Akshat-Raj c3f65f6
This commit fixes formatting
Akshat-Raj File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| ## Cross-Runtime Integration Tests | ||
|
|
||
| The `tests/cross_runtime/` test suite verifies that `.soul` files can be successfully exported to and imported from third-party agent frameworks without losing Core Memory, Semantic Memory, or Episodic Memory. | ||
|
|
||
| Currently supported frameworks and CLIs: | ||
| * **LangChain** (`langchain_core`) | ||
| * **CrewAI** (`crewai`) | ||
| * **PocketPaw** (via the `soul-protocol` Python Runtime API) | ||
|
|
||
| ### Running the Tests | ||
|
|
||
| To run these integration tests locally, execute: | ||
|
|
||
| ```bash | ||
| uv run pytest tests/cross_runtime/ -v | ||
| ``` | ||
|
|
||
| Optional Dependencies: | ||
| To keep the core soul-protocol lightweight, third-party frameworks like LangChain and CrewAI are not strictly required in the default development environment. If LangChain or CrewAI is not installed then the tests will skip instead of failing. | ||
|
|
||
| Python Version Constraints: The crewai ecosystem is currently unstable on newer Python runtimes. | ||
|
|
||
| These tests will automatically run and pass in the official CI pipeline under supported environments (Python <= 3.12). To run the full suite locally without skips, ensure those specific packages are installed and you are using a compatible Python version. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| def pytest_configure(config): | ||
| config.addinivalue_line("markers", "cross_runtime: marks tests for .soul round-trip") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| import pytest | ||
|
|
||
| from soul_protocol import Soul | ||
|
|
||
| pytest.importorskip("crewai", reason="CrewAI is not installed, skipping integration test.") | ||
| from crewai import Agent | ||
|
|
||
|
|
||
| @pytest.mark.cross_runtime | ||
| @pytest.mark.asyncio | ||
| async def test_crewai_integration(tmp_soul_file, tmp_path): | ||
| """A populated soul file must be exported and then imported and must contain the correct persona.""" | ||
|
|
||
| source_file = tmp_soul_file | ||
| soul = await Soul.awaken(source_file) | ||
|
|
||
| await soul.edit_core_memory(persona="CrewAI Agent") | ||
| await soul.note("User's favorite framework is CrewAI") | ||
|
|
||
| crew_agent = Agent( | ||
| role="Programmer", | ||
| goal="Write code", | ||
| backstory=soul.to_system_prompt(), | ||
| verbose=False, | ||
| allow_delegation=False, | ||
| ) | ||
|
|
||
| assert "CrewAI Agent" in crew_agent.backstory | ||
|
|
||
| destination_file = str(tmp_path / "crewai_exported.soul") | ||
| await soul.export(destination_file) | ||
|
|
||
| recovered_soul = await Soul.awaken(destination_file) | ||
| assert "CrewAI Agent" in recovered_soul.to_system_prompt() | ||
|
|
||
| memories = await recovered_soul.recall("favorite framework") | ||
| assert any("favorite framework is CrewAI" in m.content for m in memories), ( | ||
| "Semantic memory failed to persist" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| import pytest | ||
|
|
||
| from soul_protocol import Soul | ||
|
|
||
| pytest.importorskip( | ||
| "langchain_core", reason="LangChain is not installed, skipping integration test." | ||
| ) | ||
| from langchain_core.messages import SystemMessage | ||
|
|
||
|
|
||
| @pytest.mark.cross_runtime | ||
| @pytest.mark.asyncio | ||
| async def test_langchain_integration(tmp_soul_file, tmp_path): | ||
| """A populated soul file must be exported and then imported and must contain the correct persona.""" | ||
|
|
||
| source_file = tmp_soul_file | ||
| soul = await Soul.awaken(source_file) | ||
|
|
||
| await soul.edit_core_memory(persona="LangChain Agent") | ||
| await soul.note("User's favorite framework is LangChain") | ||
| langchain_message = SystemMessage(content=soul.to_system_prompt()) | ||
|
|
||
| assert "LangChain Agent" in langchain_message.content | ||
| destination_file = str(tmp_path / "langchain_exported.soul") | ||
|
|
||
| await soul.export(destination_file) | ||
| recovered_soul = await Soul.awaken(destination_file) | ||
| assert "LangChain Agent" in recovered_soul.to_system_prompt() | ||
|
|
||
| memories = await recovered_soul.recall("favorite framework") | ||
| assert any("favorite framework is LangChain" in m.content for m in memories), ( | ||
| "Semantic memory failed to persist" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| import pytest | ||
|
|
||
| from soul_protocol import Soul | ||
|
|
||
| pytest.importorskip("pocketpaw", reason="Pocketpaw is not installed, skipping integration test.") | ||
| from pocketpaw.config import Settings | ||
| from pocketpaw.soul.manager import SoulManager | ||
|
|
||
|
|
||
| @pytest.mark.cross_runtime | ||
| @pytest.mark.asyncio | ||
| async def test_pocketpaw_integration(tmp_soul_file, tmp_path): | ||
| """A populated soul file must be exported and then imported by pocketpaw and must contain the correct persona.""" | ||
|
|
||
| source_file = tmp_soul_file | ||
| soul = await Soul.awaken(source_file) | ||
|
Akshat-Raj marked this conversation as resolved.
|
||
|
|
||
| await soul.edit_core_memory(persona="PocketPaw Agent") | ||
| await soul.note("User is using soul-protocol") | ||
|
|
||
| exported_file = str(tmp_path / "pocketpaw_exported.soul") | ||
| await soul.export(exported_file) | ||
|
|
||
| settings = Settings( | ||
| soul_path=str(tmp_path / "pocketpaw_loaded.soul"), soul_name="pocketpaw_loaded" | ||
| ) | ||
| manager = SoulManager(settings) | ||
| await manager.import_from_file(exported_file) | ||
|
|
||
| assert manager.soul is not None, "PocketPaw failed to load the soul" | ||
| assert manager.soul.get_core_memory().persona == "PocketPaw Agent", ( | ||
| "Core memory failed to persist through PocketPaw import" | ||
| ) | ||
|
|
||
| memories = await manager.soul.recall("soul-protocol") | ||
| assert any("soul-protocol" in m.content for m in memories), ( | ||
| "Sequential memory failed to persist" | ||
| ) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.