Skip to content

tests: Move langchain under toxgen, add variants #4349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 20 commits into
base: master
Choose a base branch
from
Draft

tests: Move langchain under toxgen, add variants #4349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
2cc39ea
tests: Update tox.ini
sentrivana Apr 29, 2025
0beed64
tests: Move anthropic under toxgen
sentrivana Apr 29, 2025
b73b738
adjust versions
sentrivana Apr 29, 2025
04f99f7
Merge branch 'master' into ivana/toxgen/move-more-stuff
sentrivana Apr 29, 2025
944f1b2
tests: Move langchain under toxgen
sentrivana Apr 29, 2025
b50d0f4
Merge branch 'master' into ivana/toxgen/move-even-more-stuff
sentrivana Apr 29, 2025
705d738
notiktoken
sentrivana Apr 29, 2025
98415e6
Merge branch 'master' into ivana/toxgen/move-even-more-stuff
sentrivana May 6, 2025
7666b0e
.
sentrivana May 6, 2025
0c68281
.
sentrivana May 6, 2025
00f42a7
update readme
sentrivana May 6, 2025
2bc7332
wording
sentrivana May 6, 2025
eb5da6d
.
sentrivana May 6, 2025
04874ab
dont provide a key
sentrivana May 6, 2025
47eb9fa
put key back
sentrivana May 6, 2025
0dbfb0c
Merge branch 'master' into ivana/toxgen/move-even-more-stuff
sentrivana May 6, 2025
39fd74d
.
sentrivana May 6, 2025
579e773
lets try this
sentrivana May 6, 2025
b45e641
.
sentrivana May 6, 2025
8584d8f
fix parsing error
sentrivana May 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions scripts/populate_tox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ integration_name: {
},
"python": python_version_specifier,
"include": package_version_specifier,
"variants": another_integration_name,
}
```

Expand Down Expand Up @@ -153,6 +154,30 @@ be expressed like so:
}
```

### `variants`

You might want to test a single integration against two different sets of
dependencies. For example, `langchain` should be tested both with and without
`tiktoken`. In that case:

- add the variant as its own entry in `TEST_SUITE_CONFIG`
- link it from the base variant via the `variants` key

```python
{
"langchain": {
"variants": ["langchain-notiktoken"],
...
},
"langchain-notiktoken": {
...
}
}
```

Both `langchain` and `langchain-notiktoken` will be run as part of the Langchain
CI workflow.


## How-Tos

Expand Down
17 changes: 17 additions & 0 deletions scripts/populate_tox/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,23 @@
"huggingface_hub": {
"package": "huggingface_hub",
},
"langchain": {
"package": "langchain",
"deps": {
"*": ["openai", "tiktoken"],
"<0.2": ["httpx<0.28.0"],
">0.2": ["langchain-community"],
},
"variants": ["langchain_notiktoken"],
},
"langchain_notiktoken": {
"package": "langchain",
"deps": {
"*": ["openai"],
"<0.2": ["httpx<0.28.0"],
">0.2": ["langchain_community"],
},
},
"launchdarkly": {
"package": "launchdarkly-server-sdk",
},
Expand Down
13 changes: 11 additions & 2 deletions scripts/populate_tox/populate_tox.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,6 @@
"chalice",
"gcp",
"httpx",
"langchain",
"langchain_notiktoken",
"openai",
"openai_notiktoken",
"pure_eval",
Expand Down Expand Up @@ -648,6 +646,17 @@ def main(fail_on_changes: bool = False) -> None:
}
)

variants = TEST_SUITE_CONFIG[integration].get("variants") or []
for variant in variants:
packages[group].append(
{
"name": variant,
"package": package,
"extra": extra,
"releases": test_releases,
}
)

if fail_on_changes:
old_file_hash = get_file_hash()

Expand Down
20 changes: 0 additions & 20 deletions scripts/populate_tox/tox.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,6 @@ envlist =
{py3.9,py3.11,py3.12}-httpx-v{0.25,0.27}
{py3.9,py3.12,py3.13}-httpx-latest

# Langchain
{py3.9,py3.11,py3.12}-langchain-v0.1
{py3.9,py3.11,py3.12}-langchain-v0.3
{py3.9,py3.11,py3.12}-langchain-latest
{py3.9,py3.11,py3.12}-langchain-notiktoken

# OpenAI
{py3.9,py3.11,py3.12}-openai-v1.0
{py3.9,py3.11,py3.12}-openai-v1.22
Expand Down Expand Up @@ -236,20 +230,6 @@ deps =
httpx-v0.27: httpx~=0.27.0
httpx-latest: httpx

# Langchain
langchain-v0.1: openai~=1.0.0
langchain-v0.1: langchain~=0.1.11
langchain-v0.1: tiktoken~=0.6.0
langchain-v0.1: httpx<0.28.0
langchain-v0.3: langchain~=0.3.0
langchain-v0.3: langchain-community
langchain-v0.3: tiktoken
langchain-v0.3: openai
langchain-{latest,notiktoken}: langchain
langchain-{latest,notiktoken}: langchain-openai
langchain-{latest,notiktoken}: openai>=1.6.1
langchain-latest: tiktoken~=0.6.0

# OpenAI
openai: pytest-asyncio
openai-v1.0: openai~=1.0.0
Expand Down
3 changes: 3 additions & 0 deletions tests/integrations/langchain/test_langchain.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ def test_langchain_agent(
model_name="gpt-3.5-turbo",
temperature=0,
openai_api_key="badkey",
streaming=True,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prevents a "wrong API key" error on langchain < 1.1

)
agent = create_openai_tools_agent(llm, [get_word_length], prompt)

Expand Down Expand Up @@ -232,6 +233,7 @@ def test_langchain_error(sentry_init, capture_events):
model_name="gpt-3.5-turbo",
temperature=0,
openai_api_key="badkey",
streaming=True,
)
agent = create_openai_tools_agent(llm, [get_word_length], prompt)

Expand Down Expand Up @@ -327,6 +329,7 @@ def test_span_origin(sentry_init, capture_events):
model_name="gpt-3.5-turbo",
temperature=0,
openai_api_key="badkey",
streaming=True,
)
agent = create_openai_tools_agent(llm, [get_word_length], prompt)

Expand Down
53 changes: 32 additions & 21 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# The file (and all resulting CI YAMLs) then need to be regenerated via
# "scripts/generate-test-files.sh".
#
# Last generated: 2025-05-06T10:23:50.156629+00:00
# Last generated: 2025-05-06T12:51:20.834069+00:00

[tox]
requires =
Expand Down Expand Up @@ -77,12 +77,6 @@ envlist =
{py3.9,py3.11,py3.12}-httpx-v{0.25,0.27}
{py3.9,py3.12,py3.13}-httpx-latest

# Langchain
{py3.9,py3.11,py3.12}-langchain-v0.1
{py3.9,py3.11,py3.12}-langchain-v0.3
{py3.9,py3.11,py3.12}-langchain-latest
{py3.9,py3.11,py3.12}-langchain-notiktoken

# OpenAI
{py3.9,py3.11,py3.12}-openai-v1.0
{py3.9,py3.11,py3.12}-openai-v1.22
Expand Down Expand Up @@ -145,6 +139,16 @@ envlist =
{py3.9,py3.11,py3.12}-cohere-v5.11.4
{py3.9,py3.11,py3.12}-cohere-v5.15.0

{py3.9,py3.10,py3.11}-langchain-v0.0.354
{py3.9,py3.11,py3.12}-langchain-v0.1.20
{py3.9,py3.11,py3.12}-langchain-v0.2.17
{py3.9,py3.12,py3.13}-langchain-v0.3.25

{py3.9,py3.10,py3.11}-langchain_notiktoken-v0.0.354
{py3.9,py3.11,py3.12}-langchain_notiktoken-v0.1.20
{py3.9,py3.11,py3.12}-langchain_notiktoken-v0.2.17
{py3.9,py3.12,py3.13}-langchain_notiktoken-v0.3.25

{py3.8,py3.10,py3.11}-huggingface_hub-v0.22.2
{py3.8,py3.10,py3.11}-huggingface_hub-v0.25.2
{py3.8,py3.12,py3.13}-huggingface_hub-v0.28.1
Expand Down Expand Up @@ -398,20 +402,6 @@ deps =
httpx-v0.27: httpx~=0.27.0
httpx-latest: httpx

# Langchain
langchain-v0.1: openai~=1.0.0
langchain-v0.1: langchain~=0.1.11
langchain-v0.1: tiktoken~=0.6.0
langchain-v0.1: httpx<0.28.0
langchain-v0.3: langchain~=0.3.0
langchain-v0.3: langchain-community
langchain-v0.3: tiktoken
langchain-v0.3: openai
langchain-{latest,notiktoken}: langchain
langchain-{latest,notiktoken}: langchain-openai
langchain-{latest,notiktoken}: openai>=1.6.1
langchain-latest: tiktoken~=0.6.0

# OpenAI
openai: pytest-asyncio
openai-v1.0: openai~=1.0.0
Expand Down Expand Up @@ -514,6 +504,27 @@ deps =
cohere-v5.11.4: cohere==5.11.4
cohere-v5.15.0: cohere==5.15.0

langchain-v0.0.354: langchain==0.0.354
langchain-v0.1.20: langchain==0.1.20
langchain-v0.2.17: langchain==0.2.17
langchain-v0.3.25: langchain==0.3.25
langchain: openai
langchain: tiktoken
langchain-v0.0.354: httpx<0.28.0
langchain-v0.1.20: httpx<0.28.0
langchain-v0.2.17: langchain-community
langchain-v0.3.25: langchain-community

langchain_notiktoken-v0.0.354: langchain==0.0.354
langchain_notiktoken-v0.1.20: langchain==0.1.20
langchain_notiktoken-v0.2.17: langchain==0.2.17
langchain_notiktoken-v0.3.25: langchain==0.3.25
langchain_notiktoken: openai
langchain_notiktoken-v0.0.354: httpx<0.28.0
langchain_notiktoken-v0.1.20: httpx<0.28.0
langchain_notiktoken-v0.2.17: langchain_community
langchain_notiktoken-v0.3.25: langchain_community

huggingface_hub-v0.22.2: huggingface_hub==0.22.2
huggingface_hub-v0.25.2: huggingface_hub==0.25.2
huggingface_hub-v0.28.1: huggingface_hub==0.28.1
Expand Down
Loading