Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Full Claude Sonnet 3.5 Support #234

Merged
merged 30 commits into from
Sep 23, 2024
Merged

Full Claude Sonnet 3.5 Support #234

merged 30 commits into from
Sep 23, 2024

Conversation

dillonalaird
Copy link
Member

@dillonalaird dillonalaird commented Sep 11, 2024 •
edited
Loading

This PR adds several big items:

  • Adds AnthropicVisionAgent and makes it the default VisionAgent as well as VisionAgentCoder
  • Adds OpenAIVisionAgent and OpenAIVisionAgentCoder
  • Enables VisionAgent to view media to quickly answer questions about media
  • Adds fine_tune_id to owl_v2 and florence2_sam2_image, and changes meta_tools to handle fine_tune_ids for generic object detectors. This helps an issue where sometimes Claude 3.5 will prefer florence2_phrase_grounding because it can be fine tuned
  • Fixes issue with VisionAgent prompt that causes Claude 3.5 to occasionally output incorrect JSON responses
  • Passes the plan thoughts from planning to the coder, the plan thoughts now also include more information such as thresholds

shankar-vision-eng
shankar-vision-eng requested changes Sep 13, 2024
View reviewed changes
Copy link
Collaborator

@shankar-vision-eng shankar-vision-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments

vision_agent/agent/vision_agent.py Show resolved Hide resolved
vision_agent/agent/vision_agent_coder.py Show resolved Hide resolved
vision_agent/lmm/lmm.py Show resolved Hide resolved
vision_agent/tools/tools.py Show resolved Hide resolved
vision_agent/tools/tools.py Show resolved Hide resolved
@dillonalaird
resize image for claude
1b3d8a8
@dillonalaird
only resize if above size
2b112b5
@dillonalaird
renamed claude to anthropic for consistency
9e71697
@dillonalaird
added openai classes and made anthropic default
e4485fa
@dillonalaird
add ability to view images
bd8d245
@dillonalaird
add florence2 fine tune to owl_v2 args
d64f86d
@dillonalaird
added fine tune id for florence2sam2
ad54abb
@dillonalaird
add generic OD fine tuning
7432e10
@dillonalaird
fixed type error
7d27d63
@dillonalaird
added comment
e13d019
@dillonalaird
fix prompt for florence2 sam2 video tracking
c3c210b
@dillonalaird
fixed import bug
39a8548
@dillonalaird
updated fine tuning names in prompts
30f00f7
@dillonalaird
improve json parsing
51ca06b
@dillonalaird
update json extract, add tests
891def5
@dillonalaird
removed old code
0d9c00b
@dillonalaird
minor improvements to prompt to improve benchmark
54785de
@dillonalaird
pass plan thoughts to coder
fb5cfc3
@dillonalaird
fixed comments
64df1e8
@dillonalaird
fix type and lint errors
14fc101
@dillonalaird
update tests
957ed56
@dillonalaird
make imports easier, pass more code info
152ac13
@dillonalaird
update prompts
c4ee089
@dillonalaird
standardize fps to 1
94f9501
@dillonalaird
rename functions to make them easier to understand by llm
85e2e8a
@dillonalaird
add openai vision agent coder
9b26db3
shankar-vision-eng
shankar-vision-eng approved these changes Sep 23, 2024
View reviewed changes
Copy link
Collaborator

@shankar-vision-eng shankar-vision-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dillonalaird dillonalaird merged commit 696da6c into main Sep 23, 2024
8 checks passed
@dillonalaird dillonalaird deleted the full-claude-35-support branch September 23, 2024 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shankar-vision-eng shankar-vision-eng shankar-vision-eng approved these changes

@MingruiZhang MingruiZhang MingruiZhang approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dillonalaird @MingruiZhang @shankar-vision-eng