8000 Add Agentic Solver Example by jimthompson5802 · Pull Request #5 · wandb/connections · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add Agentic Solver Example #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Add Agentic Solver Example #5

wants to merge 27 commits into from

Conversation

jimthompson5802
Copy link
@jimthompson5802 jimthompson5802 commented Dec 8, 2024 •
edited
Loading

Example of an agentic solver. This implementation is a variant of the one found in this repo: https://github.com/jimthompson5802/connection_solver.

Some key features of this agentic solver:

  • uses langchain and langgraph frameworks
  • LLM based tools to:
    • LLM generator to create embedding vectors
    • LLM selector for candidate word groups based created by embedding vectors
    • LLM word group recommendation generator and selector
    • Natural Language Puzzle Planner workflow using natural language description of the workflow
    • LLM one-away error analyzer
  • Two phase solver process
    • Phase 1: Use Embedding Vecotor recommendation generation
    • Phase 2: Use LLM puzzle recommendation generation if Phase 1 encounters a mistake
  • Use of multiple LLMs:
    • gpt-3.5-turbo for the agent's planner
    • gpt-4o for generating puzzle recommendations and extract words from image
  • Code-based invalid group detection
  • sqlite3 database to store vocabulary and embedding vectors

Example run

$ python agentic_solver.py --num_samples 50

# output
{'check_final_solution': {'accuracy': {'mean': 3.18}, 'match': {'true_count': 36, 'true_fraction': 0.72}}, 'model_latency': {'mean': 314.7343885421753}}

Here run log for the above run:
agentic_solver_log_num_samples_50.txt

Screenshot of W&B Dashboard
image

Comparison with Other Solvers

Solver num_samples Weave Metrics
agentic_solver.py 50 {'check_final_solution': {'accuracy': {'mean': 3.18}, 'match': {'true_count': 36, 'true_fraction': 0.72}}, 'model_latency': {'mean': 314.7343885421753}}
iterative.py 50 {'check_final_solution': { 'match': {'true_count': 33, 'true_fraction': 0.66}, 'accuracy': {'mean': 3.04}}, 'model_latency': {'mean': 37.08885391712189}}
one_shot.py 50 {'check_final_solution': {'accuracy': {'mean': 1.32}, 'match': {'true_count': 5, 'true_fraction': 0.1}}, 'model_latency': {'mean': 2.800875635147095}}
alpha.py 50 {'model_output': {'score': {'mean': 6.5}}, 'check_final_solution': {'match': {'true_count': 6, 'true_fraction': 0.12}, 'accuracy': {'mean': 1.22}}, 'model_latency': {'mean': 46.59139933109284}}

Run logs for one_shot.py, iterative.py and alpha.py:
one_shot_and_iterative_run_logs_50_samples.txt
alpha_solver_run_log_50_samples.txt

Notes:

  • this PR adds these packages to requirements.txt: pandas, scikit-learn, aiosqlite

@jimthompson5802
setup
fbe23c7
@jimthompson5802
add langchain/langgraph
7fc73bc
@jimthompson5802
add dependency for agentic solution
7a86372
@jimthompson5802
agentic solver dependency
f71301b
@jimthompson5802
WIP: basic flow working off-line test
1e71ac8
@jimthompson5802
WIP
78d2cfc
@jimthompson5802
inital working version with automated checking - standalone test
6c8d9fb
@jimthompson5802
code clean-up
fa44354
@jimthompson5802
fix: string for one_away designation and miss converting json string
5fb33a0
@jimthompson5802
fix: one_away response handling
92632b6
@jimthompson5802
revised end of run test output
2180cf9
@jimthompson5802
code: clean-up
036abd3
@jimthompson5802
code: more clean-up
c32431a
@jimthompson5802
WIP: code clean-up
23b3b58
@jimthompson5802
working with agentic simulator
c019ca2
@jimthompson5802
updated
dbc2d06
@jimthompson5802
refactor for async
36dfbca
@jimthompson5802
WIP
fb91816
@jimthompson5802
fully working version
77be606
@jimthompson5802
refactor: use temporary file for sqlite3 db
d1c5aa3
@jimthompson5802
WIP
1f89f20
@jimthompson5802
WIP: working aiosqlite with sequential processing
80dabd4
@jimthompson5802
fully working version with weave
21a4f9e
@jimthompson5802
code: clean up
2041245
@jimthompson5802
code: update docstrings
983813e
@jimthompson5802
feat: integrate weave decorator
cc50a7f
@socket-security Socket Security
Copy link
socket-security bot commented Dec 8, 2024 •
edited
Loading

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
pypi/aiosqlite@0.20.0 filesystem, network Transitive: environment, eval, shell, unsafe +2 646 kB amyreese
pypi/langgraph@0.2.56 environment, eval, network 0 525 kB hwchase17, nfcampos
pypi/scikit-learn@1.5.2 None 0 0 B

View full report↗︎

@jimthompson5802 jimthompson5802 marked this pull request as ready for review December 8, 2024 04:25
@jimthompson5802
Copy link
Author

@tcapelle @scottire Please review this PR. I'm hoping this will be a good addition to the existing set of sample Connections Puzzle solvers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jimthompson5802
0