8000 Added the janky version of openai embeddings by shabani1 · Pull Request #45 · lexy-ai/lexy · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Added the janky version of openai embeddings #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 15, 2024
Merged

Added the janky version of openai embeddings #45

merged 1 commit into from
Jan 15, 2024

Conversation

shabani1
Copy link
Contributor

What

Added a transformer for OpenAI embeddings.

It's currently janky in that you can only pass one sentence at a time when using it in a binding. We'll have to fix this by updating lexy.core.celery_tasks.save_records_to_index so that it accepts additional types of mappings/sequences. More to come.

Minimal example

from lexy_py import LexyClient

lx = LexyClient()

# create collection
collection = lx.create_collection('openai_collection')

# create index
index_fields = {
    "text": {"type": "string"},
    "embedding": {"type": "embedding", "extras": {"dims": 1536, "model": "text.embeddings.openai-ada-002"}}
}
index = lx.create_index(
    index_id='openai_text_embeddings',
    description='OpenAI text embeddings',
    index_fields=index_fields
)

# create binding
binding = lx.create_binding(
    collection_id='openai_collection',
    transformer_id='text.embeddings.openai-ada-002',
    index_id='openai_text_embeddings',
    description='OpenAI text embeddings',
    transformer_params=dict(lexy_index_fields=['embedding'])
)

# add documents
collection.add_documents([
    {"content": "OpenAI is an artificial intelligence company based in San Francisco."},
    {"content": "Lexy is a data platform for building AI powered applications."}
])

# query index
index.query('AI companies')

Why

Most people start off using embeddings from OpenAI.

Test plan

  • Run pytest sdk-python
  • Run examples/tests.ipynb and verify that there are no errors
  • Run examples/tutorial.ipynb and verify that the tutorial works as expected
  • Run examples/images.ipynb and verify that the tutorial works as expected

@shabani1 shabani1 merged commit 1f68f96 into main Jan 15, 2024
@shabani1 shabani1 deleted the rs/openai-embeddings branch January 15, 2024 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shabani1
0