8000 Expanded document model with image upload and retrieval by shabani1 · Pull Request #39 · lexy-ai/lexy · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Expanded document model with image upload and retrieval #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 5, 2024
Merged

Expanded document model with image upload and retrieval #39

merged 4 commits into from
Jan 5, 2024

Conversation

shabani1
Copy link
Contributor
@shabani1 shabani1 commented Jan 5, 2024

What

IMPORTANT: This PR contains breaking changes. Read the Miscellaneous section below.

First addition of file based documents, with support for adding Image documents to a collection. This builds on top of the multimodal groundwork added in #37.

  • Support for file-based documents with S3-backed storage
    • Currently adding images only, though the framework supports any file objects
    • Client can upload files using upload_documents method
    • Images are stored in S3 (with optional thumbnails) and retrieved via presigned urls
    • Client Document objects contain logic for obtaining and refreshing retrieval urls
  • Updated image.embeddings.clip transformer to accept Document objects in addition to images
    • Document objects will retrieve their image using presigned url
  • New package storage for cloud storage logic (e.g., s3 clients)
  • Added Alembic for database migrations
  • Added a config field to the Collection model
  • Index query option to return Document object with results
  • New requirements for server
    • Pillow, AWS clients, httpx

This PR does NOT include the following, which need to be added as part of a future PR:

  • New classes for the Document model (e.g., ImageDocument, FileDocument, S3Document, etc.)
    • For now, these attributes are being stored in Document.meta
    • Will move them to their own classes once we have a better idea of what attributes are used
  • Extended documentation and tutorial for multimodal
    • I've added a very concise example in examples/images.ipynb but need to make it more detailed and create a tutorial page in the docs
  • Streamlined AWS configuration
    • Currently there are no checks for AWS credentials or other checks like whether the S3 bucket exists and is accessible

Why

At Lexy, we believe that AGI should be able to see images and access files. This PR is the next step in supporting any arbitrary file-based documents. It adds support for storage and retrieval of those documents, and is currently restricted to images.

Test plan

  • Run pytest sdk-python in terminal
  • Run all cells in examples/tests.ipynb and verify that there are no errors
  • Run all cells in examples/tutorial.ipynb and verify that the tutorial works as expected
  • Run all cells in examples/images.ipynb and verify that the tutorial works as expected

Miscellaneous

This PR contains some breaking changes. The easiest thing to do is nuke and rebuild your containers. If you don't want to do so, you can avoid it by running the following.

# install new dependencies, including alembic and AWS cli
source venv/bin/activate
poetry install --no-root --with test,docs,dev -E "lexy_transformers"

# configure AWS: use the cli to configure or place credentials in `.env`
aws configure

# install new dependencies on your docker containers 
docker exec lexy-server poetry install --no-root -E "lexy_transformers"
docker exec lexy-celeryworker poetry install --no-root -E "lexy_transformers"

# run DB migrations
alembic upgrade head

Configuring AWS

You can use aws configure (recommended) or put AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your .env file.

You'll also need to specify an S3 bucket for file storage (for which your AWS credentials should have full access). You can do so by adding S3_BUCKET=<name-of-your-S3-bucket> to your .env file, or by updating the value of s3_bucket in lexy/core/config.py.

@shabani1 shabani1 merged commit ef3d839 into main Jan 5, 2024
@shabani1 shabani1 deleted the rs/expanded-document-model branch January 5, 2024 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shabani1
0