Additional bindings logic #12
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…database, and new transformers for testing
shabani1
added a commit
that referenced
this pull request
Oct 18, 2023
# What This is a follow up to #12 allowing for the use of `save_results_to_index` instead of having to use two different DB tasks to save embeddings and non-embeddings. - It uses a very inefficient conversion of Numpy arrays to lists - Will update this when updating to Pydantic 2.0 - It requires use of `text_embedding_transformer` instead of `text_embedding` - The former simply returns the result of the latter as `{'embedding': result}` - Need to create a decorator to wrap output with column names as part of a future PR # Test plan Following the test plan of #12, adding a new document to the default collection should now generate two tasks, one for text embedding and one for word counts.
shabani1
added a commit
that referenced
this pull request
Oct 22, 2023
# What
This PR adds the `lexy_transformer` decorator. The decorator can be
imported and applied as follows.
```python
from lexy.transformers import lexy_transformer
@lexy_transformer(name="text.embeddings.minilm")
def text_embeddings(sentences: list[str]) -> torch.Tensor:
...
```
When applied to a function, the decorator will:
* Register the function as a celery shared task with the name
`lexy.transformers.{name}`
* Add the kwarg `lexy_index_fields` to the function signature
* When run without the argument, the decorated function behaves as it
normally would
* With the argument, the decorated function returns its output with the
labels specified in `lexy_index_fields` (see example below)
```python
@lexy_transformer(name="add_and_subtract")
def add_and_subtract(a, b):
return a + b, a - b
add_and_subtract(5, 3)
# returns (8, 2)
add_and_subtract(5, 3, lexy_index_fields=["sum", "difference"])
# returns [{'sum': 8, 'difference': 2}]
```
# Test plan
Similar to the test plan for #12, though bindings need the additional
keyword argument `lexy_index_fields`.
- Create a new transformer with the following payload:
```json
{
"transformer_id": "text.counter.word_counter",
"path": "lexy.transformers.counter.word_counter",
"description": "Returns count of words and the longest word"
}
```
- Create a new index with the following payload:
```json
{
"index_id": "word_counts",
"description": "Word counts",
"index_table_schema": {},
"index_fields": {
"word_count": {"type": "int"},
"longest_word": {"type": "string", "optional": true}
}
}
```
- Ensure that the index table `zzidx__word_counts` has been created
- This is a manual step right now, run
`lexy.core.events.create_new_index_table('word_counts')`
- Create a new binding with the following payload:
```json
{
"collection_id": "default",
"transformer_id": "text.counter.word_counter",
"index_id": "word_counts",
"description": "New binding for word counts",
"execution_params": {},
"transformer_params": {
"lexy_index_fields": ["word_count", "longest_word"]
},
"filters": {}
}
```
The test should produce tasks for each document in the default
collection using the `word_counter` transformer, and save the results in
the index table `zzidx__word_counts`.
- Finally, the existing binding for `default_text_embeddings` (i.e.,
binding with `id=1`) should be patched with the following payload:
```json
{
"transformer_params": {
"lexy_index_fields": ["embedding"]
}
}
```
Now, any new documents added to the default collection should trigger
two jobs, one for each binding.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
This PR adds logic for processing a new binding.
save_result_to_indexworks only for embedding fieldssave_records_to_indexworks only for non-embedding fieldsTest plan
{ "transformer_id": "text.counter.word_counter", "path": "lexy.transformers.counter.word_counter", "description": "Returns count of words and the longest word" }{ "index_id": "word_counts", "description": "Word counts", "index_table_schema": {}, "index_fields": { "word_count": {"type": "int"}, "longest_word": {"type": "string", "optional": true} } }zzidx__word_countshas been createdlexy.core.events.create_new_index_table('word_counts'){ "collection_id": "default", "transformer_id": "text.counter.word_counter", "index_id": "word_counts", "description": "New binding for word counts", "execution_params": {}, "transformer_params": {}, "filters": {} }The test should produce tasks for each document in the default collection using the
word_countertransformer, and save the results in the index tablezzidx__word_counts.