Closed
Description
[ *] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
TestsetGenerator.generate_with_langchain_docs() returns an empty TestDataset object with TypeError: unsupported operand type(s) for -: 'str' and 'int'
Ragas version: 0.1.8
Python version: 3.10.0
Code to Reproduce
from ragas import adapt
from ragas.testset.evolutions import simple, reasoning, multi_context, conditional
from ragas.testset.generator import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
chat = ChatOpenAI(
model="gpt-3.5-turbo",
openai_api_key="token",
openai_api_base=inference_server_url,
max_tokens=1000,
temperature=0.2,
)
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
distributions = {
simple: 0.6,
multi_context: 0.2,
reasoning: 0.2
}
generator = TestsetGenerator.from_langchain(
generator_llm=chat,
critic_llm=chat,
embeddings=embeddings,
)
generator.adapt(language='russian', evolutions=[simple, reasoning, conditional, multi_context])
testset = generator.generate_with_langchain_docs(documents, 1, distributions=distributions, raise_exceptions=False, with_debugging_logs=True)
Error trace
Runner in Executor raised an exception
Traceback (most recent call last):
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\executor.py", line 79, in _aresults
r = await future
File "c:\Users\beznosov_pb\AppData\Local\miniconda3\envs\llmetric\lib\asyncio\tasks.py", line 575, in _wait_for_one
return f.result() # May raise f.exception().
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\executor.py", line 38, in sema_coro
return await coro
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\executor.py", line 112, in wrapped_callable_async
return counter, await callable(*args, **kwargs)
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\testset\evolutions.py", line 144, in evolve
return await self.generate_datarow(
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\testset\evolutions.py", line 210, in generate_datarow
selected_nodes = [
File "C:\Users\beznosov_pb\Documents\RAGAS\ragas3\ragas\src\ragas\testset\evolutions.py", line 213, in <listcomp>
if int(i) - 1 < len(current_nodes.nodes)
TypeError: unsupported operand type(s) for -: 'str' and 'int'
Generating: 100%|██████████| 1/1 [01:39<00:00, 99.74s/it]
Expected behavior
TestsetGenerator.generate_with_langchain_docs() returns a non-empty TestDataset object
Additional context
The error can actually be corrected by editing the file editing ragas\src\ragas\testset\evolutions.py:
210 selected_nodes = [
211 current_nodes.nodes[int(i) - 1]
212 for i in relevant_context_indices
213 if int(i) - 1 < len(current_nodes.nodes)
214 ]
If I were confident in the reliability of this fix, I would create a PR