#1558 - Add a LLM-based TextExtractor (OpenAI API compatible) #1559
Conversation
5cbb8f7 to
b130005
Compare
|
looks good at first glance - maybe a more specific name for the module? |
In theory, we could add more functionality to it (such as ParseFilters or similar). Other LLM-based crawelrs have a rich feature set, which we could also add to it. However, I am open to rename it. We could call it |
the 'stormcrawler' part is redundant. Maybe 'ai' is fine as it is |
Thank you for contributing to Apache StormCrawler.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes
Is there a issue associated with this PR? Is it referenced in the commit message?
Does your PR title start with
#XXXXwhereXXXXis the issue number you are trying to resolve?Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
Is the code properly formatted with
mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?For code changes
mvn clean verify?Note
This PR introduces a new TextExtractor implementation — OpenAITextExtractor — which utilizes a Large Language Model (LLM) via an OpenAI-compatible API to extract meaningful content from HTML documents. This allows for semantically rich, context-aware extraction far beyond traditional rule-based methods. Traditional extraction using static rules is brittle and struggles with modern, dynamic layouts. Leveraging LLMs like GPT-4 or LLaMA 3 enables understanding of HTML semantics, resulting in improved text extraction quality.
Key Features are:
Breaking Change (Potential)
This change renames the previously default TextExtractor class and introduces a new TextExtractor interface, making the implementation pluggable. While this may be considered a breaking change, the existing implementation was used privately and not designed for public extensibility. Therefore, in practice, it should be safe — but feedback is welcome.