WIP: Foundation Model Integration #616
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
OCR now includes bboxes, so these are written into the marker lines, now matches pdftext outputs more cloesly
Force span html, because mathml tags inside the span text field are escaped when outputting
Inline detection model is now removed If `--fix_lines` is enabled, math lines should automatically be fixed by the new model Line builder is significantly simpler now :)
To align better with the outputted mathML, while also supporting LaTeX outputs from the LLMs
Provider line bboxes can be wrong even when the text inside is valid. When this happens across a full page, we catch it thanks to layout and other checks. This fixes the case where a few lines have bad boxes, the provider line bboxes are replaced by bbox of the detected line that they are merged into
Prevents re-OCRing of lines inside equations, since the EquationProcessor will handle that anyways Significantly cuts down time on math heavy documents
…uri/marker into foundation-integration
Matches the input distribution of the model much better since it never really trained on `\n` characters, significantly higher number of retained lines now, less than half the runtime on average
Line merging was broken when there were vertical lines in the page, causing the input texts and image slices for OCR to be mis-aligned Since we now use the detected line boxes to replace provider line boxes, this was showing up in the `--fix_lines` mode.
Old logic could not handle the case where a single span had multiple matches. Now the remaining text after a match is preserved and considered for future matches as well
Empty list has a falsy value which was causing it to always be overwritten by the default list when passed in as an arg
Structured extraction
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switching over
OCRBuilderandEquationProcessorto use the new foundation modelAlso removes the need for the inline math detection model, while greatly simplifying
LineBuilderIn addition, adds a special mode -
--fix_linesenabling the model to re-write lines with formatting, math, or garbled text. When this flag is set, every line in the document is passed through the OCR model for potential re-writingPending: