8000 Issue with Markdown style links. · Issue #27 · tsproisl/SoMaJo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Issue with Markdown style links. #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
PhilipMay opened this issue Feb 6, 2024 · 3 comments
Closed

Issue with Markdown style links. #27

PhilipMay opened this issue Feb 6, 2024 · 3 comments

Comments

@PhilipMay
Copy link

Links in this format: "MD link <https://heise.de> example." have an issue.

Code:

text = "MD link <https://heise.de> example."
sentences = somajo.tokenize_text([text])
for sentence in sentences:
    for token in sentence:
        print(f"{token.text}\t{token.token_class}\t{token.extra_info}")

Returns:

MD	regular	
link	regular	
<	symbol	SpaceAfter=No
https://heise.de/>	URL	
example	regular	SpaceAfter=No
.	symbol

Should return something like this:

MD	regular	
link	regular	
<	symbol	SpaceAfter=No
https://heise.de/	URL
>	symbol
example	regular	SpaceAfter=No
.	symbol

Full code: https://colab.research.google.com/drive/16-CKdzp20Gin02emrLVeHfFFir2veK8M?usp=sharing
@tsproisl
Copy link
Owner
tsproisl commented Feb 8, 2024

This should be closely related to #26 and can probably be fixed in the same manner. I’ll see if I can take a closer look tomorrow.

tsproisl pushed a commit that referenced this issue Feb 9, 2024
Correctly tokenize URLs in angle brackets (#27)
fb788e6
@tsproisl
Copy link
Owner
tsproisl commented Feb 9, 2024

It should be fixed in version 2.4.1!

@PhilipMay
Copy link
Author

it is fixed. Many thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PhilipMay @tsproisl
0