10000 Add lexer for Typst · Issue #877 · alecthomas/chroma · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add lexer for Typst #877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
1 task done
miliog opened this issue Oct 30, 2023 · 4 comments · Fixed by #1007
Closed
1 task done

Add lexer for Typst #877

miliog opened this issue Oct 30, 2023 · 4 comments · Fixed by #1007
Labels
help wanted lexer missing Missing a lexer, please contribute

Comments

@miliog
Copy link
miliog commented Oct 30, 2023

Is there an existing issue for this?

  • I have searched the existing issues

What is the missing lexer?

Typst is a new markup language and I've noticed that chroma doesn't support it.

The syntax documentation can be found here:
https://typst.app/docs/reference/syntax/

Links to existing syntax definitions

Github Linguist uses the following:
https://github.com/michidk/typst-grammar/blob/master/grammars/tmlanguage.json

@miliog miliog added help wanted lexer missing Missing a lexer, please contribute labels Oct 30, 2023
@miliog
Copy link
Author
miliog commented Dec 16, 2023

I converted the new pygments lexer (credits go to @jvoisin) to the Chroma XML format using python3 _tools/pygments2chroma_xml.py pygments.lexers.typst.TypstLexer > lexers/embedded/typst.xml. See miliog@d05a092.

Only the line

<rule pattern="(_|^|+|-|/|*|-&gt;|&lt;-|!=|==)\b"><token type="Operator"/></rule>

was broken and an error was shown while running the test. To fix it, I escaped the special characters +, -, /, and * by preceding them with a backslash (\):

<rule pattern="(_|\^|\+|\-|\/|\*|&gt;|&lt;|!=|==)\b"><token type="Operator"/></rule>

However, there are still a few problems remaining, as shown below:

image

My example works just fine in pygments accord 8000 ing to https://pygments.org/demo/:

image

@jonaspleyer
Copy link

FYI I have looked at this problem and put some incremental work into the typst lexer at https://github.com/pygments/pygments It will probably be the easiest if we first develop there and then simply copy the lexer over just as you did before. Happy for feedback on my recent PR over there:
pygments/pygments#2656

oliverpool added a commit to oliverpool/chroma that referenced this issue Jul 8, 2024
@oliverpool
Copy link
Contributor

Combining the latest state pygments/pygments#2724 (with Bin instead of Binary) and the conversion fix proposed in #982, the result look good:
image

If anyone wants to play with it, the converted xml is attached (as TXT to please github...)
typst.xml.txt

alecthomas pushed a commit that referenced this issue Jul 10, 2024
Discovered while building on top of
#877 (comment)
(by @miliog)

The pygments "regex.words" should be escaped, since they may contain
special regex chars (like `^` or `+`).
oliverpool added a commit to oliverpool/chroma that referenced this issue Oct 6, 2024
oliverpool added a commit to oliverpool/chroma that referenced this issue Oct 6, 2024
@oliverpool
Copy link
Contributor

Pygment's MR just got merged, I just opened a PR here, feel free to test it locally: #1007

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted lexer missing Missing a lexer, please contribute
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants
0