ft2fea-feature file dumper #479

graphicore · 2016-01-22T21:15:53Z

I'm proposing to add a feature file generator to fontTools that I just wrote for a font project of mine.
The tool is in an early state so this could be a living Pull-Request until it's sufficient for you.

This PR has two parts: the fea-rendering library ft2fea.py and a command line tool ft2feaCLI.py.
I decided to add the CLI so that I can use the fea-generator while I'm completing it. Otherwise I had to come up with another way to silence output that I don't need in my build process.

The tool outputs GDEF, GPOS and GSUB. However, I only implemented the GPOS Lookup Types 4, 5 and 6 (MarkToBase, MarkToLigature, MarkToMark) yet.

The command line tool features a query api that makes it possible to export just selected parts of the fea and honors dependency relations (like a Lookupflag that requires a defined Mark Attachment Class from GDEF) .

Sample output of $ ft2feaCLI.py fontfile.otf > dumped.fea

The help output of the tool:

$ ./ft2feaCLI.py -h
usage: ft2feaCLI.py [-h] [-r, --request [REQUEST]]
                    [-w, --whitelist [WHITELIST]]
                    [-b, --blacklist [BLACKLIST]] [-m, --mute [MUTE]]
                    FONT-PATH

Generate an OpenType Feature file (fea) from an otf/ttf font.

positional arguments:
  FONT-PATH             A ttf or otf OpenType font file.

optional arguments:
  -h, --help            show this help message and exit
  -r, --request [REQUEST]
                        Select the fea contents to export. Default: "**"
  -w, --whitelist [WHITELIST]
                        Whitelist fea contents.
  -b, --blacklist [BLACKLIST]
                        Blacklist fea contents.
  -m, --mute [MUTE]     Mute the printed output of fea contents.

The --request, --whitelist, --blacklist and --mute options; all take
as argument the same type of selector definition, which specifies the
the members of each set. See "The Selector Specification" below.

-r, --request
      Define the items that should be exported. If no request is
      defined (default) all items are requested (like: -r "**").
      If an empty request is defined (like: -r "") no item is requested.
      Items that are not directly requested are marked as "Maybe"
      which means that they are exported if they are a dependency
      of other items.
      E.g.: Lookups are dependencies of features. In order to export
      a feature, the lookups of the feature will be exported as well.
      If all of its lookups are are filtered (by --whitelist
      or --blacklist) the feature will not be exported even if it
      was requested.
      Similarly, LookupFlags of lookups can make a lookup dependent
      on the GDEF items "markAttachClasses" (Flag: MarkAttachmentType)
      and/or "markGlyphSets" (Flag: UseMarkFilteringSet) and some
      Lookup Types can themselves be dependent on other Lookups in
      the same table (GDEF or GPOS).
      NOTE: These Lookup Types are not yet implemented.

-w, --whitelist
      If a whitelist is present only items that are selected
      by the whitelist are allowed for output. This also means that
      this can block other not directly blocked items from output,
      if their dependencies are blocked.
      (default is like: -w "**")

-b, --blacklist
      If a blacklist is present only items that are *NOT* selected
      by the blacklist are allowed for output. his also means that
      this can block other not directly blocked items from output,
      if their dependencies are blocked.

-m, --mute
      While --whitelist and --blacklist try to keep the fea output
      valid when blocking other items (by taking care of dependencies)
      mute is not that careful. Mute allows to select items to not
      being printed, while dependent items will still be outputted.
      This is useful e.g. in a build process if all lookups of a
      feature should be used, but the features itself and thus the
      application of it are predefined somewhere else.
      E.g.: -r "GPOS feature mkmk" -m "GPOS feature mkmk" will output
      only the lookups used by the mkmk feature and if needed some
      class definitions of the GDEF table for Lookup-Flags. But
      the feature definition "feature mkmk { ... } mkmk;" will be
      omitted.
      Another scenario may be the inspection of the dependencies of an item
      while shutting of other noise.

The Selector Specification
==========================

A "selector" is made up of one or more single selectors, separated by
semicolons: "single-selector;single-selector;single-selector;

A "single-selector" is made up of one or more "selector-parts", separated
by whitespace: "part1 part2 part3". These parts represent a parent-child
like hierarchy: part3 is contained in part2 and part2 is contained in
part1.

A "selector-part" is made up of one ore more "selector-tags", separated
by the pipe "|" symbol:  "tag1|tag2|tag3". Each part of a tag represents
an alternative (logical OR) on the hierarchy level of the "selector-part"

Selector-Tags
-------------

"*" and "**" are wildcard tags.

"*" selects all items on the current hierarchy level. Thus it's not
    needed to have further tags separated with "|" in the same selector-part
"**" selects all items on the current hierarchy level and on all hierarchy
     levels below. Thus after a "**" it's not needed to add any further
     parts to select deeper levels. A single-selector ends after the
     first "**"

### Available tags and their hierarchy level (by indentation)

  languagesystem
  GDEF
      glyphClassDef
      attachList
      ligCaretList
      markAttachClasses
      markGlyphSets
  GSUB and GPOS
      script
          DFLT latn arab ... [script tags]
      language
          DEU dflt ARA URD ... [language tags]
      feature
          mkm mark calt dlig liga init medi ... [feature tags]
      lookup
          gpos1 gsub2 ... [see 'Lookup Type Selector Tags' below]// gpos1

 ### Lookup Type Selector Tags:
 from: https://www.microsoft.com/typography/otspec/gpos.htm

     gpos1: Single adjustment (GPOS Lookup-Type 1)
     gpos2: Pair adjustment
     gpos3: Cursive attachment
     gpos4: MarkToBase attachment
     gpos5: MarkToLigature attachment
     gpos6: MarkToMark attachment
     gpos7: Context positioning
     gpos8: Chained Context positioning
     gpos9: Extension positioning
     gsub1: Single
     gsub2: Multiple
     gsub3: Alternate
     gsub4: Ligature
     gsub5: Context
     gsub6: Chaining Context
     gsub7: Extension Substitution
     gsub8: Reverse chaining context single

EXAMPLES
========

$ fea2ft path/to/font.otf > features.fea
    Export all features of font.otf and save them to the file features.fea

$ fea2ft -r "GPOS" path/to/font.otf
    export just the GPOS table and GDEF dependencies if there.

$ ft2fea -r "GPOS feature mkmk|mark" path/to/font.otf
    export the mkmk and mark feature of the GPOS table (if available)

$ ft2fea -r "* script DFLT" path/to/font.otf
    export everything registered under the DFLT script

$ ft2fea -r "languagesystem" path/to/font.otf
    Print all languagesystem definitions for the font.
    NOTE: control over "languagesystem" is very limited at the moment.
    It is is not used as a dependency of script and language, thus all
    languagesystem definitions are either printed or not.

$ ft2fea -r "GSUB feature medi" -m "GDEF; GSUB feature *" path/to/font.otf
    export the lookups used by the medi feature of GSUB

$ ft2fea -r "GPOS lookup gpos4|gpos5|gpos6" path/to/font.otf
    export only the lookups of type MarkToBase (gpos4) or
    MarkToLigature (gpos5) or MarkToMark(gpos6)

$ ft2fea -r "* * gpos4|gpos5|gpos6" path/to/font.otf
    export only the lookups of type MarkToBase (gpos4) or
    MarkToLigature (gpos5) or MarkToMark(gpos6). Note, nothing but
    "GPOS ligature" as children named like "gpos4", thus the wildcards
    do just fine.

$ ft2fea -request "GPOS|GSUB feature *" \
            -whitelist "GPOS|GSUB; GPOS|GSUB **" \
            -mute "* feature *"  \
            path/to/font.otf
    Select features of GPOS or GSUB but filter dependencies that are not
    in GPOS or GSUB, i.e. lookups depending on class definitions in GDEF.
    Mute the feature blocks, so that only the lookups are exported.

twardoch · 2016-01-22T23:17:58Z

Cool!
You have a typo ("mkm " instead of "mkmk " in the usage string). Other than that, splendid stuff!

behdad · 2016-01-23T11:44:22Z

Thanks Lasse!

We definitely have plans to do this. My personal preference is to finish otlLib's build stuff, then add unbuild, and then add this feature to feaLib that would interface with unbuilt stuff rather than the internal tables directly. But that definitely is not a requirement per se.

anthrotype · 2016-01-23T11:53:07Z

Hey, this is great! Thank you for sharing this here, Lasse :)

As @behdad anticipated in #468 (comment), we are indeed planning to add support for writing OTL tables to feature files in various formats (fea, vtp, mti). Behdad proposed to define an API for "unbuilding" otData-defined objects to a set of Python primitive types (tuples, dict, set, etc.), i.e. the reverse of what otlLib.builder does. With that in place, then it would be easier to conceive writers outputting this data in various formats, including .FEA.

Anyway, yours seems to go in the same direction. I look forward to seeing more of it!

brawer · 2016-01-23T12:35:45Z

This is great, Lasse! My only concern is the lack of testing. Without unit tests, the codebase is hard to maintain.

OK to wait a week or so? Then I should have the unbuild() part done.

graphicore · 2016-01-23T15:11:32Z

Thank you all for the feedback.

interface with unbuilt stuff rather than the internal tables directly

The unbuild plans sound good to me. I wouldn't mind to switch.

Without unit tests, the codebase is hard to maintain.

I agree. That can be done though.

OK to wait a week or so? Then I should have the unbuild() part done.

I'm not in a hurry. In fact I would suggest to keep the PR open for a while and let it evolve. Adding tests, using unbuild, that's fine for me. After all you are the maintainers of fontTools, so you should decide what goes into it.
I needed that thing right now, that's why it exists now, And I'll probably add some more lookups and some of the TODO's at the top of ft2feaCLI.py as I need it.

With some more distance and a bit of spare time I think I'd review the implementation of the filtering/query code in the CLI (ExportAggregator), because that's rather at the edge of "keep it simple", it has probably too much implicit knowledge about the behavior of the printing functions of ft2fea. Maybe the structure returned by unbuild can act as a intermediary format, that is then filtered, or a structure representing a fea-file could be created explicitly (containing otLib objects or unbuild data) , traversing and printing that would be trivial and simplify the printing functions, because the getStatus stuff could be removed.

… state

…pType 1).

…r Format 3 (GSUB LookupType 6 Format 3)

…er Lookups)

Original Source: https://github.com/graphicore/fonttools/tree/ft2fea PR in fontTools: fonttools/fonttools#479

anthrotype · 2016-12-20T13:39:04Z

~~This is mostly superseded by #776 and #779.~~
Thanks Lasse anyway!

Dunno, maybe your your console script could be adapted to work with Martin's implementation.
Maybe a main entry point for the ast module?
Let's think about it.

adrientetar · 2016-12-20T13:49:34Z

Can't ft2fea be used for dumping binary fonts feature to an FEA file, which fea2fea (the other PR) can't (only takes FEA as input)?

anthrotype · 2016-12-20T14:14:03Z

Ah true, I confused the two things.. 😐
Sorry!

So yeah, we still need to finish that unbuilding bit.

behdad · 2016-12-20T20:47:48Z

Still would be nice if we can generate AST from binary and use AST writer. That way changing whitespace and other things will be uniformly applied to both kinds of output.

justvanrossum · 2018-02-07T06:58:22Z

This code is "just" a snippet, and while it would be great to have it integrated more solidly into fonttools, it's seems that no movement is being made into that direction. I'd rather have this code actually be in Snippets/ than that it sits to rust here in this PR.

justvanrossum · 2018-02-07T07:06:58Z

...although it currently doesn't run on Python 3, which is a bummer:

Traceback (most recent call last):
  File "ft2feaCLI.py", line 669, in <module>
    main()
  File "ft2feaCLI.py", line 638, in main
    aggregator.validate()
  File "ft2feaCLI.py", line 388, in validate
    self.validateCommonGTable(self.font[tableTag], required)
  File "ft2feaCLI.py", line 92, in wrapper
    registered = self.registry.get(key, None)
TypeError: unhashable type: 'table_G_P_O_S_'

behdad · 2018-02-07T17:36:31Z

Can someone take this on and do the proper implementation please?

behdad · 2022-08-16T20:16:44Z

@graphicore can you bring this up to date so it runs on Python 3 and drop it into Snippets? If not we'll have to close this.

graphicore force-pushed the ft2fea branch from 037b2f8 to 51cbbf6 Compare January 28, 2016 12:34

graphicore added 21 commits February 25, 2016 17:49

[ft2fea] Initial commit, broken state

d840097

[ft2fea] Print GDEF MarkAttachClassDef and MarkGlyphSetsDef

927697a

[ft2fea] Print GDEF GlyphClassDef and LigCaretList

aa52dfd

[ft2fea] Restructuring and cleanup, removed CLI related code, working…

edca422

… state

[ft2fea] Initial commit shell command, rough state

e69cf29

[ft2fea] Minor changes

8000

650731f

[fea2ft] Debugging ExportAggregator

9cffb9d

[ft2fea] Use argparse for the CLI

ea76f26

[ft2fea] Add [-m, --mute] option and some documentation via -h. Cleanup

e8b154e

[ft2fea] Don't try to derive anchor types from anchor instances.

79664db

[ft2fea] Add --suffix argument to control makeName behavior.

cba5a1d

[ft2fea] Allow empty Lookupflags to happen.

b0630e7

[ft2fea] Make function formatGlyphClassDef from inline code.

781f73c

[ft2fea] Add formatLookupSingleSubstitution (GSUB LookupType 1).

5faa2e6

[ft2fea] Fix a typo in -h help output (thanks Adam Twardoch)

6f5be60

[ft2fea] Add formatValueRecord and formatSingleAdjustment (GPOS Looku…

23467da

…pType 1).

[ft2fea] Add formatLookupMutipleSubstitution (GSUB LookupType 2).

240748f

[ft2fea] Add formatLookupAlternateSubstitution (GSUB LookupType 3).

2ba34c1

[ft2fea] Add formatLookupLigatureSubstitution (GSUB LookupType 4).

61c73a8

[ft2fea] Add incomplete formatLookupChainingContextualSubstitution fo…

b82ab8e

…r Format 3 (GSUB LookupType 6 Format 3)

[ft2fea] Output Lookup in topological sort order (dependencies to oth…

ab061c4

…er Lookups)

graphicore force-pushed the ft2fea branch from 51cbbf6 to d0016d3 Compare February 25, 2016 16:50

[ft2fea] Support NULL Anchors

de57f56

graphicore added a commit to Tarobish/Mirza that referenced this pull request Feb 25, 2016

Including a fork of ft2fea until it has a home for itself.

9c39622

Original Source: https://github.com/graphicore/fonttools/tree/ft2fea PR in fontTools: fonttools/fonttools#479

graphicore added a commit to Tarobish/Katibeh that referenced this pull request Feb 25, 2016

Include and use a fork of ft2fea until it has a home for itself.

94e5e4b

Original Source: https://github.com/graphicore/fonttools/tree/ft2fea PR in fontTools: fonttools/fonttools#479

adrientetar mentioned this pull request Dec 19, 2016

[feaLib] Generate Adobe feature files from feaLib AST #772

Closed

mhosken mentioned this pull request Dec 20, 2016

Add fea generation support #776

Merged

miguelsousa mentioned this pull request Feb 6, 2018

ttx: option to dump GPOS, GSUB with feature file syntax? #1129

Open

twardoch mentioned this pull request Apr 24, 2019

fontTools.fx — a simple aggregate subpackage to guide me through various functionalities #1589

Open

Base automatically changed from master to main February 17, 2021 11:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ft2fea-feature file dumper #479

ft2fea-feature file dumper #479

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ft2fea-feature file dumper #479

Are you sure you want to change the base?

ft2fea-feature file dumper #479

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!