Release 0.0.2 -- inference server & other improvements! #108

francoishernandez · 2024-09-19T13:08:02Z

francoishernandez
Sep 19, 2024
Maintainer

We just released 0.0.2, which has been quite an iteration since 0.0.1.
Some long awaited changes are finally making it to the main branch!

Llama3.1 Support: We've boosted support for Llama3.1, including RoPE scaling and the introduction of mapped_tokens to efficiently handle specific prompt tokens.
Inference Server: The first version of our inference server is live! This sets the foundation for more features and facilitated iteration for upcoming use cases. Check out the example recipe!
Improved Configuration Management: Tokenization and inference-related settings are now embedded within the model configuration itself, making predictions and serving more seamless.

bfloat16 Support: Unlock better performance on specialized hardware with our new bfloat16 support, balancing memory use and precision.
WMT22 Recipes: New recipes featuring TowerInstruct and Llama3.1 LLMs to simplify experimentation and setup.