Oumi v0.1.11 Release Notes 🚀

@wizeng23

What's Changed

Add vllm to gpu optional dependencies by @wizeng23 in #1614
[HallOumi] Update inference notebook by @wizeng23 in #1613
Update llama4 GCP jobs for non-dev environments. by @taenin in #1621
Update transformers to 4.51.0 by @wizeng23 in #1620
Lazy load skypilot by @taenin in #1622
Add additional_model_kwargs and additional_trainer_kwargs to train function by @hommayushi3 in #1624
Added 3 Pixmo vision-language datasets by @jrwana in #1523
[GRPO] Add notebook to demonstrate GRPO & evaluation for letter counting by @wizeng23 in #1625
[Remote Inference] Update Default Params by @kaisopos in #1630
Update trl to 0.16 by @wizeng23 in #1631
Support custom processor args in ModelParams by @nikg4 in #1634
Support BerryBench evaluation by @wizeng23 in #1635
[Remote Inference] Error checking for api_key by @kaisopos in #1638
Rename cnn_mnist_example to cnn_mnist_tutorial by @wizeng23 in #1640
[Remote Inference][GCP] Constructing api_url from the Project ID and Region by @kaisopos in #1636

New Contributors

@jrwana made their first contribution in #1523

Full Changelog: v0.1.11...v0.1.12

@hommayushi3

Oumi v0.1.11 Release Notes 🚀

Key Highlights

Model Support 🤖

Integrated Llama 4 (Scout and Maverick variants) with complete workflow configs 🦙
Added LoRA training for Phi3, Phi4, and Qwen2.5-VL multimodal models 🖼️

Developer Experience 💻

Introduced MLflow integration for experiment tracking 📝
Enhanced CLI with convenient alias functionality ⌨️

HallOumi Framework 🧠

Added examples for Halloumi
Added dedicated inference notebooks for both generative and classifier approaches 📓

Welcome to our new contributors @hommayushi3 and @gabrielaugz! 👋

For details, see the [full changelog](v0.1.10...v0.1.11).

@nikg4

What's Changed

Increment pillow version for compatibility with Python 3.13 by @nikg4 in #1566
[Evaluation] Bug: Multiple GPUs attempt to save in the same folder by @kaisopos in #1567

Full Changelog: v0.1.9...v0.1.10

@wizeng23

What's Changed

Add QwQ full fine-tune and QLoRA configs by @wizeng23 in #1518
Update TRL to 0.15 and fix Liger/dataset code by @wizeng23 in #1507
[tiny] Remove vLLM Colab link and fix Alpaca Eval quickstart by @wizeng23 in #1530
Evaluation: Inference optimizations by @kaisopos in #1522
Qwen2.5 VL: Replace "from source" install with transformers>=0.49 by @nikg4 in #1528
[Evaluation] Renaming evaluation_platform → evaluation_backend by @kaisopos in #1526
[tiny] Clean up datasets code by @wizeng23 in #1529
Minor logging improvements in BaseMapDataset by @nikg4 in #1532
Upload scripts used in a Weekly Walkthrough by @taenin in #1533
Update VisionLanguageConversationFeatureGenerator by @nikg4 in #1531
[docs] add security.md by @oelachqar in #1534
[Evaluation] Custom evaluation notebook: a reliability classifier by @kaisopos in #1535
Multimodal: Limit max number of images per Conversation by @nikg4 in #1536
Auto-populate and validate params specific to vision_language_sft collator in TrainingConfig by @nikg4 in #1537
Update Oumi Env to use Rich formatting by @taenin in #1541
Update oumi launch to use Rich formatting by @taenin in #1543
Update oumi evaluate to use rich formatting. by @taenin in #1544
Update the CLI to replace all prints with Rich prints. by @taenin in #1547
Render the oumi env command as a shell block in bug reports. by @taenin in #1548
Define Conversation proto bufs by @nikg4 in #1550
[Evaluation] Modifying Alpaca Eval results format to be consistent with LM Harness by @kaisopos in #1551
Augmenting logging training/model statistics by @optas in #1545
Misc no-op code cleanups by @nikg4 in #1553
Add code used for the evaluation demo. by @taenin in #1556
Add OUMI_FORCE_EDITABLE_INSTALL env var to do editable Oumi install from source in job configs by @wizeng23 in #1420
Add letter counting GRPO example by @wizeng23 in #1539
Remove UV install from notebooks as this breaks colab by @taenin in #1558
[Evaluation] Updates in hallucination notebook by @kaisopos in #1552
[Evaluations] Custom evals: Adding support for eval_kwargs by @kaisopos in #1557
Logging message update in log_number_of_model_parameters by @nikg4 in #1560
[Evaluation][Custom] Removing restrictions and better error checking by @kaisopos in #1561
Support text truncation (max_length) for vision_language_sft collator by @nikg4 in #1559
phi 4 multimodal training version 1 ( with limitations ) by @optas in #1555
Phi-4 basic inference with native/vllm by @optas in #1563
[minor] phi4 train improvements by @optas in #1564
Fix printing errors in oumi env for non-string values. by @taenin in #1565

Full Changelog: v0.1.8...v0.1.9

@nikg4

What's Changed

GRPO trainer: Minimal initial integration by @nikg4 in #1482
Update oumi infer to fall back to interactive mode if no input path is specified. by @taenin in #1483
Add sample DDP/GCP config for GRPO trainer by @nikg4 in #1485
Temporary fix for chat template issue with multimodal inference w/ in-process vLLM engine by @nikg4 in #1486
[tiny] Update async_eval.yaml comments to reference correct class by @wizeng23 in #1488
Fix a bug where overriding remote_params fails via the CLI (oumi infer) by @taenin in #1487
Define GrpoParams under configs by @nikg4 in #1490
Support more GRPO params by @nikg4 in #1491
Minor updates to oumi env by @nikg4 in #1492
Warn instead of error when device not found for MFU calculation by @wizeng23 in #1489
Updated all CLI endpoints to support oumi:// prefix by @Spaarsh in #1468
Fix chat template issue for nested content parts used for VLMs by @nikg4 in #1493
Ctseng777/judge by @ctseng777 in #1474
[Evaluation] Modularization & enabling custom evaluations by @kaisopos in #1484
Update documentation formatting for BaseModel by @taenin in #1494
Fix log_samples not propagating from eval_kwargs by @jgreer013 in #1496
[Evaluation] Adding support for logging model samples for all backends by @kaisopos in #1499
Support for deprecated input param ( evaluation_platform) by @kaisopos in #1500
Limiting the AlpacaEval number of samples for quickstart by @kaisopos in #1501
Add recurring tests to keep our test badges updated. by @taenin in #1498
Add a schedule for our GPU, CPU, and doc tests by @taenin in #1503
Update the GPU Tests badge to use results from main by @taenin in #1504
vLLM version increment by @nikg4 in #1502
Minor logging improvements by @nikg4 in #1505
[Evaluation] Save Utils: Moving, fixes, and unit tests by @kaisopos in #1506
Update sample GRPO script to validate num_generations by @nikg4 in #1509
Resolve warning about --dispatch batches deprecated param by @nikg4 in #1510
[Evaluation] Re-enabling evaluations with Math Hard (leaderboard_math_hard) by @kaisopos in #1511
Update docker image and build script by @oelachqar in #1508
Add Qwen QwQ Lora config by @wizeng23 in #1514
Add QwQ eval/infer configs by @wizeng23 in #1515
[Evaluation] Instantiating an inference engine (if needed) when running custom evaluations by @kaisopos in #1513
Switch eval yaml configs to use evaluation_platform by @wizeng23 in #1516
Mark BaseMapDataset as typing.Sized by @nikg4 in #1517
VLM collator refactor by @nikg4 in #1512

Full Changelog: v0.1.7...v0.1.8

@taenin

What's Changed

Update the RemoteInferenceEngine to appropriately handle openai format batch prediction endpoints. by @taenin in #1472
Fix local models to not break the registry. by @taenin in #1476
Create an inference config for Claude Sonnet 3.7 by @taenin in #1479
Add notebook for fine-tuning MiniMath-R1-1.5B by @jgreer013 in #1480
[Evaluation] Migrate LM Harness integration point from simple_evaluate to evaluate by @kaisopos in #1455
[tiny]Update trl to 0.14 by @wizeng23 in #1478

Full Changelog: v0.1.6...v0.1.7

@taenin

What's Changed

Update RemoteParams to no longer require an API URL. by @taenin in #1452
[Tiny] Update default training params for Qwen2-VL-2B-Instruct by @optas in #1454
[Tiny] Add more warnings for "special" requirements of Qwen2.5-VL by @optas in #1453
Minor cleanup of oumi fetch by @taenin in #1463
Support for multi-image VLM training by @nikg4 in #1448
Remove a temp workaround in pad_sequences on the left side by @nikg4 in #1464
[tiny] Add warning that Oumi doesn't support Intel Macs by @wizeng23 in #1467
VLM-related logging improvements by @nikg4 in #1469
Fix Oumi launcher to be able to run on RunPod and Lambda by @wizeng23 in #1470
Enable pre-release install for uv in pyproject.toml by @wizeng23 in #1466

Full Changelog: v0.1.5...v0.1.6

@wizeng23

What's Changed

Fix the remainder of our configs by @wizeng23 in #1356
Adopt new Llama 3.1 HF names by @wizeng23 in #1357
Define OUMI_USE_SPOT_VM env var and start using it to override use_spot param by @xrdaukar in #1359
Support HuggingFaceM4/Docmatix dataset by @vishwamartur in #1342
[nit] update default issue names by @oelachqar in #1367
Update sft_datasets.md by @penfever in #1349
Have GitHub Trending image hyperlink to GitHub Trending page by @wizeng23 in #1370
Update the link for the trending banner. by @taenin in #1371
Move code to disable caching in model.config to a helper function by @xrdaukar in #1378
Update transformers version to 4.48 by @wizeng23 in #1372
Update notebooks to improve their Colab experience by @wizeng23 in #1380
Add proper labels and types to new Bugs and Feature Requests. by @taenin in #1383
Upgrade omegaconf to 2.4.0dev3 by @wizeng23 in #1384
Support HuggingFaceM4/the_cauldron dataset by @vishwamartur in #1366
Update our FAQ for tips about installing oumi on Windows by @taenin in #1385
Cleanup HuggingFaceM4/Docmatix and HuggingFaceM4/the_cauldron multimodal datasets by @xrdaukar in #1387
Remove uneeded env vars from job configs by @wizeng23 in #1390
Remove transformer version override for HuggingFaceTB/SmolVLM-Instruct in launcher script by @xrdaukar in #1388
[Small Refactor] Moving the inference engine def outside the inference config by @kaisopos in #1395
Evaluation - LM Harness: Adding vLLM support by @kaisopos in #1379
Remove Docmatix dataset references from docstrings VLM config examples by @xrdaukar in #1397
Fixed broken link in Oumi - A Tour.ipynb notebook by @ciaralema in #1398
Fix broken links in notebooks. by @taenin in #1402
Create a client for communicating with a Slurm node via SSH. by @taenin in #1389
[tiny] Remove references to missing job configs in README by @wizeng23 in #1404
Train+Inference with Qwen 2.5 VL (3B) by @optas in #1396
Add a Slurm cluster and cloud to the oumi launcher. by @taenin in #1406
Move pretokenize script from scripts/pretokenize/ to scripts/datasets/pretokenize/ by @xrdaukar in #1412
Create a script to save Conversation-s from SFT datasets into .jsonl file by @xrdaukar in #1413
[Evaluation] LM Harness refactor by @kaisopos in #1410
Update save_conversations tool by @xrdaukar in #1421
[SambaNova] Integrate SambaNova Systems to oumi inference by @ctseng777 in #1415
[Μinor] Equating Qwen's 2.5 chat-template to version's 2.0 by @optas in #1419
Add requirements header to configs and clean them up by @wizeng23 in #1411
Updated oumi infer to support CLI argument for system prompt by @Spaarsh in #1422
[Evaluation] LM Harness remote server support by @kaisopos in #1414
[Feature] Add Tulu3 SFT Mixture Dataset Support by @bwalshe in #1381
Support Multimodal inference with multiple images and PDF-s in NATIVE engine by @xrdaukar in #1424
Update notebooks to run on Colab by @wizeng23 in #1423
Add calm recipe. by @taenin in #1425
Update VLM sample oumi infer -i commands by @xrdaukar in #1428
Provide example show to start SGLang server using Docker by @xrdaukar in #1429
Multi-image support in SGLang inference engine by @xrdaukar in #1426
Calm readme by @emrecanacikgoz in #1432
WildChat-50M Reproduction by @penfever in #1433
Add WildChat support by @penfever in #1348
Create pad_to_max_dim_and_stack() function in torch_utils by @xrdaukar in #1435
use deterministic by @penfever in #1434
Additional HF trainer parameters for config by @penfever in #1436
Set a better default for vllm inference GPU usage. by @taenin in #1437
Added fetch command and modified infer command to resolve oumi:// by @Spaarsh in #1439
Require an inference config for oumi infer. by @taenin in #1443
Make the tulu3 unit tests hermetic. by @taenin in #1446
Add 2 more sample PDF-s with 1 and 2 pages under testdata/pdfs by @xrdaukar in #1427
Enable ability to override list values in config via CLI by @wizeng23 in #1430
Renamed CALM to CoALM by @jgreer013 in #1450
Add support for Docmatix dataset to multimodal training script by @xrdaukar in #1449
Update oumi launch status to show clusters with no running jobs. by @taenin in #1451

New Contributors

@vishwamartur made their first contribution in #1342
@ctseng777 made their first contribution in #1415
@Spaarsh made their first contribution in #1422
@bwalshe made their first contribution in #1381
@emrecanacikgoz made their first contribution in #1432

Full Changelog: v0.1.4...v0.1.5

@taenin

What's Changed

Add memory cleanup calls in e2e integration tests by @xrdaukar in #1277
Set up versioning for our documentation by @taenin in #1275
Make qwen2-VL evaluation job pass by @xrdaukar in #1278
Add multi-modal (vlm) notebook with Llama 11B by @optas in #1258
Documentation: Inference -> List supported models by @kaisopos in #1279
[tiny] update website link by @oelachqar in #1280
Update all documentation links to the new doc URL by @taenin in #1281
Update Oumi - A Tour.ipynb by @brragorn in #1282
Documentation: Judge (minor edits) by @kaisopos in #1283
Fix citation by @oelachqar in #1285
Add Deepseek R1 1.5B/32B configs by @wizeng23 in #1276
Misc eval configs cleanup by @xrdaukar in #1286
[docs] Describe parallel evaluation by @xrdaukar in #1284
Update microsoft/Phi-3-vision-128k-instruct training config by @xrdaukar in #1287
Add Together Deepseek R1 inference config by @wizeng23 in #1289
[minor] vlm notebook minor updates (doc referencing, freeze visual backbone) by @optas in #1288
Add missing -m oumi evaluate argument in eval config by @xrdaukar in #1291
[docs] Add more references to VL-SFT and SFT notebooks by @xrdaukar in #1293
Eval config change for deepseek-ai/DeepSeek-R1-Distill-Llama-70B by @xrdaukar in #1292
[notebooks] Update intro & installation instruction by @oelachqar in #1294
Update notebook intros by @oelachqar in #1296
[notebooks] Update installation instructions for colab by @oelachqar in #1297
Add Apache license header to src/oumi/**/*.py by @wizeng23 in #1290
Minor updates to VLM Multimodal notebook by @xrdaukar in #1299
[docs] Add latest notebooks and update references by @oelachqar in #1300
[tiny] Add docs auto-generated .rst files to gitignore by @wizeng23 in #1298
[tiny] use GitHub link for header by @oelachqar in #1301
[docs][tiny] update inference engines reference by @oelachqar in #1302
Update README/docs to add new DeepSeek models by @wizeng23 in #1304
[docs] Use pip install oumi over pip install . by @wizeng23 in #1305
Tune VLM SFT configs by @xrdaukar in #1306
Tune VLM configs for SmolVLM and Qwen2-VL by @xrdaukar in #1307
Update config/notebook pip installs to use PyPI by @wizeng23 in #1308
[tiny] upgrade torch version by @oelachqar in #1295
Update logging and unit tests related to chat templates by @xrdaukar in #1311
fix(docs): "interested by joining" to "interested in joining" by @CharlesCNorton in #1312
Add HF_TOKEN instructions to Oumi Multimodal notebook by @xrdaukar in #1313
Update configuration.md by @penfever in #1314
remove duplicate keys in config example by @lucyknada in #1315
[Notebooks] Update VLM notebook by @xrdaukar in #1317
Update parasail_inference_engine.py by @jgreer013 in #1320
Fix typo and update warning message for OUMI trainer by @xrdaukar in #1319
[Notebooks] Add a note that a notebook kernel restart may be needed after pip install oumi by @xrdaukar in #1318
Update Phi3 to support multiple images by @xrdaukar in #1321
Add more detailed comment headers to YAML configs by @wizeng23 in #1310
[Notebooks] Add a note to Tour notebook to restart kernel after the first pip install by @xrdaukar in #1327
Tweak --mem-fraction-static param in sample SGLang configs by @xrdaukar in #1328
Disallow using DatasetParams field names as keys in DatasetParams.dataset_kwargs by @xrdaukar in #1324
Support dataset_name_override dataset_kwarg by @xrdaukar in #1188
Add an util and a test marker for HF token by @xrdaukar in #1329
Update llama3-instruct chat template to align with the original models template by @xrdaukar in #1326
chore: update launcher.sh by @eltociear in #1333
[Notebooks] Minor improvements in VLM and CNN notebooks by @xrdaukar in #1335
Update VLM cluster names in sample commands by @xrdaukar in #1336
Update our README and docs with the github trending badge. by @taenin in #1340
Update README.md - Add DeepSeek to supported models by @mkoukoumidis in #1343
Update index.md - Add DeepSeek to supported models by @mkoukoumidis in #1344
Update "GPU Tests" status badge in README page by @xrdaukar in #1345

New Contributors

@CharlesCNorton made their first contribution in #1312
@lucyknada made their first contribution in #1315
@eltociear made their first contribution in #1333

Full Changelog: v0.1.3...v0.1.4

@kaisopos

What's Changed

Documentation: Judge | Custom Model page by @kaisopos in #1195
[WIP] Add a notebook for using CNN with custom dataset by @xrdaukar in #1196
[Cherrypick for launch] Evaluate: return dict of results by @kaisopos in #1197
Configs Train/Infer/Eval and Llama 3.3v (70b) by @optas in #1200
Adding an integration test for evaluation fn's output (see PR-1197) by @kaisopos in #1199
[docs] Add more details and cross-references related to customization by @xrdaukar in #1198
Define single_gpu test marker by @xrdaukar in #1201
Native inference: Don't set min_p, temperature in GenerationConfig if sampling is disabled by @xrdaukar in #1202
Update tests to make them runnable on GCP by @xrdaukar in #1203
Add newline before pformat(train_config) by @xrdaukar in #1204
GCP tests launcher script changes by @xrdaukar in #1205
[Evaluation] Bug: serialization by @kaisopos in #1207
[docs] Add inference snippet for together.ai and DeepSeek APIs by @oelachqar in #1208
Exclude multi_gpu tests from GitHub GPU tests by @xrdaukar in #1210
Update e2e tests to support multi-GPU machines by @xrdaukar in #1206
Add wrappers for remote inference engines by @oelachqar in #1209
Vision-Lang & Inference (including LoRA) by @optas in #1174
[BugFix] Throw a runtime error for quantized models & inference=VLLM by @kaisopos in #1212
Fix most job configs by @wizeng23 in #1213
e2e tests update by @xrdaukar in #1216
[Notebook] Evaluation with Oumi by @kaisopos in #1218
gpt2: move include_performance_metrics param from script to yaml by @xrdaukar in #1217
Simplify inference engine API by @oelachqar in #1214
Move configs to experimental by @wizeng23 in #1215
[docs] Update index page by @oelachqar in #1220
Update ConsoleLogger to write to STDOUT by @xrdaukar in #1221
Set use_spot to False in our JobConfigs by @wizeng23 in #1222
Delete oumi[optional] install target by @wizeng23 in #1224
Scaffolding and the first testcase for e2e evaluation tests by @xrdaukar in #1225
[docs] Update inference engines doc page by @oelachqar in #1227
Clean-up inference engine builder by @oelachqar in #1226
[VLLM Engine] Enabling BitsAndBytes quantization by @kaisopos in #1223
Add example distillation notebook by @jgreer013 in #1228
Add a script to pre-download models for gpu_tests by @xrdaukar in #1231
Fix multi-GPU inference integration test by @xrdaukar in #1229
[tiny][docs] Update PEFT/LoRA content by @optas in #1233
[BugFix] GGUF does not work with VLLM by @kaisopos in #1232
Re-enable parallel evaluation for VLM-s by @xrdaukar in #1235
Add multimodal exemplar dataset in our provided mini-datasets by @optas in #1234
[Tiny] renaming a field name (init_lora_weights) by @optas in #1236
Add more e2e evaluation tests by @xrdaukar in #1237
Fix pyright breakage when vllm and llama_cpp are not installed by @taenin in #1240
Update our oumi launch documentation. by @taenin in #1239
Update index.md title for "Join the Community!" by @mkoukoumidis in #1242
Update quickstart.md - nit for Oumi support request by @mkoukoumidis in #1241
[VLLM Engine] Improve support for GGUF models (incl. auto-download) by @kaisopos in #1238
Update README.md title to "Join the Community!" by @mkoukoumidis in #1243
Update quickstart.md by @brragorn in #1251
Update quickstart.md by @brragorn in #1253
Update quickstart.md by @brragorn in #1252
Update quickstart.md by @brragorn in #1250
[Minor refactor] Moving model caching to oumi.utils by @kaisopos in #1246
Add more details to troubleshooting FAQ by @wizeng23 in #1249
Update training_methods.md - Change compute requirement suggestions by @mkoukoumidis in #1245
Update train.md - nit description change by @mkoukoumidis in #1244
[docs] misc docs feedback by @oelachqar in #1248
[tiny] Qwen2-VL activate experimental datapipes by @optas in #1247
Update Oumi - A Tour.ipynb by @brragorn in #1254
[docs] more docs feedback by @oelachqar in #1255
Update supported_models.md by @penfever in #1256
Rename experimental_use_torch_datapipes data param by @xrdaukar in #1257
Add pypi release workflow using testpypi by @oelachqar in #1259
Update workflow names by @oelachqar in #1262
Update default idle_minutes_to_autostop to 1 hour. by @taenin in #1264
update pypi release workflow to use trusted env by @oelachqar in #1265
Add padding_side param to internal model config by @xrdaukar in #1260
Documentation: Updates on Evaluation/Judge (based on Manos' feedback) by @kaisopos in #1261
[tiny] less strict requirements by @oelachqar in #1266
Add Deepseek R1 Distill Llama 8B/70B configs by @wizeng23 in #1263
Update index.md to highlight beta stage by @mkoukoumidis in #1268
Update README.md to highlight beta stage by @mkoukoumidis in #1267
Disable pre-release packages by @oelachqar in #1270
Update common_workflows.md - Clarify OpenAI is just an example by @mkoukoumidis in #1271
Documentation: Evaluation page (update to highlight multi-modal) by @kaisopos in #1269
Update launch.md by @taenin in #1272
Add pypi release workflow by @oelachqar in #1273
Documentation: Judge | minor edit (bold) by @kaisopos in #1274

Full Changelog: v0.1.2...v0.1.3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

What's Changed

New Contributors

Contributors

Oumi v0.1.11 Release Notes 🚀

Key Highlights

Model Support 🤖

Developer Experience 💻

HallOumi Framework 🧠

Contributors

What's Changed

Contributors

What's Changed

Contributors

What's Changed

Contributors

What's Changed

Contributors

What's Changed

Contributors

What's Changed

New Contributors

Contributors

What's Changed

New Contributors

Contributors

What's Changed

Contributors

Releases: oumi-ai/oumi

v0.1.12

What's Changed

New Contributors

Contributors

v0.1.11

Oumi v0.1.11 Release Notes 🚀

Key Highlights

Model Support 🤖

Developer Experience 💻

HallOumi Framework 🧠

Contributors

v0.1.10

What's Changed

Contributors

0.1.9

What's Changed

Contributors

v0.1.8

What's Changed

Contributors

v0.1.7

What's Changed

Contributors

v0.1.6

What's Changed

Contributors

v0.1.5

What's Changed

New Contributors

Contributors

v0.1.4

What's Changed

New Contributors

Contributors

v0.1.3

What's Changed

Contributors