Here's a working config for nVidia Nemotron Mini 4B Instruct #4234

superjamie · 2024-11-24T00:02:00Z

Nemotron is a model by nVidia with a template format I had not seen before:

Here is a tested working config file:

context_size: 4096
f16: true
mmap: true
name: Nemotron-Mini-4B-Instruct
parameters:
  model: bartowski/Nemotron-Mini-4B-Instruct-GGUF/Nemotron-Mini-4B-Instruct-Q8_0.gguf
stopwords:
- </s>
template:
  chat: |
    <extra_id_1>User
    {{.Input}}
    <extra_id_1>Assistant
    
  chat_message: |
    {{if eq .RoleName "assistant"}}<extra_id_1>Assistant{{else if eq .RoleName "system"}}<extra_id_0>System{{else if eq .RoleName "user"}}<extra_id_1>User{{end}}
    {{.Content}}
  completion: |
    {{.Input}}

I did not do Tool usage because I do not use it.

superjamie added the enhancement New feature or request label Nov 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Here's a working config for nVidia Nemotron Mini 4B Instruct #4234

Here's a working config for nVidia Nemotron Mini 4B Instruct #4234

Here's a working config for nVidia Nemotron Mini 4B Instruct #4234

Here's a working config for nVidia Nemotron Mini 4B Instruct #4234

Comments