Experimental Scala 3 bindings for llama.cpp using Slinc.
Add Slinc runtime and llm4s to your build.sbt:
libraryDependencies += "fr.hammons" %% "slinc-runtime" % "0.5.0"
libraryDependencies += "com.donderom" %% "llm4s" % "0.5.1"For JDK 17 add .jvmopts file in the project root:
--add-modules=jdk.incubator.foreign
--enable-native-access=ALL-UNNAMED
Version compatibility:
| llm4s | Slinc runtime | Scala | JDK | llama.cpp (commit hash) |
|---|---|---|---|---|
| 0.4+ | --- | --- | --- | 70d26ac3883009946e737525506fa88f52727132 (Jul 23) |
| 0.3+ | --- | --- | --- | a6803cab946c817fb7aaf2a40b317f5d3e373bd1 (Jul 14) |
| 0.1+ | 0.5.0 | 3.3.0-RC3 | 17, 19 | 447ccbe8c39332fcdd0d98a041b6e2ff6f06219d (Jun 25) |
import java.nio.file.Paths
import com.donderom.llm4s.*
val lib = Paths.get("path/to/libllama.so")
val model = Paths.get("path/to/ggml-model.bin")
val contextParams = ContextParams(threads = 6, seed = 1337)
val llm = Llm(lib = lib, model = model, params = contextParams)
val prompt = "Deep learning is "
val params = LlmParams(context = contextParams)
// To print generation as it goes
llm(prompt, params).foreach: stream =>
stream.foreach: token =>
print(token)
// Or build a string
llm(prompt, params).foreach(stream => println(stream.mkString))
llm.close()