Easy and efficient Python interop for Elixir.
- 🛡️ Robust & Isolated: Run multiple Python interpreters in separate OS processes, preventing GIL issues from affecting your Elixir application.
- 📦 Declarative Environments:
Leverages
uvto manage Python versions and dependencies, embedding them into your application's release for consistent deployments. - ✨ Ergonomic 8000 Interface: A powerful and efficient interface with explicit control over data passing between Elixir and Python processes.
- 🤸 Flexible:
Supports custom Python environments,
asynciocode, and integration with external Python projects. - ⏩ Forward Compatibility: Built on stable foundations, so future versions of Python or Elixir are unlikely to require Snex updates to use — they should work day one!
defmodule SnexTest.NumpyInterpreter do
use Snex.Interpreter,
pyproject_toml: """
[project]
name = "my-numpy-project"
version = "0.0.0"
requires-python = "==3.11.*"
dependencies = ["numpy>=2"]
"""
end
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, """
...> import numpy as np
...> matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (s, s), dtype=int)
...> """, %{"s" => 6}, returning: "np.linalg.norm(matrix)")
{:ok, 6.0}- Elixir
>= 1.18 - uv - a fast Python package & project manager, used by Snex to create and manage Python environments. It has to be available at compilation time but isn't needed at runtime.
- Python
>= 3.11- this is the minimum supported version you can run your scripts with. You don't need to have it installed — Snex will fetch it withuv.
def deps do
[
{:snex, "~> 0.1.0"}
]
endYou can define your Python project settings using use Snex.Interpreter in your module.
Set a required Python version and any dependencies —both the Python binary & the dependencies will be fetched & synced at compile time with uv, and put into your application's priv directory.
defmodule SnexTest.NumpyInterpreter do
use Snex.Interpreter,
pyproject_toml: """
[project]
name = "my-numpy-project"
version = "0.0.0"
requires-python = "==3.11.*"
dependencies = ["numpy>=2"]
"""
endThe modules using Snex.Interpreter have to be start_linked to use.
Each Snex.Interpreter (BEAM) process manages a separate Python (OS) process.
The main way of interacting with the interpreter process is Snex.pyeval/4 (and other arities).
This is the function that runs Python code, returns data from the interpreter, and more.
iex> {:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, %Snex.Env{} = env} = Snex.make_env(interpreter)
...>
iex> Snex.pyeval(env, """
...> import numpy as np
...> matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (6, 6), dtype=int)
...> scalar = np.linalg.norm(matrix)
...> """, returning: "scalar")
{:ok, 6.0}The :returning option can take any valid Python expression, or an Elixir list of them:
iex> {:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(interpreter)
...>
iex> Snex.pyeval(env, "x = 3", returning: ["x", "x*2", "x**2"])
{:ok, [3, 6, 9]}Snex.Env struct, also called "environment", is an Elixir-side reference to Python-side variable context in which your Python code will run.
New environments can be allocated with Snex.make_env/3 (and other arities).
Environments are mutable, and will be modified by your Python code. In Python parlance, they are global & local symbol table your Python code is executed with.
Important
Environments are garbage collected
When a %Snex.Env{} value is cleaned up by the BEAM VM, the Python process is signalled to deallocate the environment associated with that value.
Reusing a single environment, you can use variables defined in the previous Snex.pyeval/4 calls:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
...> # `pyeval` does not return a value if not given a `returning` opt
iex> :ok = Snex.pyeval(env, "x = 10")
...>
...> # additional data can be provided for `pyeval` to put in the environment
...> # before running the code
iex> :ok = Snex.pyeval(env, "y = x * z", %{"z" => 2})
...>
...> # `pyeval` can also be called with `:returning` opt alone
iex> Snex.pyeval(env, returning: ["x", "y", "z"])
{:ok, [10, 20, 2]}Using Snex.make_env/2 and Snex.make_env/3, you can also create a new environment:
- copying variables from an old environment
Snex.make_env(interpreter, from: old_env)
- copying variables from multiple environments (later override previous)
Snex.make_env(interpreter, from: [ oldest_env, {older_env, only: ["pool"]}, {old_env, except: ["pool"]} ]))
- setting some initial variables (taking precedence over variables from
:from)Snex.make_env(interpreter, %{"hello" => 42.0}, from: {old_env, only: ["world"]})
Warning
The environments you copy from have to belong to the same interpreter!
User data sent between Python and Elixir is subject to "standard" JSON serialization and deserialization using JSON on the Elixir side and json on the Python side.
Among other things, this means that Python tuples will be serialized as arrays, while Elixir atoms and binaries will be serialized as strings.
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, "x = ('hello', y)", %{"y" => :world}, returning: "x")
{:ok, ["hello", "world"]}Code ran by Snex lives in an asyncio loop.
You can include async functions in your snippets and await them on the top level:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, """
...> import asyncio
...> async def do_thing():
...> await asyncio.sleep(0.01)
...> return "hello"
...>
...> result = await do_thing()
...> """, returning: ["result"])
{:ok, "hello"}A good way to run any blocking code is to prepare and use your own thread or process pools:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, pool_env} = Snex.make_env(inp)
...>
iex> :ok = Snex.pyeval(pool_env, """
...> import asyncio
...> from concurrent.futures import ThreadPoolExecutor
...>
...> pool = ThreadPoolExecutor(max_workers=cnt)
...> loop = asyncio.get_running_loop()
...> """, %{"cnt" => 5})
...>
...> # You can keep the pool environment around and copy it into new ones
iex> {:ok, env} = Snex.make_env(inp, from: {pool_env, only: ["pool", "loop"]})
...>
iex> {:ok, "world!"} = Snex.pyeval(env, """
...> def blocking_io():
...> return "world!"
...>
...> res = await loop.run_in_executor(pool, blocking_io)
...> """, returning: "res")
{:ok, "world!"}You can reference your existing project path in use Snex.Interpreter.
The existing pyproject.toml and uv.lock will be used to seed the Python environment.
defmodule SnexTest.MyProject do
use Snex.Interpreter,
project_path: "test/my_python_proj"
end
# $ cat test/my_python_proj/foo.py
# def bar():
# return "hi from bar"
# Provide the project's path at runtime - you'll likely want to use
# :code.priv_dir(:your_otp_app) and construct a path relative to that.
iex> {:ok, inp} = SnexTest.MyProject.start_link(environment: %{
...> "PYTHONPATH" => "test/my_python_proj"
...> })
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, "import foo", returning: "foo.bar()")
{:ok, "hi from bar"}