Grab the latest release of nn~ ! Be sure to download the correct version for your installation.
Uncompress the .tar.gz file in the Package folder of your Max installation, i.e. in Documents/Max 8/Packages/.
You can then instantiate an nn~ object. Depending on your installation, you might get a quarantine warning, such as this one
In most cases, proceeding with the removal will disable this warning until the next update of nn~. If MacOS continues to block the external you can try to codesign locally and remove the external from quarantine:
cd "~/Max X/Packages/nn_tilde
sudo codesign --deep --force --sign - support/*.dylib
sudo codesign --deep --force --sign - externals/*/Contents/MacOS/*
xattr -r -d com.apple.quarantine externals/*/Contents/MacOS/* Right click on the nn~ object to open the help patch, and follow the tabs to learn more about this project.
Uncompress the .tar.gz file in the Package folder of your Pd installation, i.e. in Documents/Pd/externals/. You can then add a new path in the Pd/File/Preferences/Path menu pointing to the nn_tilde folder.
On MacOS, unlike Max/MSP, PureData doesn't include an automatic quarantine removal procedure. Therefore we have to do it manually. Fire up a terminal, and cd to the nn_tilde folder. The following command will fix the issue
xattr -r -d com.apple.quarantine .At its core, nn~ is a translation layer between Max/MSP or PureData and the libtorch c++ interface for deep learning. Alone, nn~ is like an empty shell, and requires pretrained models to operate. You can find a few RAVE models here, or a few vschaos2 models here.
Pretrained model for nn~ are torchscript files, with a .ts extension. You can add these files to nn_tilde/models folders, or any place accessible
8000
through Max / Pd filesystem (Max: Options/File Preferences, PureData: File/Preferences/Path).
New : since v1.6.0, some models are directly downloadable through IRCAM Forum API.
Once this is done, you can load a model with nn~ by providing its name as first argument (for example, here isis.ts located inside nn_tilde/models for Max, or among the PureData patch):
| Max / MSP | PureData |
|---|---|
Note that you have to include the .ts extension in the PureData version. Depending on the model loaded, there will be a different number of inlets / outlets, corresponding to the different inputs and outputs of the model.
Coming with v1.6.0, the nn.info object allows model inspection and fetching avilable models for download on the IRCAM-API. With this object, you can get available methods and attributes for a given model. For example, you can see below that a RAVE model has three different methods : encode, decode, and forward.
Models can have several methods, that correspond to several processing pipelines the model can achieve. Hence, each method can have a different number in inlets / outlets. The method is given as the third argument (for exemple, decode above), and equals forward by default.
It is possible the internal state of the module through attributes, that are model-dependent and defined at exportation. Model attributes can be set using messages, with the following syntax:
set ATTRIBUTE_NAME ATTRIBUTE_VAL_1 ATTRIBUTE_VAL_2Using Max/MSP and PureData graphical objects, this can lead to an intuitive way to modify the behavior of the model, as shown below where we have two model attributes (i.e. generation temperature and generation mode), and the special enable attribute.
| Max / MSP | PureData |
|---|---|
Internally, nn~ has a circular buffer mechanism that helps maintain a reasonable computational load. You can modify its size through the use of an additional integer after the method declaration, as shown below
| Max / MSP | PureData |
|---|---|
The Max/MSP release of nn~ includes additional externals, namely mc.nn~ and mcs.nn~, allowing the use of the multicanal abilities of Max 8+ to simplify the patching process with nn~ and optionally decrease the computational load.
In the following examples, two audio files are being encoded then decoded by the same model in parallel
This patch can be improved both visually and computationally speaking by using mc.nn~ and using batch operations
Using mc.nn~ we build the multicanal signals over the different batches. In the example above, each multicanal signal will have 2 different canals. We also propose the mcs.nn~ external that builds multicanal signals over the different dimensions, as shown in the example below
In the example above, the two multicanals signals yielded by the nn~ rave encode 2 object have 16 canals each, corresponding to the 16 latent dimensions. This can help patching, while keeping the batching abilities of mc.nn~ by creating an explicit number of inlets / oulets corresponding to the number of examples we want to process in parallel.
To recap, the regular nn~ operates on a single example, and has as many inlets / outlets as the model has inputs / outputs. The mc.nn~ external is like nn~, but can process multiple examples at the same time. The mcs.nn~ variant is a bit different, and can process mulitple examples at the same time, but will have one inlet / outlet per examples.
Enable / Disable computation to save up computation without deleting the model. Similar to how a bypass function would work.
Dynamically reloads the model. Can be useful if you want to periodically update the state of a model during a training.
Prints methods / attributes of the loaded model.
Prints models downloadable through API.
Download a model from the API.
Deletes a downloaded model.
### load
Change dynamically the incoming model.
### method
Change dynamically the used method.
- Download the latest libtorch (CPU) here and unzip it to a known directory
- Run the following commands:
git clone https://github.com/acids-ircam/nn_tilde --recurse-submodules
cd nn_tilde
curl -L https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh > miniconda.sh
chmod +x ./miniconda.sh
bash ./miniconda.sh -b -u -p ./env
source ./env/bin/activate
pip install -r requirements.txt
conda install -c conda-forge curl
mkdir build
cd build
mkdir puredata_include
curl -L https://raw.githubusercontent.com/pure-data/pure-data/master/src/m_pd.h -o puredata_include/m_pd.h
export CC=$(brew --prefix llvm)/bin/clang
export CXX=$(brew --prefix llvm)/bin/clang++
cd build
cmake ../src -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_PREFIX_PATH=../env/lib/python3.12/site-packages/torch -DCMAKE_BUILD_TYPE=Release -DPUREDATA_INCLUDE_DIR=../puredata_include -DCMAKE_OSX_ARCHITECTURES=arm64
cmake --build . --config Releaseplease replace arm64 in the last line by x86_64 if you want compile for 64 bits. You can remove -DPUREDATA_INCLUDE_DIR=../puredata_include to compile only for Max. The Max package is produced in src/, and Pd external in build/frontend/puredata/Release.
- Download Libtorch (CPU) and dependencies here and unzip to a known directory.
- Install Visual Studio Redistribuable
- Run the following commands (here for Git Bash):
git clone https://github.com/acids-ircam/nn_tilde --recurse-submodules
cd nn_tilde
curl -L https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.6.0%2Bcpu.zip > "libtorch.zip"
unzip libtorch.zip
mkdir pd
cd pd
curl -L https://msp.ucsd.edu/Software/pd-0.55-2.msw.zip -o pd.zip
unzip pd.zip
mv pd*/src .
mv pd*/bin .
cd ..
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.bat
./vcpkg.exe integrate install
./vcpkg.exe install curl
cd ..
mkdir build
cd build
mkdir puredata_include
curl -L https://raw.githubusercontent.com/pure-data/pure-data/master/src/m_pd.h -o puredata_include/m_pd.h
export CC=$(brew --prefix llvm)/bin/clang
export CXX=$(brew --prefix llvm)/bin/clang++
cd build
cmake ../src -G "Visual Studio 17 2022" -DTorch_DIR=../libtorch/share/cmake/Torch -DPUREDATA_INCLUDE_DIR=../pd/src -DPUREDATA_BIN_DIR=../pd/bin -A x64
cmake --build . --config ReleaseYou can remove -DPUREDATA_INCLUDE_DIR=../puredata_include to compile only for Max. The Max package is produced in src/, and Pd external in build/frontend/puredata/Release.
not availble in v1.6.0, planned in next version ; please take previous versions if needed
While nn~ can be compiled and used on Raspberry Pi, you may have to consider using lighter deep learning models. We currently only support 64bit OS.
Install nn~ for PureData using
curl -s https://raw.githubusercontent.com/acids-ircam/nn_tilde/master/install/raspberrypi.sh | bashThis work is led at IRCAM, and has been funded by the following projects
- ANR MakiMono
- ACTOR
- DAFNE+ N° 101061548
