NFiles is a Common Lisp library to help manage file persistence and loading, in particular user-centric files like configuration files.
- Performance:
- Data is not persisted to disk if it does not need to.
- Files are read only once (unless modified externally).
- Extensibility: Persist any data structure the way you want it.
- Reliability: no corruption and no data loss should occur.
- Dynamic and customizable path expansion.
- Extensible serialization and deserialization.
- Cached reads and writes.
When a
fileobject expands to the same path as another one, areadorwriteon it won’t do anything in case there was no change since last write. - (Experimental!) On-the-fly PGP encryption.
- Profile support.
- On read errors, existing files are backed up.
- On write errors, no file is written to disk, the existing file is preserved.
- A
remote-filecan point to a URL, which is automatically downloaded if the local file is not found.
This package was developed after dealing with a problem when delivering Common Lisp images: when an image is generated, path expansion may already be resolved and thus hard-coded within the image, which makes it unfit for delivery. Consider this:
> (defvar *foo-path* (uiop:xdg-config-home))
*FOO-PATH*
> *foo-path*
#P"/home/johndoe/.config/"Now if I ship this image to my friend Kaboom, *foo-path* will expand to
#P"/home/johndoe/.config/"on their machine instead of the expected
#P"/home/kaboom/.config/"A basic session:
(defvar *config-file* (make-instance 'nfiles:config-file :base-path "my-app/init.lisp"))
(nfiles:expand *config-file*)
; => #P"/home/johndoe/.config/my-app/init.lisp"
(setf (nfiles:content *config-file*) "Hello file!") ; The file is written to disk.
(nfiles:content *config-file*)
; => "Hello file!"The following convenience macro ensures the file is updated when done with the body:
(nfiles:with-file-content (content *config-file*)
(format t "Length: ~a~%" (length content))
(setf content (serapeum:string-replace "file" content "config")))The with-paths helper allows for let-style bindings of the expanded paths:
(let ((file1 (make-instance 'nfiles:file))
(file2 (make-instance 'nfiles:file :base-path "alt")))
(nfiles:with-paths ((path1 file1)
(path2 file2))
(list path1 path2)))A remote-file works the same but needs some specialization:
(defmethod nfiles:fetch ((profile nfiles:profile) (file remote-counter-file) &key)
(dex:get (nfiles:url file)))
;; Optional:
(defmethod nfiles:check ((profile nfiles:profile) (file remote-counter-file) content &key)
(let ((path (nfiles:expand file)))
(ironclad:byte-array-to-hex-string
(ironclad:digest-file :sha3 path))))
(let ((file (make-instance 'nfiles:remote-file
;; The URL to download from if the file is not found on disk.
:url (quri:uri "https://example.org")
;; Without base-path, the file won't be saved to disk.
:base-path "/tmp/index.html"
:checksum "794df316afac91572b899b52b54f53f04ef71f275a01c44b776013573445868c95317fc4a173a973e90addec7792ff8b637bdd80b1a6c60b03814a6544652a90")))
;; On access, file is automatically downloaded if needed and the checksum is verified:
(nfiles:content file)
;; ...
)The tests contain more useful examples.
NFiles was designed with configurability in mind. All configuration happens through
subclassing of the file and profile classes together with method
specialization.
All configuration methods are specialized against profile and file so that
the user can easily compose the behaviour:
- Profile-customization impacts all files using that profile;
- File-customization impacts the files of that specific type (or subtype) regardless of their profile.
Of course you can specialize against both!
The specialization methods are divided into the following:
resolve: This is where path resolution is done. On call site, prefer theexpandconvenience wrapper.deserializeandserialize: This is how the content is transformed to the file on disk. These functions are meant to be called by thewrite-fileandread-filemethods.read-fileandwrite-file: This is how the file is read and written to disk. These functions are responsible for calling thedeserializeandserializemethods.fetch: This generic function is only called forremote-fileobjects. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.check: Likefetch, this generic function is only called forremote-fileobjects to test the integrity of the downloaded file. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.
Some NFiles-specific conditions are raised in case of exceptional situations to provide for interactive and customizable behaviour:
external-modification: The file was modified externally. See theon-external-modificationslot to automate what to do in this case.- Read error restarts can also customized, see the
on-read-errorslot to automate what to do in this case. process-error: This may be raised for instance whengpgfails to encrypt. Theuse-recipientrestart is provided to retry with the given recipient.
It’s pure Common Lisp and all compilers plus all operating systems should be supported.
Some notes:
- All compilers but SBCL depend on IOlib to preserve file attributes.
- File attributes might not be preserved on Windows.
- Improve PGP support.
- Support OS-level locks (à-la Emacs / LibreOffice).
- Improve portability, in particular preservation of file attributes may not work on Windows.
- Compressing
write-fileandread-file(for instance with zstd / lz). But is it such a good idea? Users should prefer compression at the level of the file system.
NFiles was originally developed for user file management in Nyxt, so the “N” may stand for it, or “New”, or whatever poetic meaning you may find behind it!