LWN: Comments on "NFS: the new millennium"

NFS: the new millennium

mathstuf — Fri, 12 Aug 2022 02:44:29 +0000

See the GP for the full thing; LWN will elide links and the link quoted is indeed the LWN truncation.

NFS: the new millennium

seamus — Wed, 10 Aug 2022 00:05:44 +0000

Broken/incomplete(?) link

NFS: the new millennium

Wol — Sun, 10 Jul 2022 12:04:51 +0000

I don't think fsync was lying. It just assumed disks were telling the truth ...

Cheers,
Wol

NFS: the new millennium

ssmith32 — Sun, 10 Jul 2022 08:58:56 +0000

Wasn't fsync caught "lying" a while ago? And disks as well, to inflate performance numbers?

Or have I not kept up? I guess SSDs can probably hit their numbers & still guarantee data is truly flushed from every volatile cache...

NFS: the new millennium

lathiat — Thu, 30 Jun 2022 04:13:31 +0000

Yep that's what I though too. It was quite a famous issue where it would re-assemble fragments incorrectly but then pass the checksum and corrupt the UDP data inflight. Partly to do with the NFS/RPC packet size versus MTU.

All good.. just a misunderstanding :D I just happened to discuss the issue with a colleague the day the first post came out :)

NFS: the new millennium

neilbrown — Tue, 28 Jun 2022 23:12:26 +0000

> I assumed you were talking about UDP fragmentation!

You aren't the only one (I think). See https://lwn.net/Articles/898842/

NFS: the new millennium

Sesse — Tue, 28 Jun 2022 08:47:52 +0000

I assumed you were talking about UDP fragmentation!

NFS: the new millennium

neilbrown — Mon, 27 Jun 2022 23:31:03 +0000

> You hit diminishing returns pretty quickly above 1MB on any storage device I've ever worked on.

I suspect you are correct.
How about a database update that needs to perform lots of random writes before a sync is needed? Any single NFS write must be contiguous, so there must be several. Without the v3 COMMIT, every one of those random writes would need to be committed before the write could return.

Or what about writing lots of small files. The NFS client doesn't need to COMMIT until the writeback timer fires, or memory reclaim wants the cache back, or a sync() is requested. Would it not be more efficient to commit when there are lots of dirty files, rather than once for each file?

But the real point is that EVERY other layer in the storage stack has the two phases: write then sync/flush/commit. Why are you so sure that NFS doesn't benefit from also have the same two phases?

NFS: the new millennium

neilbrown — Mon, 27 Jun 2022 23:24:45 +0000

> Neil, seems you forgot about the dangerous fragmentation alluded to in the last installment!

I didn't forget - but there wasn't really anything to say. The fact that fragmentation (of the community, or of the protocol) might happen when separate people have separate needs should be obvious. A possible example of this is JMAP which risks fragmenting the IMAP community. Beyond the fact that I didn't like JMAP when I reviewed it 6 years ago, I don't know any details of this or whether it is a real fragmentation.

But NFS didn't fragment (to my knowledge). The community had regular Connectathon gatherings to ensure interoperability and to discuss issues. And importantly the IETF process was started that allowed anyone to contribute to NFSv4. So the IETF process, which I did mention, likely headed off any possible risk of protocol fragmentation.

NFS: the new millennium

jra — Mon, 27 Jun 2022 19:03:05 +0000

> SMB will tell you it has received a lump of data whereas NFS will tell you when it has been committed to storage. Is this still true?

No. SMB2 write has a flag that tells it not to return success until the write has hit stable storage.

NFS: the new millennium

lathiat — Mon, 27 Jun 2022 03:55:54 +0000

Neil, seems you forgot about the dangerous fragmentation alluded to in the last installment!

Great review though, loved it, thanks! Long time NFS user from web hosting environments since circa 2006.

NFS: the new millennium

janfrode — Sun, 26 Jun 2022 21:05:59 +0000

I've seen large throughput improvements by increasing rsize from 1 MB to 16 MB with Oracle dNFS client, towards NFS-Ganesha on top of GPFS (ESS). The ESS does 8+2p erasure coding, with 1 MB strip size, so 16 MB IOs is the optional size. I don't think wsize was as important, since it could buffer up and do full block size writes on server side -- but this large rsize was needed to ensure full block (16MB) reads.

I don't think other NFS clients support larger than 1 MB rsize/wsize, which seems unfortunate for this kind of storage backend.

NFS: the new millennium

willy — Sun, 26 Jun 2022 12:15:48 +0000

I guess that's going to depend on the backend storage device. You hit diminishing returns pretty quickly above 1MB on any storage device I've ever worked on. Even a 5-disc RAID-5 with 256kB stripe width would handle 1MB writes with aplomb.

NFS: the new millennium

neilbrown — Sun, 26 Jun 2022 01:51:57 +0000

> I am very surprised that you skipped over .....

I skipped over a lot of things. If it didn't relate to state management, then I felt free to skip it unless it helped tell the story.

> (https://www.xkyle.com/solving-the-nfs-16-group-limit-prob...)

That link contains a section "The Best Solution Ever!: A New Option for the NFS Server". I'm glad my contribution there was appreciated :-)

> deferring permissions to an external server with the sec mount option seems to be the only way.

Yes. At least you do need to have the NFS server determine the list of gids. You don't need to use the sec= mount option. Using the default sec=sys works fine with the Linux NFS server if you use --manage-gids. The NetApp server has similar functionality. Others might too.

> Is this a protocol or implementation limit?

It is a limit in the RPC protocol

> I’d love more discussion on the evolution of various security functionality

Not really my area of expertise, sorry.

NFS: the new millennium

zerolagtime — Sat, 25 Jun 2022 23:40:57 +0000

Thanks for a great look at the internals.
I am very surprised that you skipped over a still-standing limit on the number of gids that will be compared on a complex network.
16 gids is the limit (https://www.xkyle.com/solving-the-nfs-16-group-limit-prob...) and deferring permissions to an external server with the sec mount option seems to be the only way. Is this a protocol or implementation limit?
I’d love more discussion on the evolution of various security functionality, like xattrs, encryption algorithm, and the impact felt by stateful firewalls trying to accommodate port ranges.

NFS: the new millennium

neilbrown — Sat, 25 Jun 2022 23:10:04 +0000

NFS limits the size of a single write request in that the server specifies a maximum that the client must honour.
Typically 1MB on Linux. There are costs in making to bigger, so just setting to 1GB wouldn't work. When NFSv3 was first described UDP was still common and 64KB is a hard limit there.
Is there no value in merging multiple 1MB writes on the server?

NFS: the new millennium

willy — Sat, 25 Jun 2022 22:57:47 +0000

I suppose by "a competent local cache", I include the ability for the client to merge writes, which Linux will do.

NFS: the new millennium

neilbrown — Sat, 25 Jun 2022 22:29:29 +0000

> the NFSv2 semantics were just fine for this usage.

Not really. The NFSv2 semantics require the server to sync each request individually, though maybe it could merge concurrent requests by delaying the sync for the first.
The NFSv3 semantics make it easy for the server to gather lots of writes in its cache and sync them all together.

NFS: the new millennium

willy — Sat, 25 Jun 2022 21:55:05 +0000

On a system with a competent local cache, the kernel returns success to the application after the write is copied to the local cache. NFSv3 offers no improvement here because we weren't even calling WRITE in this path. WRITE was called on writeback and on fsync, and the NFSv2 semantics were just fine for this usage.

NFS: the new millennium

pbonzini — Sat, 25 Jun 2022 16:14:39 +0000

The write is only durable after fsync, write is not enough. So even on POSIX systems it's useful to know the moment when the write has been received.

NFS: the new millennium

willy — Sat, 25 Jun 2022 11:42:02 +0000

> SMB will tell you it has received a lump of data whereas NFS will tell you when it has been committed to storage.

NFS v2 only allowed for the WRITE command to be acknowledged when the data was on stable storage. NFS v3 turned it into a two-phase commit, allowing the server to tell the client both "I have received it" and "It is now stable".

Two phase commit isn't particularly useful to clients with a competent local cache. The latency of a write is almost irrelevant; you need to know when the write is durable, not just when it's visible to others. It was of great interest to non-Unix clients, though.

NFS: the new millennium

willy — Sat, 25 Jun 2022 11:36:56 +0000

A protocol which came between NFSv3 and v4 that certainly informed the design of v4 was WebNFS. It also integrated several protocols into one and discarded the use of Portmap.

NFS: the new millennium

wtarreau — Sat, 25 Jun 2022 10:12:11 +0000

Thanks for this detailed explanation and for the numerous links, Neil!

NFS: the new millennium

gerdesj — Sat, 25 Jun 2022 00:39:26 +0000

Great write up about something that I generally take for granted but by 'eck, NFS has shifted some bytes from A to B for me alone. It's nice to get some real insights into the background of these things from someone who knows what they are on about.

NFS for me has a killer feature when compared to SMB and I was only made aware of it by Veeam. This may or may not still be true: SMB will tell you it has received a lump of data whereas NFS will tell you when it has been committed to storage. Is this still true? When you are doing backups to NAS, which involves some pretty monstrous files, this is rather important.

Your 5TB backup is pretty useless with a hole in the middle of it due to a transient error that allowed a block or two to wander off and have a smoke behind the bikesheds and then sidling off for a night out in town instead of resting on disc like good data. OK, a decent backup app will have lots of failsafes available such as reading backups back against the source but ideally data should flow from A to B safely out of the box.

Given the sheer complexity of file access these days - pick a protocol, pick a medium, wedge in a VPN or two and shake it up and see what happens! It's a wonder that files seem to turn up as requested, in one piece.