Set a default max shard size at 4GiB #755
Conversation
This is the largest that is supported on the client side, furthermore there are larger issues that need to be addressed to be able to support shard sizes larger than this to reduce the number of errors.
There was an issue with this method consuming lots of memory and CPU when it was run over large data sets. This new method uses operating system features to determine if there is space available on the disk. This is a quick fix to address major concerns until we have better accounting of storage and bandwidth usage.
|
I would expect out of memory exceptions caused by usedSpace calculation. We had this problem in the past: storj-archived/storjshare-daemon#247 (comment) I will test this with my ~4TB farmer. |
|
Yes, that's why I switched it from using |
|
I installed your branch but |
|
You may need to add |
|
Looks better now. The check is running. I don't see any memory problems. Please take a look at: #738 The SpaceCheck ignores my allocated size and is only looking on the total free space of my drive. That will not fix this issue. |
|
spaceAvailable should be set, we can test to confirm. The issue with ignoring the allocated size will need to be addressed in another PR (as that check is currently expensive to make). |
|
Yea that is correct. At the moment we only see the total used space of that drive and can't see how many of that are storj shards. That was the part with the out of memory exception we don't want to get again. Ok #738 and storj-archived/storjshare-daemon#307 can't be fix that easy. |
|
Yeah, they require some larger overhauling. However immediately concerns should be addressed with storj-archived/bridge#547 for storj-archived/storjshare-daemon#307 |
|
I merged models. Now bridge needs to update the package json to use that |
|
Models isn't part of this, but I think we're good on this one to merge. |
|
And then we can make a release at |
This is the largest that is supported on the client side, furthermore there are
larger issues that need to be addressed to be able to support shard sizes larger
than this to reduce the number of errors.
cc: @littleskunk @tempestb @aleitner
Goes along with pull requests:
Related issues: