[rclone v1.63.0] remote Nextcloud not working after upgrade from 1.62.2 to 1.63.0 #7103
Comments
|
This looks like a bug in the regex by not allowing any directory after the username: rclone/backend/webdav/webdav.go Line 572 in 00512e1 Can you try the following:
That should probably work. |
|
Thanks for the quick reply. Yes, by using the endpoint without a remote directory in the service-URL it works (as you described above). Since I'm using Is the fix above considered to be a short term workaround and will the regex be fixed to allow for directories in the URL again in one of the next rclone updates? (would save me from updating all the configs on several machines ;-) |
|
I don't use WebDAV or NextCloud personally so I am not sure whether that regex is an accident or is as expected. @devnoname120 / @ncw will know better. |
|
I tried relaxing the regexp checking the URL - please give this a go @mpsd v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked on branch fix-7103-nextcloud-chunked (uploaded in 15-30 mins) @devnoname120 @tcoupin you both had a hand in the development of the nextcloud chunking - does this seem sensible? |
|
@ncw I haven't checked yet the possible ramifications in the current implementation of allowing to specify a directory in the WebDAV url. Questions:
|
|
@ncw I tested the @devnoname120 my use cases are less driven by permission considerations but more of "can't delete other data than in my current use case" by simply not seeing this data from the other use cases (or in other words: I can't accidently wipe all data below root directory since I have not mapped root). I would prefer to continue to have this "line of defense" in the one time setting up the remote config but to have it in mind in every time I write or adjust a shell script or a one time shell command. Thank you again for looking into this and helping me. |
|
@mpsd Do you use If you use direct commands such as Overall, unless there is a WebDAV-specific behavior with subpaths (such as authentication, etc.) then I don't think that it should be in the scope of backends to handle this properly. Depending on what we choose though, we may want to at least throw a warning explaining the reason why the URL is rejected. What do you think? |
Great. Are you happy with this fix 705572d @devnoname120 ? I've seen this usage before with webdav that people put subdirectories on the URL. @devnoname120 wrote
It does I believe but rclone doesn't know about them.
Users are using it, so we should try not to break backwards compatibility!
Yes, see the alias backend.
The
Rclone is reasonably careful not to let
I think that if it works OK we should allow it as it is currently in use and we try very hard not to break backwards compat! |
|
@ncw Do chunked uploads still complete successfully when a path is specified in the WebDAV URL? |
|
@mpsd can you try a chunked upload easily? |
|
Hi both, yes I can confirm, having a Tested it with chunk size 2M on against Nextcloud v27 and works with both: using my-nextcloud-remote with and without additional dirpath. |
|
@mpsd Did you use a |
|
@devnoname120 I used |
Thanks |
|
@devnoname120 Here we go -
|
|
@mpsd Hmmm I see. Looks like we need to remove the additional path when building the temporary chunks upload directory. |
|
@devnoname120 just cross-tested with 2023/07/02 15:29:56 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone-v1.62.2-linux-amd64/rclone" "-vvv" "copy" "rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip" "my-remotename:"] 2023/07/02 15:30:01 DEBUG : 5 go routines active |
…error Fix rclone#7103 Before this change the RegExp validating the endpoint URL was a bit too strict allowing only /dav/files/USER due to chunking limitations. This patch adds back support for /dav/files/USER/dir/subdir etc. Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
|
@mpsd Could you try this? #7108 Here is a download link to the compiled executable for |
|
@devnoname120 perfect! works! Tested with chunked upload (filesize > 10M) and without (filesize < 10M). Both worked successful. Here is for completeness the debug output for the filesize > 10M: 2023/07/02 18:54:57 DEBUG : 7 go routines active |
…error Fix #7103 Before this change the RegExp validating the endpoint URL was a bit too strict allowing only /dav/files/USER due to chunking limitations. This patch adds back support for /dav/files/USER/dir/subdir etc. Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
|
I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.63.1 @mpsd If you could do one more test on the beta v1.64.0-beta.7129.b4c7b240d when it is uploaded that would be very helpful. I'll merge this to the stable branch when it looks ok to you. |
|
because I want the minimal patches to fix our opensuse package. i mean in doubt I will do |
|
This is now a snapshot package based on the master branch This config was working until 1.63.0. I notice the URL differs from mpsd's config dump. but tbh if this would require migration in the url field then the error message is not very helpful and should guide the user to a proper solution. Something along the lines: The current error makes it look like a code bug. With the change to the url field in the config the current code works. but as i said some more guidance for the user might be good. |
|
I went with Fix on the way to the distribution https://build.opensuse.org/request/show/1097518 |
|
Note that the v1.63-stable branch contains the patches which will go into v1.63.1 I will release v1.63.1 very soon |
|
@darix Could you upstream your patch? By creating a pull request there. 🙂 |
|
done² |
|
I use rclone to access a shared nextcloud folder via webdav. This is not working with 1.63.1. error: config: rclone version: |
|
as the error message tells you ... your URL needs to not end in |
|
as described in nextcloud docs to accessing public shares over WebDAV the URL must be |
|
then you need to set the chunksize to 0. according to the comments the /webdav endpoint does not implement chunking. |
You're misreading the docs. That is talking about accessing Public shares. As in publicly shared files from Nextcloud's share feature. Look further down on that page: It needs to be |
|
On another note, I'm still having trouble with Nextcloud with the current beta. Edit: apparently this is currently due to the Music app holding a lock on folders preventing deletion. Disable/remove the Music app for the time being. From the local machine before move: As you can see... it didn't move the folder. The files are also untouched on the backend. Let's try moving them on the backend: Alright, they moved successfully and kept permissions. Let's try checking the directory contents using webdav from the local machine again: rclone still sees the directory as unchanged. I am able to upload new files just fine. Moving files inside the remote seems to not affect anything. This error was in the Nextcloud logs: Attempting to delete from Nextcloud fails. Edit: apparently this is currently due to the Music app holding a lock on folders preventing deletion. Disable/remove the Music app for the time being. |
|
@Drakonas You are not supposed to manually mess up with the filesystem where NextCloud stores the user files. If you do that, then the filesystem cache of NextCloud doesn't match the actual filesystem. See here for a fix: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/occ_command.html#file-operations With regard to the |
I see. Understood. I wondered if that was the case. That is unfortunate. I'll make sure to do all operations over webdav then. I also figured out the issue... Apparently the Music app currently prevents folders from being deleted due to a bug. Also, can confirm... after doing files:scan and files:cleanup, move commands work fine with the latest beta. It found 0 errors, proably because I moved the data back to match what was in the cache before continuing. Apologies for the false alarm. |
Starting with Rclone v1.63, which is used in the Nextcloud tests for synchronization, the client relies on the correct WebDAV endpoint url, see rclone/rclone#7103
|
this is still a thing, right? rclone 1.64 2023/10/12 14:36:12 Failed to create file system for ":webdav:": the remote url looks incorrect. Note that nextcloud chunked uploads require you to use the /dav/files/USER endpoint instead of /webdav. Please check 'rclone config show remotename' to verify that the url field ends in /dav/files/USERNAME But when i use webdav-vendor=owncloud it works, yet, the webdav is actually a nextcloud v25. in rclone 1.53.3 (ubuntu repo) it does still work with webdav-vendor=nextcloud. Am i missing something? |
|
@devnoname120 yes, it is a public share! |
That sounds like a good plan to me. |
please add that to the docs over at https://rclone.org/webdav/ or even better, make it a default if it's using this new self-terminating "feature" is literally breaking existing rclone configurations world wide that relied on read-only or small file usage, so yes.. it would be wise, if it would not terminate by default. in my case, I actually can't access defaults should always reflect a functional state or at least notify about required changes. I'm running this successfully now: |
|
I encourage you to modify the documentation and do a pull request on your own.
I agree that rclone should fallback to the previous no-chunk upload mechanism + display a warning instead of erroring out. See the 3rd comment above yours.
With regard to shadowdrive it's their problem if they botched up Nextcloud's APIs. We don't support non-compliant Nextcloud setups. You should switch the remote type to plain WebDAV if the remote is not a properly-configured Nextcloud server.
…On Oct 19, 2023, 13:43 +0200, Jan-Stefan Janetzky ***@***.***>, wrote:
@darix wrote:
> then you need to set the chunksize to 0. according to the comments the /webdav endpoint does not implement chunking.
please add that to the docs over at https://rclone.org/webdav/ or even better, make it a default if it's using /webdav..
this new "feature" is literally breaking existing rclone configurations world wide.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@markusHaferkamp put Note that you misspelled it as |
|
@markusHaferkamp when I try your config with rclone v1.67.0 I get this This means the code that makes the error was never called and that is the only place it is called. If you aren't seeing that then there is something wrong, like you aren't running the version of rclone you think you are or you are using the wrong config or something like that. |
|
@ncw Well, amn't I the idiot! You're right: Restic runs with root privileges, and has its own
edit. |
Thank you for developing rclone!
What is the problem you are having with rclone?
After upgrading from version 1.62.2 to 1.63.0 the use of Nextcloud remote dumps with
2023/07/01 12:43:26 Failed to create file system for "***my-remotename***": chunked upload with nextcloud must use /dav/files/USER endpoint not /webdavI use already that hinted endpoint (/dav/files/USER), have tried to delete and add the remote via
rclone configagain and from scratch and got the same error.My
rclone configentry isEditing existing "my-remotename" remote with options:
What is your rclone version (output from
rclone version)rclone v1.63.0
Which OS you are using and how many bits (e.g. Windows 7, 64 bit)
Debian Linux 11.7
Which cloud storage system are you using? (e.g. Google Drive)
Nextcloud v27 (worked with v1.62.2 / doesn't work with 1.63.0)
The command you were trying to run (e.g.
rclone copy /tmp remote:tmp)All rclone commands like
lsorcopyetc.A log from the command with the
-vvflag (e.g. output fromrclone -vv copy /tmp remote:tmp)rclone -vv ls "***my-remotename***":2023/07/01 12:56:23 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "-vv" "ls" "***my-remotename***:"]2023/07/01 12:56:23 DEBUG : Creating backend with remote "***my-remotename***:"2023/07/01 12:56:23 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"2023/07/01 12:56:23 DEBUG : found headers:2023/07/01 12:56:23 Failed to create file system for "***my-remotename***:": chunked upload with nextcloud must use /dav/files/USER endpoint not /webdavHow to use GitHub
The text was updated successfully, but these errors were encountered: