-
Notifications
You must be signed in to change notification settings - Fork 9
Error if body not fully downloaded #947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,5 @@ | ||
use http::{HeaderName, header::CONTENT_LENGTH}; | ||
use std::num::ParseIntError; | ||
use std::{fs, io}; | ||
|
||
#[derive(thiserror::Error, Debug)] | ||
|
@@ -8,6 +10,35 @@ pub enum DownloadError { | |
|
||
#[error("I/O error while downloading file: {0}")] | ||
IoError(#[from] std::io::Error), | ||
|
||
#[error("Missing required header: `{0}`")] | ||
MissingRequiredHeader(HeaderName), | ||
|
||
#[error("Failed to convert header value for `{0}` into a string")] | ||
HeaderEncodingError(HeaderName), | ||
|
||
#[error("Cannot parse into an integer: {0}")] | ||
CannotParseInteger(ParseIntError), | ||
|
||
#[error("Expected `{expected}` bytes received `{received}`")] | ||
UnexpectedBytes { | ||
expected: u64, | ||
received: u64, | ||
path_deleted: Result<(), std::io::Error>, | ||
}, | ||
} | ||
|
||
impl DownloadError { | ||
/// Do not suggest a retry if the failure is due to disk error or if the path could not be cleaned | ||
pub fn retry_suggested(&self) -> bool { | ||
!matches!( | ||
self, | ||
DownloadError::UnexpectedBytes { | ||
path_deleted: Err(_), | ||
.. | ||
} | DownloadError::IoError(_) | ||
) | ||
} | ||
} | ||
|
||
/// Downloads a file via HTTP(S) to a local path | ||
|
@@ -32,9 +63,27 @@ pub fn download_file( | |
destination: impl AsRef<std::path::Path>, | ||
) -> Result<(), DownloadError> { | ||
let response = ureq::get(uri.as_ref()).call().map_err(Box::new)?; | ||
let mut reader = response.into_body().into_reader(); | ||
let mut file = fs::File::create(destination.as_ref())?; | ||
io::copy(&mut reader, &mut file)?; | ||
let expected: u64 = response | ||
.headers() | ||
.get(CONTENT_LENGTH) | ||
.ok_or_else(|| DownloadError::MissingRequiredHeader(CONTENT_LENGTH))? | ||
.to_str() | ||
.map_err(|_| DownloadError::HeaderEncodingError(CONTENT_LENGTH))? | ||
.parse() | ||
.map_err(DownloadError::CannotParseInteger)?; | ||
Comment on lines
+68
to
+73
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we want to change this function to require a Seems like this might be a bit surprising for a generic There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps it'd make sense to use the That would also eliminate the need to manually get and parse the header. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 100% overlooked that, thanks. Great suggestion. I'm wondering if ureq BodyReader is able to somehow fail the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I found that the behavior is documented in the changelog: https://github.com/algesten/ureq/blob/main/CHANGELOG.md#140:
I also experimentally confirm that ureq does the correct thing in the chunked and fixed content-length case (errors the from the IO reader), see below. So, it looks like this case is handled and we don't need to explicitly check. Closing my PR. Fixed length
Chunked
|
||
|
||
Ok(()) | ||
let mut file = fs::File::create(destination.as_ref())?; | ||
let received = io::copy(&mut response.into_body().into_reader(), &mut file)?; | ||
// Ensure file is closed | ||
drop(file); | ||
if received == expected { | ||
Ok(()) | ||
} else { | ||
let path_deleted = fs::remove_file(&destination); | ||
Err(DownloadError::UnexpectedBytes { | ||
expected, | ||
received, | ||
path_deleted, | ||
}) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got a little clever with this representation. I'm not sure if this is a net good or bad idea (to return a Result here).
My logic for this is: It seems unlikely that the delete would fail if we were able to create and write to the file, however, it's still possible so we need to represent it. It seems wrong to return an
DownloadError::IoError
as that explicitly says it's for io errors during download (as opposed to cleanup). We could add another variant just for this case, but it would end up having almost the same information.The primary error here is the bytes not written, whether or not the disk could be cleaned is a secondary concern.
I'm open to alternative suggestions.