8000 feat(blob): multipart upload by Teages · Pull Request #71 · nuxt-hub/core · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(blob): multipart upload #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
324 changes: 323 additions & 1 deletion docs/content/docs/2.storage/3.blob.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,32 @@ export default eventHandler(async (event) => {
})
```

See an example on the Vue side:

```vue [pages/upload.vue]
<script setup lang="ts">
async function uploadImage (e: Event) {
const form = e.target as HTMLFormElement

await $fetch('/api/files', {
method: 'POST',
body: new FormData(form)
}).catch((err) => alert('Failed to upload image:\n'+ err.data?.message))

form.reset()
}
</script>

<template>
<form @submit.prevent="uploadImage">
<label>Upload an image: <input type="file" name="image"></label>
<button type="submit">
Upload
</button>
</form>
</template>
```

#### Params

::field-group
Expand Down Expand Up @@ -248,6 +274,212 @@ Returns a [`BlobObject`](#blobobject) or an array of [`BlobObject`](#blobobject)

Throws an error if `file` doesn't meet the requirements.

### `handleMultipartUpload()`

Handle the request to support multipart upload.

```ts [server/api/files/multipart/[action\\]/[...pathname\\].ts]
export default eventHandler(async (event) => {
return await hubBlob().handleMultipartUpload(event)
})
```

::important
Make sure your route includes `[action]` and `[...pathname]` params.
::

On the client side, you can use the `useMultipartUpload()` composable to upload a file in parts.

```vue
<script setup lang="ts">
async function uploadFile(file: File) {
const upload = useMultipartUpload('/api/files/multipart')

const { progress, completed, abort } = upload(file)
}
</script>
```

::note{to="#usemultipartupload"}
See [`useMultipartUpload()`](#usemultipartupload) on usage details.
::

#### Params

::field-group
::field{name="contentType" type="string"}
The content type of the blob.
::
::field{name="contentLength" type="string"}
The content length of the blob.
::
::field{name="addRandomSuffix" type="boolean"}
If `true`, a random suffix will be added to the blob's name. Defaults to `false`.
::
::

### `createMultipartUpload()`

::note
We suggest to use [`handleMultipartUpload()`](#handlemultipartupload) method to handle the multipart upload request.
::

Start a new multipart upload.

```ts [server/api/files/multipart/[...pathname\\].post.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)

const mpu = await hubBlob().createMultipartUpload(pathname)

return {
uploadId: mpu.uploadId,
pathname: mpu.pathname,
}
})
```

#### Params

::field-group
::field{name="pathname" type="String"}
The name of the blob to serve.
::
::field{name="options" type="Object"}
The put options. Any other provided field will be stored in the blob's metadata.
::field{name="contentType" type="String"}
The content type of the blob. If not given, it will be inferred from the Blob or the file extension.
::
::field{name="contentLength" type="String"}
The content length of the blob.
::
::field{name="addRandomSuffix" type="Boolean"}
If `true`, a random suffix will be added to the blob's name. Defaults to `true`.
::
::
::

#### Return

Returns a `BlobMultipartUpload`

### `resumeMultipartUpload()`

::note
We suggest to use [`handleMultipartUpload()`](#handlemultipartupload) method to handle the multipart upload request.
::

Continue processing of unfinished multipart upload.

To upload a part of the multipart upload, you can use the `uploadPart()` method:

```ts [server/api/files/multipart/[...pathname\\].put.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)
const { uploadId, partNumber } = getQuery(event)

const stream = getRequestWebStream(event)!
const body = await streamToArrayBuffer(stream, contentLength)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
return await mpu.uploadPart(partNumber, body)
})
```

Complete the upload by calling `complete()` method:

```ts [server/api/files/multipart/complete.post.ts]
export default eventHandler(async (event) => {
const { pathname, uploadId } = getQuery(event)
const parts = await readBody(event)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
return await mpu.complete(parts)
})
```

If you want to cancel the upload, you need to call `abort()` method:

```ts [server/api/files/multipart/[...pathname\\].delete.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)
const { uploadId } = getQuery(event)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
await mpu.abort()

return sendNoContent(event)
})
```

A simple example of multipart upload in client with above routes:

```ts [utils/multipart-upload.ts]
async function uploadLargeFile(file: File) {
const chunkSize = 10 * 1024 * 1024 // 10MB

const count = Math.ceil(file.size / chunkSize)
const { pathname, uploadId } = await $fetch(
`/api/files/multipart/${file.name}`,
{ method: 'POST' },
)

const uploaded = []

for (let i = 0; i < count; i++) {
const start = i * chunkSize
const end = Math.min(start + chunkSize, file.size)
const partNumber = i + 1
const chunk = file.slice(start, end)

const part = await $fetch(
`/api/files/multipart/${pathname}`,
{
method: 'PUT',
query: { uploadId, partNumber },
body: chunk,
},
)

uploaded.push(part)
}

return await $fetch(
'/api/files/multipart/complete',
{
method: 'POST',
query: { pathname, uploadId },
body: { parts: uploaded },
},
)
}
```

#### Params

::field-group
::field{name="pathname" type="String"}
The name of the blob to serve.
::
::field{name="uploadId" type="String"}
The upload ID of the multipart upload.
::
::

#### Return

Returns a `BlobMultipartUpload`


#### Params

::field-group
::field{name="event" type="H3Event" required}
The event to handle.
::
::


## `ensureBlob()`

`ensureBlob()` is a handy util to validate a `Blob` by checking its size and type:
Expand Down Expand Up @@ -284,6 +516,10 @@ Throws an error if `file` doesn't meet the requirements.

## Composables

::note
The following composables are meant to be used in the Vue side of your application (not the `server/` directory).
::

### `useUpload()`

`useUpload` is to handle file uploads in your Nuxt application.
Expand Down Expand Up @@ -327,6 +563,59 @@ async function onFileSelect({ target }: Event) {
::
::

#### Return

Return a `MultipartUpload` function that can be used to upload a file in parts.

```ts
const { completed, progress, abort } = upload(file)
const data = await completed
```

### `useMultipartUpload()`

Application composable that creates a multipart upload helper.

```ts [utils/multipart-upload.ts]
export const mpu = useMultipartUpload('/api/files/multipart')
```

#### Params

::field-group
::field{name="baseURL" type="string"}
The base URL of the multipart upload API handled by [`handleMultipartUpload()`](#handlemultipartupload).
::
::field{name="options"}
The options for the multipart upload helper.
::field{name="partSize" type="number"}
The size of each part of the file to be uploaded. Defaults to `10MB`.
::
::field{name="concurrent" type="number"}
The maximum number of concurrent uploads. Defaults to `1`.
::
::field{name="maxRetry" type="number"}
The maximum number of retry attempts for the whole upload. Defaults to `3`.
::
::field{name="prefix" type="string"}
The prefix to use for the blob pathname.
::
::field{name="fetchOptions" type="Omit<FetchOptions, 'method' | 'baseURL' | 'body' | 'parseResponse' | 'responseType'>"}
Override the ofetch options.
The `query` and `headers` will be merged with the options provided by the uploader.
::
::
::

#### Return

Return a `MultipartUpload` function that can be used to upload a file in parts.

```ts
const { completed, progress, abort } = mpu(file)
const data = await completed
```

## Types

### `BlobObject`
Expand All @@ -340,6 +629,40 @@ interface BlobObject {
}
```

### `BlobMultipartUpload`

```ts
export interface BlobMultipartUpload {
pathname: string
uploadId: string
uploadPart(
partNumber: number,
value: string | ReadableStream<any> | ArrayBuffer | ArrayBufferView | Blob
): Promise<BlobUploadedPart>
abort(): Promise<void>
complete(uploadedParts: BlobUploadedPart[]): Promise<BlobObject>
}
```

### `BlobUploadedPart`

```ts
export interface BlobUploadedPart {
partNumber: number;
etag: string;
}
```

### `MultipartUploader`

```ts
export type MultipartUploader = (file: File) => {
completed: Promise<SerializeObject<BlobObject> | undefined>
progress: Readonly<Ref<number>>
abort: () => Promise<void>
}
```

### `BlobListResult`

```ts
Expand All @@ -351,7 +674,6 @@ interface BlobListResult {
}
```


## Examples

### List blobs with pagination
Expand Down
Loading
Loading
0