8000 docs: article about AI · nuxt-hub/core@e3f7865 · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Commit e3f7865

Browse files
atinuxatinux
committed
docs: article about AI
1 parent 0a363cc commit e3f7865

File tree

2 files changed

+286
-17
lines changed

2 files changed

+286
-17
lines changed

‎docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -467,8 +467,6 @@ Or go to https://admin.hub.nuxt.com and select your project.
467467

468468
Congratulations! You've now built a fully functional drawing application using Nuxt and Cloudflare R2 for storage. Users can create drawings, save them to the cloud, and access them from anywhere.
469469

470-
Next, we are going to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI.
471-
472470
Feel free to expand on this foundation and add your own unique features to make Atidraw yours!
473471

474472
::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"}
@@ -477,3 +475,5 @@ Feel free to expand on this foundation and add your own unique features to make
477475
::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"}
478476
The demo is available at **draw.nuxt.dev**.
479477
::
478+
479+
Checkout the next article on how to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI: [Cloudflare AI for User Experience](/blog/cloudflare-ai-for-user-experience).

‎docs/content/5.blog/3.cloudflare-ai-for-user-experience.md

Lines changed: 284 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,39 +8,308 @@ authors:
88
src: https://avatars.githubusercontent.com/u/904724?v=4
99
to: https://x.com/atinux
1010
username: atinux
11-
date: 2024-08-12
11+
date: 2024-08-26
1212
category: Tutorial
1313
draft: true
1414
---
1515

1616
## Introduction
1717

18-
The application is [github.com/atinux/atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt.
18+
Let's improve [Atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt.
1919

20-
It has basic features such as:
20+
The application has basic features such as:
2121
- Auth with Google, GitHub or Anonymously based on [`nuxt-auth-utils`](https://github.com/Atinux/nuxt-auth-utils)
2222
- Draw with [`signature_pad`](https://github.com/szimek/signature_pad)
2323
- Upload and store drawings with Cloudflare R2 using [`hubBlob()`](/docs/features/blob)
24-
- Deploy to the Edge with [nuxthub deploy](https://github.com/nuxt-hub/cli) using a [GitHub action](./.github/workflows/deploy.yml)
2524

2625
You can play with it on [draw.nuxt.dev](https://draw.nuxt.dev).
2726

28-
::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.jpg" controls class="w-full h-auto" class="border rounded dark:border-gray-800 md:w-2/3"}
29-
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.webm" type="video/webm"}
30-
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.mp4" type="video/mp4"}
31-
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.ogg" type="video/ogg"}
27+
::tip
28+
Checkout how we created Atidraw in the ["Code, Draw, Deploy: A drawing app with Nuxt & Cloudflare R2"](/blog/drawing-app-with-nuxt-and-cloudflare-r2) blog post.
3229
::
3330

34-
Or deploy it directory on your NuxtHub account:
31+
## Cloudflare AI Pricing
3532

36-
::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"}
37-
:img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"}
33+
This is important to know how Cloudflare AI models are billed.
34+
35+
Cloudflare free allocation allows anyone to use a total of 10,000 Neurons per day at no charge.
36+
37+
Neurons are Cloudflare way of measuring AI outputs across different models. To give you a sense of what you can accomplish with 10,000 Neurons, you can generate:
38+
- 100-200 LLM responses
39+
- 500 translations
40+
- 500 seconds of speech-to-text audio
41+
- 10,000 text classifications
42+
- 1,500 - 15,000 embeddings
43+
44+
Once you reach the free allocation, you can use the AI models with a pay-as-you-go model of $0.011 / 1,000 Neurons.
45+
46+
::tip
47+
**AI Models in beta are free to use**, at this time, all the models we use in this tutorial are in beta, **so you we use them for free**.
48+
::
49+
50+
::note{to="https://developers.cloudflare.com/workers-ai/platform/pricing" target="_blank"}
51+
Read more about **Cloudflare AI pricing**.
52+
::
53+
54+
## Add AI to Nuxt
55+
56+
To add AI to our Nuxt application, we need to enable the AI feature on our `nuxt.config.ts` file.
57+
58+
```ts [nuxt.config.ts]
59+
export default defineNuxtConfig({
60+
hub: {
61+
ai: true, // <--- Enable AI
62+
blob: true,
63+
}
64+
})
65+
```
66+
67+
Then, we want to make sure our project is linked to our NuxtHub account.
68+
69+
```bash [Terminal]
70+
npx nuxthub link
71+
```
72+
73+
::note
74+
This will allow the module to call AI models from your linked Cloudflare account in development mode.
75+
::
76+
77+
::tip{icon="i-ph-rocket-launch"}
78+
That's it! We can now use the AI models from Cloudflare using [`hubAI()`](/docs/features/ai).
3879
::
3980

40-
## Adding AI
81+
I've been thinking on two features:
82+
- Generate the alternative text for the user drawings, improving the accessibility of the application & SEO.
83+
- Generate an image based from the drawing and the alternative text, as a way to make the application more interactive.
84+
85+
Before starting, I took a look at [Cloudflare's multi modal playground](https://multi-modal.ai.cloudflare.com/) and played with some models.
86+
87+
:nuxt-img{alt="Atidraw AI models" dataZoomSrc="/images/blog/atidraw-ai-models.png" height="515" src="/images/blog/atidraw-ai-models.png" width="915"}
88+
89+
This guided me to the following models:
90+
- [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) for the alternative text
91+
- [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) for the alternative drawing
92+
93+
::note{to="https://developers.cloudflare.com/workers-ai/models/" target="_blank"}
94+
See all **Cloudflare AI models** available.
95+
::
96+
97+
## Image Alternative Text
98+
99+
To generate the alternative text for the user drawings, we use the [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) model.
100+
101+
Let's see our current `/api/upload` route:
102+
103+
```ts [server/api/upload.post.ts]
104+
export default eventHandler(async (event) => {
105+
// Make sure the user is authenticated to upload
106+
const { user } = await requireUserSession(event)
107+
108+
// Read the form data
109+
const form = await readFormData(event)
110+
const drawing = form.get('drawing') as File
111+
112+
// Make sure the drawing is a jpeg image and is not larger than 1MB
113+
ensureBlob(drawing, { maxSize: '1MB', types: ['image/jpeg'] })
114+
115+
// Create a new pathname to be smaller than the last one uploaded (for a desc ordering)
116+
const name = `${new Date('2050-01-01').getTime() - Date.now()}`
117+
// Store the image in the R2 bucket with the `drawings/` prefix
118+
return hubBlob().put(`${name}.jpg`, drawing, {
119+
prefix: 'drawings/',
120+
addRandomSuffix: true,
121+
customMetadata: {
122+
userProvider: user.provider,
123+
userId: user.id,
124+
userName: user.name,
125+
userAvatar: user.avatar,
126+
userUrl: user.url,
127+
},
128+
})
129+
})
130+
```
131+
132+
The `LLaVA` model is expecting a `Uint8Array` of the image and a `prompt` to generate the alternative text:
133+
134+
```ts
135+
const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', {
136+
prompt: 'Describe this drawing in one sentence.',
137+
// Convert the drawing from File to Uint8Array
138+
image: [...new Uint8Array(await drawing.arrayBuffer())],
139+
}))
140+
```
141+
142+
We can add the description to the custom metadata of the drawing:
143+
144+
```diff [server/api/upload.post.ts]
145+
export default eventHandler(async (event) => {
146+
// ...
147+
+ const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', {
148+
+ prompt: 'Describe this drawing in one sentence.',
149+
+ image: [...new Uint8Array(await drawing.arrayBuffer())],
150+
+ }))
41151

42-
While working on [`hubAI()`](/docs/features/ai), I wanted to add a feature to generate the alt text for the user drawings.
152+
// ...
153+
return hubBlob().put(`${name}.jpg`, drawing, {
154+
// ...
155+
customMetadata: {
156+
// ...
157+
userUrl: user.url,
158+
+ description
159+
},
160+
})
161+
})
162+
```
43163

44-
- Image Alt text generation with [`hubAI()`](/docs/features/ai) and [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/)
45-
- Gerate an image based from the drawing with [`hubAI()`](/docs/features/ai) and [Stable Diffusion](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/)
164+
Lastly, we need to update our listing page to display the description in the `<img>` tag but also in the `title` attribute so the user can see the description when hovering the image.
165+
166+
```vue [app/pages/index.vue]
167+
<script setup lang="ts">
168+
const { data } = await useFetch('/api/drawings')
169+
</script>
170+
171+
<template>
172+
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-8">
173+
<div v-for="drawing in data?.blobs" :key="drawing.pathname" class="flex flex-col gap-2">
174+
<img
175+
:src="`/drawings/${drawing.pathname}`"
176+
:alt="drawing.customMetadata.description"
177+
:title="drawing.customMetadata.description"
178+
/>
179+
<!-- ... -->
180+
</div>
181+
</div>
182+
</template>
183+
```
184+
185+
Let's see the result:
186+
187+
::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"}
188+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.webm" type="video/webm"}
189+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.mp4" type="video/mp4"}
190+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.ogg" type="video/ogg"}
191+
::
192+
193+
## Alternative Drawing
194+
195+
To generate a new drawing based from the drawing and the alternative text, we need to use the [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) model.
196+
197+
```ts
198+
await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', {
199+
image: imageAsUint8Array,
200+
prompt: imageAltText,
201+
guidance: 8,
202+
strength: 0.5,
203+
})
204+
```
205+
206+
It takes the following inputs:
207+
- `image` (as `Uint8Array`) to use as a reference for the image generation
208+
- `prompt` to guides the model in generating the image
209+
- `guidance` to control how closely the generated image adheres to the given text prompt
210+
- `strength` to controls how much the model changes the input image, with higher values creating bigger changes.
211+
212+
Let's update our `/api/upload` route to generate the alternative drawing and store it in the R2 bucket.
213+
214+
```ts [server/api/upload.post.ts]
215+
export default eventHandler(async (event) => {
216+
// ...
217+
218+
const aiImage = await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', {
219+
prompt: description || 'A drawing',
220+
guidance: 8,
221+
strength: 0.5,
222+
image: [...new Uint8Array(await drawing.arrayBuffer())],
223+
})
224+
.then((blob: Blob | Uint8Array) => {
225+
// Convert Uint8Array to Blob
226+
if (blob instanceof Uint8Array) {
227+
blob = new Blob([blob])
228+
}
229+
// Store the image in the R2 bucket with the `ai/` prefix
230+
return hubBlob().put(`${name}.png`, blob, {
231+
prefix: 'ai/',
232+
addRandomSuffix: true,
233+
contentType: 'image/png',
234+
})
235+
})
236+
237+
// ...
238+
return hubBlob().put(`${name}.jpg`, drawing, {
239+
// ...
240+
customMetadata: {
241+
// ...
242+
aiImage: aiImage.pathname,
243+
},
244+
})
245+
})
246+
```
247+
248+
As you can see, we store the pathname of the AI generated image in the custom metadata of the drawing.
249+
250+
Now, we can display the AI generated image in the listing page when hovering the user's drawing:
251+
252+
```vue [app/pages/index.vue]
253+
<script setup lang="ts">
254+
const { data } = await useFetch('/api/drawings')
255+
</script>
256+
257+
<template>
258+
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-8">
259+
<div v-for="drawing in data?.blobs" :key="drawing.pathname" class="flex flex-col gap-2">
260+
<div
261+
class="group relative max-w-[400px]"
262+
:title="drawing.customMetadata.description"
263+
>
264+
<!-- User drawing -->
265+
<img
266+
:src="`/drawings/${drawing.pathname}`"
267+
:alt="drawing.customMetadata.description"
268+
class="w-full rounded aspect-1"
269+
loading="lazy"
270+
>
271+
<!-- AI generated image, displayed on hover -->
272+
<img
273+
:src="`/drawings/${drawing.customMetadata?.aiImage}`"
274+
:alt="`AI generated image of ${drawing.customMetadata?.description}`"
275+
class="w-full rounded aspect-1 absolute inset-0 opacity-0 group-hover:opacity-100"
276+
loading="lazy"
277+
>
278+
</div>
279+
</div>
280+
</div>
281+
</template>
282+
```
283+
284+
I am quite happy with the result:
285+
286+
::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"}
287+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.webm" type="video/webm"}
288+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.mp4" type="video/mp4"}
289+
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.ogg" type="video/ogg"}
290+
::
291+
292+
::note
293+
Sometime the AI generated image is black, this is because the model is not able to generate an image from the description, most of the time because it is a sensitive content or misunderstood the description.
294+
::
295+
296+
## Conclusion
297+
298+
This is the end of this tutorial on how to use Cloudflare AI models in a Nuxt application. I hope you enjoyed it and that it gave you some ideas on how to use AI in your Nuxt application.
299+
300+
Feel free to expand on this foundation and add your own unique features to make Atidraw yours!
301+
302+
::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"}
303+
The source code of the app is available at **github.com/atinux/atidraw**.
304+
::
305+
::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"}
306+
The demo is available at **draw.nuxt.dev**.
307+
::
308+
309+
If you prefer, you can also deploy this project on your Cloudflare account by clicking on the button below:
310+
311+
::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"}
312+
:img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"}
313+
::
46314

315+
Happy coding & drawing!

0 commit comments

Comments
 (0)
0