docs(delivery): added first draft of e2e ack guide #11760
Conversation
|
|
✅ Deploy Preview for vector-project ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
…removed list of supported sinks/sources
Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Agree that's important. That's now in there. |
| You can set and control the acknowledgement feature either at the global level or | ||
| the sink level. Let's start with the global configuration option first, as it is | ||
| easy as flipping on a switch. As you would expect from a global configuration, the | ||
| configuration below turns end-to-end acknowledgement on for every source: |
There was a problem hiding this comment.
| configuration below turns end-to-end acknowledgement on for every source: | |
| configuration below turns end-to-end acknowledgement on for every sink: |
There was a problem hiding this comment.
Well, technically it is turning it on for every source, which is where the behavior change actually happens. This is a point of confusion.
| part of as _failed_, and the source will respond accordingly. For example, HTTP sources | ||
| will produce some 400 error code, while sources like Kafka that don't have protocol | ||
| support for rejecting messages behaves similarly to a positive acknowledgement. |
There was a problem hiding this comment.
| part of as _failed_, and the source will respond accordingly. For example, HTTP sources | |
| will produce some 400 error code, while sources like Kafka that don't have protocol | |
| support for rejecting messages behaves similarly to a positive acknowledgement. | |
| part of as _failed_, and the source will respond accordingly. For example, HTTP sources | |
| will produce a 400 or 500-level error code, while sources like Kafka that don't have protocol | |
| support for rejecting messages will simply not ack the messages, allowing them to be reprocessed. |
There was a problem hiding this comment.
Kafka actually does acknowledge the messages on a failure, as there is nothing else it can do. In that way, it behaves similar to the file source.
| ## Edge cases | ||
|
|
||
| Unsurprisingly, there are a few exceptions and edge cases for the end-to-end | ||
| acknowledgement feature. |
There was a problem hiding this comment.
I don't think these are really "edge cases" except maybe the lua one. I might break this up into a few sections like:
## Source and sink support for acknowledgements
## Acknowledgements and disk buffers
## Acknowledgements and user-space transforms
It occurs to me that we should probably document these restrictions somewhere in addition to this guide, but this is a good start. Maybe a dedicated page for acknowledgements under "Concepts": https://vector.dev/docs/about/concepts/
This is the first draft of the end-to-end acknowledgements guide. Let me know if I should expand on any talking points—currently only asking for preliminary review from @bruceg and @jszwedko