8000 feat: OS manage custom repos and snapshots by wojcik-dorota · Pull Request #832 · aiven/aiven-docs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: OS manage custom repos and snapshots #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
283 changes: 139 additions & 144 deletions docs/products/opensearch/howto/custom-repositories.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,124 @@
---
title: Create and manage custom repositories in Aiven for OpenSearch®
sidebar_label: Custom repositories
sidebar_label: Manage custom repositories
limited: true
---
import RelatedPages from "@site/src/components/RelatedPages";
import ConsoleLabel from "@site/src/components/ConsoleIcons";
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Aiven for OpenSearch lets you configure custom repositories to store snapshots in your cloud storage.
Configure custom repositories in Aiven for OpenSearch to store [snapshots](/docs/products/opensearch/howto/manage-snapshots) in your cloud storage.

## Supported storage services

You can configure custom repositories for the following object storage services:
## Prerequisites

- Amazon S3
- Google Cloud Storage (GCS)
- Microsoft Azure Blob Storage
<Tabs groupId="group1">
<TabItem value="gui" label="Aiven Console" default>

## Prerequisites
- Custom repositories
[enabled as a limited availability feature](/docs/platform/concepts/service-and-feature-releases#limited-availability-)
- Running Aiven for OpenSearch service
- Access to the [Aiven Console](https://console.aiven.io/)
- Access to a supported object storage service (AWS S3, GCS, or Azure)
- Credentials for the selected storage provider

- An active Aiven for OpenSearch service
- An Aiven authentication [token](/docs/platform/howto/create_authentication_token)
</TabItem>
<TabItem value="os-api" label="OpenSearch API">

- Custom repositories
[enabled as a limited availability feature](/docs/platform/concepts/service-and-feature-releases#limited-availability-)
- [Maintenance updates](/docs/platform/concepts/maintenance-window#maintenance-updates)
applied for your service
- [Security management enabled](/docs/products/opensearch/howto/enable-opensearch-security)
for your service
- [Snapshot permissions](https://docs.opensearch.org/docs/latest/security/access-control/permissions/#snapshot-permissions)
and
[snapshot repository permissions](https://docs.opensearch.org/docs/latest/security/access-control/permissions/#snapshot-repository-permissions)
configured

</TabItem>
<TabItem value="api" label="Aiven API">

- Custom repositories
[enabled as a limited availability feature](/docs/platform/concepts/service-and-feature-releases#limited-availability-)
- Running Aiven for OpenSearch service
- [Aiven API](/docs/tools/api) and authentication [token](/docs/platform/howto/create_authentication_token)
- Access to a supported object storage service (AWS S3, GCS, or Azure)
- Credentials for the selected storage provider

## Limitations
:::note

- The Aiven API does not support automatic snapshot scheduling. You must create, list,
and delete them manually using the API.
- Repository credentials are omitted from API responses for security reasons.
- Aiven API provides a direct interface to the OpenSearch snapshot API.

:::

## Configure custom repositories
</TabItem>
</Tabs>

## Limitations

<Tabs groupId="group1">
<TabItem value="gui" label="Aiven Console" default>

You can configure custom repositories for the following object storage services:

- Amazon S3
- Google Cloud Storage (GCS)
- Microsoft Azure Blob Storage

</TabItem>
<TabItem value="os-api" label="OpenSearch API">

- Supported storage services
- Amazon S3
- Google Cloud Storage (GCS)
- Microsoft Azure Blob Storage
- The following operations are not supported via native OpenSearch API:
- [Create a repository](/docs/products/opensearch/howto/custom-repositories#create-custom-repositories)
- [Remove a repository](/docs/products/opensearch/howto/custom-repositories#remove-a-repository)
- [Edit repository details](/docs/products/opensearch/howto/custom-repositories#view-or-edit-repository-details)
- [List custom repositories](/docs/products/opensearch/howto/custom-repositories#list-custom-repositories)

</TabItem>
<TabItem value="api" label="Aiven API">

- Supported storage services
- Amazon S3
- Google Cloud Storage (GCS)
- Microsoft Azure Blob Storage
- To
[edit repository details](/docs/products/opensearch/howto/custom-repositories#view-or-edit-repository-details),
you can only use the Aiven Console.
- The following operations are not supported via Aiven API:
- [Remove a repository](/docs/products/opensearch/howto/custom-repositories#remove-a-repository)
- [View or edit repository details](/docs/products/opensearch/howto/custom-repositories#view-or-edit-repository-details)

</TabItem>
</Tabs>

## Create custom repositories

Each repository requires a unique name, a storage type (such as S3, Azure, or GCS), and
the appropriate settings for the selected storage provider.

<Tabs groupId="group1">
<TabItem value="gui" label="Aiven Console" default>

1. Log in to the [Aiven Console](https://console.aiven.io/), go to your project, and
open your service's page.
1. Click <ConsoleLabel name="snapshots"/> in the sidebar.
1. On the **Snapshots** page, click **Add repository**.
1. In the **Add custom repository** window:
1. Enter a repository name.
1. Select a storage provider.
1. Give provider-specific details required for accessing the storage.
1. Click **Add**.

</TabItem>
<TabItem value="api" label="Aiven API">
Custom repositories are configured in the `user_config` of your Aiven for OpenSearch
service. Each repository requires a unique name, a storage type (such as S3, Azure, or
GCS), and the appropriate settings for the selected storage provider. Use the following
API request to configure custom repositories:
service. Use the following API request to configure custom repositories:

```sh
curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}" \
Expand Down Expand Up @@ -71,15 +156,22 @@ curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_n
}'
```

## Manage custom repositories
</TabItem>
</Tabs>

After configuring custom repositories, you can manage them using the Aiven API, which
provides a direct interface to the OpenSearch snapshot API.
## List custom repositories

### List custom repositories
<Tabs groupId="group1">
<TabItem value="gui" label="Aiven Console" default>

Use the Aiven API to retrieve a list of custom repositories:
1. Log in to the [Aiven Console](https://console.aiven.io/), go to your project, and
open your service's page.
1. Click <ConsoleLabel name="snapshots"/> in the sidebar.

Find your custom repositories listed on the **Snapshots** page.

</TabItem>
<TabItem value="api" label="Aiven API">
```sh
curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot"\
--header "Authorization: Bearer $TOKEN" \
Expand Down Expand Up @@ -116,135 +208,38 @@ Example response:
}
```

## Manage snapshots in custom repositories

After setting up a custom repository, you can use the Aiven API to create, list,
retrieve, and delete snapshots.
</TabItem>
</Tabs>

### Create a snapshot
## View or edit repository details

Create a snapshot in a custom repository.
<Tabs groupId="group1">
<TabItem value="gui" label="Aiven Console" default>

```sh
curl -s -X POST \
--url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot/aws-repo/first-snapshot" \
--header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/json" \
-d '{"indices": "test*", "include_global_state": false}'
```

Example response:

```json
{
"accepted": true
}
```

### List snapshots in progress
1. Log in to the [Aiven Console](https://console.aiven.io/), go to your project, and
open your service's page.
1. Click <ConsoleLabel name="snapshots"/> in the sidebar.
1. On the **Snapshots** page, find your custom repository and click
<ConsoleLabel name="actions"/> > <ConsoleLabel name="editrepo"/>.
1. Edit repository details and save your changes by clicking **Update**.

Retrieve snapshots that are still being created in a repository.
</TabItem>
<TabItem value="os-api" label="OpenSearch API">

```sh
curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot/aws-repo/_status" \
--header "Authorization: Bearer $TOKEN"
--header "Content-Type: application/json"
```
To view details on a repository, use the
[Get Snapshot Repository](https://docs.opensearch.org/docs/latest/api-reference/snapshots/get-snapshot-repository/)
native OpenSearch API endpoint.

Example response:
</TabItem>
</Tabs>

```json
{
"snapshots": [
{
"repository": "aws-repo",
"snapshot": "second-snapshot",
"state": "SUCCESS",
"shards_stats": {
"done": 1,
"failed": 0,
"total": 1
},
"uuid": "osmCbdF-RMyyUKpWD-4bJA"
}
]
}
```
## Remove a repository

### List all snapshots in a repository

Retrieve all snapshots, including completed and failed ones.

```sh
curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot/aws-repo/_all" \
--header "Authorization: Bearer $TOKEN"
--header "Content-Type: application/json"
```

Example response:

```json
{
"snapshots": [
{
"snapshot": "first-snapshot",
"state": "SUCCESS",
"indices": ["test"],
"uuid": "7cdWedW7RC6FMSktlZTCDw"
},
{
"snapshot": "second-snapshot",
"state": "SUCCESS",
"indices": ["test"],
"uuid": "osmCbdF-RMyyUKpWD-4bJA"
}
]
}
```

### Retrieve a snapshot summary

Get details of a specific snapshot.

```sh
curl -s --url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot/aws-repo/first-snapshot" \
--header "Authorization: Bearer $TOKEN"
--header "Content-Type: application/json"
```

Example response:

```json
{
"snapshots": [
{
"snapshot": "first-snapshot",
"state": "SUCCESS",
"indices": ["test"],
"uuid": "7cdWedW7RC6FMSktlZTCDw"
}
]
}
```

### Delete a snapshot

Delete a snapshot from a repository.

```sh
curl -s -X DELETE \
--url "https://api.aiven.io/v1/project/{project_name}/service/{service_name}/opensearch/_snapshot/aws-repo/first-snapshot" \
--header "Authorization: Bearer $TOKEN"
--header "Content-Type: application/json"
```

Example response:

```json
{
"acknowledged": true
}
```
1. Log in to the [Aiven Console](https://console.aiven.io/), go to your project, and
open your service's page.
1. Click <ConsoleLabel name="snapshots"/> in the sidebar.
1. On the **Snapshots** page, find your custom repository and click
<ConsoleLabel name="actions"/> > <ConsoleLabel name="removerepo"/> > **Remove**.

## Error handling

Expand Down
Loading
0