8000 🐛(backend) race condition create doc by AntoLC · Pull Request #633 · suitenumerique/docs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

🐛(backend) race condition create doc #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,10 @@ and this project adheres to
- 🐛(email) invitation emails in receivers language


## Fixed

- 🐛(backend) race condition create doc #633

## [2.2.0] - 2025-02-10

## Added
Expand Down
26 changes: 24 additions & 2 deletions src/backend/core/api/viewsets.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
from django.contrib.postgres.search import TrigramSimilarity
from django.core.exceptions import ValidationError
from django.core.files.storage import default_storage
from django.db import connection, transaction
from django.db import models as db
from django.db import transaction
from django.db.models.expressions import RawSQL
from django.db.models.functions import Left, Length
from django.http import Http404, StreamingHttpResponse
Expand Down Expand Up @@ -607,6 +607,14 @@ def retrieve(self, request, *args, **kwargs):
@transaction.atomic
def perform_create(self, serializer):
"""Set the current user as creator and owner of the newly created object."""

# locks the table to ensure safe concurrent access
with connection.cursor() as cursor:
cursor.execute(
f'LOCK TABLE "{models.Document._meta.db_table}" ' # noqa: SLF001
"IN SHARE ROW EXCLUSIVE MODE;"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add a comment explaining this mode.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be done when creating children as well? See children method on the viewset.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed with lunika, I don't think the lock should be managed by the viewset, is should be at the model/manager level.
And as you mentioned, the same could happen to children/siblings, it should be covered also.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We did try to add it in the model, see here, but it brings instability in our tests with the factory, a fix brings another instability, and so on.

This naive way is much more stable.
Most of my flakiness is gone as well:
image

)

obj = models.Document.add_root(
creator=self.request.user,
**serializer.validated_data,
Expand Down Expand Up @@ -666,10 +674,19 @@ def trashbin(self, request, *args, **kwargs):
permission_classes=[],
url_path="create-for-owner",
)
@transaction.atomic
def create_for_owner(self, request):
"""
Create a document on behalf of a specified owner (pre-existing user or invited).
"""

# locks the table to ensure safe concurrent access
with connection.cursor() as cursor:
cursor.execute(
f'LOCK TABLE "{models.Document._meta.db_table}" ' # noqa: SLF001
"IN SHARE ROW EXCLUSIVE MODE;"
)

# Deserialize and validate the data
serializer = serializers.ServerCreateDocumentSerializer(data=request.data)
if not serializer.is_valid():
Expand Down Expand Up @@ -775,7 +792,12 @@ def children(self, request, *args, **kwargs):
serializer.is_valid(raise_exception=True)

with transaction.atomic():
child_document = document.add_child(
# "select_for_update" locks the table to ensure safe concurrent access
locked_parent = models.Document.objects.select_for_update().get(
pk=document.pk
)

child_document = locked_parent.add_child(
creator=request.user,
**serializer.validated_data,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
Tests for Documents API endpoint in impress's core app: children create
"""

from concurrent.futures import ThreadPoolExecutor
from uuid import uuid4

import pytest
Expand Down Expand Up @@ -249,3 +250,41 @@ def test_api_documents_children_create_force_id_existing():
assert response.json() == {
"id": ["A document with this ID already exists. You cannot override it."]
}


@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_children_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""

user = factories.UserFactory()

client = APIClient()
client.force_login(user)

document = factories.DocumentFactory()

factories.UserDocumentAccessFactory(user=user, document=document, role="owner")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are not testing concurrency on children creation. But on parent creation. Children will be created on different parent so they have a different path, so you will never have a unique constraint error in you test.

Copy link
Collaborator Author
@AntoLC AntoLC Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, look at here: https://github.com/suitenumerique/docs/pull/633/files#diff-719c99bace85294f27e3909b1067f53da483436c448af38d93a7856fb5ae36faR290

If you remove my patch on the children viewset, this test gives an error.


def create_document():
return client.post(
f"/api/v1.0/documents/{document.id}/children/",
{
"title": "my child",
},
)

with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document)
future2 = executor.submit(create_document)

response1 = future1.result()
response2 = future2.result()

assert response1.status_code == 201
assert response2.status_code == 201

document.refresh_from_db()
assert document.numchild == 2
31 changes: 31 additions & 0 deletions src/backend/core/tests/documents/test_api_documents_create.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
Tests for Documents API endpoint in impress's core app: create
"""

from concurrent.futures import ThreadPoolExecutor
from uuid import uuid4

import pytest
Expand Down Expand Up @@ -51,6 +52,36 @@ def test_api_documents_create_authenticated_success():
assert document.accesses.filter(role="owner", user=user).exists()


@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""

def create_document(title):
user = factories.UserFactory()
client = APIClient()
client.force_login(user)
return client.post(
"/api/v1.0/documents/",
{
"title": title,
},
format="json",
)

with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document, "my document 1")
future2 = executor.submit(create_document, "my document 2")

response1 = future1.result()
response2 = future2.result()

assert response1.status_code == 201
assert response2.status_code == 201


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add test for the children use case as well?

Copy link
Member
@lunika lunika Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we are working on it but it's much more difficult to reproduce

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

def test_api_documents_create_authenticated_title_null():
"""It should be possible to create several documents with a null title."""
user = factories.UserFactory()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

# pylint: disable=W0621

from concurrent.futures import ThreadPoolExecutor
from unittest.mock import patch

from django.core import mail
Expand Down Expand Up @@ -425,6 +426,36 @@ def test_api_documents_create_for_owner_new_user_no_sub_no_fallback_allow_duplic
assert document.creator == user


@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""

def create_document(title):
user = factories.UserFactory()
client = APIClient()
client.force_login(user)
return client.post(
"/api/v1.0/documents/",
{
"title": title,
},
format="json",
)

with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document, "my document 1")
future2 = executor.submit(create_document, "my document 2")

response1 = future1.result()
response2 = future2.result()

assert response1.status_code == 201
assert response2.status_code == 201


@patch.object(ServerCreateDocumentSerializer, "_send_email_notification")
@override_settings(SERVER_TO_SERVER_API_TOKENS=["DummyToken"], LANGUAGE_CODE="de-de")
def test_api_documents_create_for_owner_with_default_language(
Expand Down
Loading
0