8000 bug: SQS FIFO queue: no messages after the first receive call · Issue #12600 · localstack/localstack · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

bug: SQS FIFO queue: no messages after the first receive call #12600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 task done
patrickomatic opened this issue May 9, 2025 · 3 comments
Open
1 task done

bug: SQS FIFO queue: no messages after the first receive call #12600

patrickomatic opened this issue May 9, 2025 · 3 comments
Labels
aws:sqs Amazon Simple Queue Service status: backlog Triaged but not yet being worked on type: bug Bug report

Comments

@patrickomatic
Copy link
patrickomatic commented May 9, 2025 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Hello, several people at my work have noticed this behavior with localstack where a FIFO queue will successfully return messages on the first receive, but then not on subsequent receives. This script demonstrates the behavior:

#!/bin/bash
set -e

queue=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/fifo-test.fifo
echo "Creating queue"
awslocal sqs create-queue --queue-name fifo-test.fifo --attributes '{"FifoQueue": "true", "VisibilityTimeout": "300", "ContentBasedDeduplication": "true" }'

for i in `seq 3`; do
  echo "Sending messages"
  awslocal sqs send-message --queue-url $queue --message-body HELLO-1 --message-group-id grp1
  awslocal sqs send-message --queue-url $queue --message-body HELLO-2 --message-group-id grp1
  awslocal sqs send-message --queue-url $queue --message-body HELLO-3 --message-group-id grp1

  echo "Received messages:"
  awslocal sqs receive-message --queue-url $queue --attribute-names All --max-number-of-messages 10
done

you'll see on the first iteration of the loop we get messages, on subsequent ones we get []. And interestingly if you run this script again, you'll get [] on all 3 loops.

Creating queue
{
    "QueueUrl": "http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/fifo-test.fifo"
}
Sending messages
{
    "MD5OfMessageBody": "c8bc452812ef812e20d9aa8bf7059234",
    "MessageId": "17b2885a-ef09-44bd-9d36-308a7dd29f73",
    "SequenceNumber": "15005013108127170560"
}
{
    "MD5OfMessageBody": "4470c5a3c22ebd058e6614e5e523e9db",
    "MessageId": "4ab6c9cd-68dc-4d80-8d68-99aa0af1c079",
    "SequenceNumber": "15005013108127170561"
}
{
    "MD5OfMessageBody": "850da42707592a6bfebc5b111d9f8aba",
    "MessageId": "539c93b6-7fdd-4b48-9551-5fc44bd8059d",
    "SequenceNumber": "15005013108127170562"
}
Received messages:
{
    "Messages": [
        {
            "MessageId": "17b2885a-ef09-44bd-9d36-308a7dd29f73",
            "ReceiptHandle": "YzFlN2VlMDEtMTEyOC00OTI5LTk0OWMtY2Q2ZGE5ZGIyZjdhIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6Zmlmby10ZXN0LmZpZm8gMTdiMjg4NWEtZWYwOS00NGJkLTlkMzYtMzA4YTdkZDI5ZjczIDE3NDY4MTM0MzIuOTg2MzI3NA==",
            "MD5OfBody": "c8bc452812ef812e20d9aa8bf7059234",
            "Body": "HELLO-1",
            "Attributes": {
                "SenderId": "000000000000",
                "SentTimestamp": "1746813430243",
                "MessageGroupId": "grp1",
                "MessageDeduplicationId": "701151af0e67213798417844820c883e91dfbc90fcbbf9b8d76646d9c6555e5e",
                "SequenceNumber": "15005013108127170560",
                "ApproximateReceiveCount": "1",
                "ApproximateFirstReceiveTimestamp": "1746813432986"
            }
        },
        {
            "MessageId": "4ab6c9cd-68dc-4d80-8d68-99aa0af1c079",
            "ReceiptHandle": "MWM4Y2NiMTYtNjA0Yy00MDBlLTk5NDUtMThkNjU2NjdkNDVmIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6Zmlmby10ZXN0LmZpZm8gNGFiNmM5Y2QtNjhkYy00ZDgwLThkNjgtOTlhYTBhZjFjMDc5IDE3NDY4MTM0MzIuOTg2NTA2",
            "MD5OfBody": "4470c5a3c22ebd058e6614e5e523e9db",
            "Body": "HELLO-2",
            "Attributes": {
                "SenderId": "000000000000",
                "SentTimestamp": "1746813431143",
                "MessageGroupId": "grp1",
                "MessageDeduplicationId": "2008da7d34221e5f0824eed69c09c9017788ff2f4bbee4394e2b937d1bb523ce",
                "SequenceNumber": "15005013108127170561",
                "ApproximateReceiveCount": "1",
                "ApproximateFirstReceiveTimestamp": "1746813432986"
            }
        },
        {
            "MessageId": "539c93b6-7fdd-4b48-9551-5fc44bd8059d",
            "ReceiptHandle": "MDgwMTQ2M2UtMjEwNS00YzJiLWE4MTMtZDU5N2Y1OTNhMmJmIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6Zmlmby10ZXN0LmZpZm8gNTM5YzkzYjYtN2ZkZC00YjQ4LTk1NTEtNWZjNDRiZDgwNTlkIDE3NDY4MTM0MzIuOTg2NTU3",
            "MD5OfBody": "850da42707592a6bfebc5b111d9f8aba",
            "Body": "HELLO-3",
            "Attributes": {
                "SenderId": "000000000000",
                "SentTimestamp": "1746813432069",
                "MessageGroupId": "grp1",
                "MessageDeduplicationId": "39e75cbd27ae646276dd1f0e2b45dfe1d271ed335310c184689e739fa4b100ad",
                "SequenceNumber": "15005013108127170562",
                "ApproximateReceiveCount": "1",
                "ApproximateFirstReceiveTimestamp": "1746813432986"
            }
        }
    ]
}
Sending messages
{
    "MD5OfMessageBody": "c8bc452812ef812e20d9aa8bf7059234",
    "MessageId": "17b2885a-ef09-44bd-9d36-308a7dd29f73",
    "SequenceNumber": "15005013108127170563"
}
{
    "MD5OfMessageBody": "4470c5a3c22ebd058e6614e5e523e9db",
    "MessageId": "4ab6c9cd-68dc-4d80-8d68-99aa0af1c079",
    "SequenceNumber": "15005013108127170564"
}
{
    "MD5OfMessageBody": "850da42707592a6bfebc5b111d9f8aba",
    "MessageId": "539c93b6-7fdd-4b48-9551-5fc44bd8059d",
    "SequenceNumber": "15005013108127170565"
}
Received messages:
{
    "Messages": []
}
Sending messages
{
    "MD5OfMessageBody": "c8bc452812ef812e20d9aa8bf7059234",
    "MessageId": "17b2885a-ef09-44bd-9d36-308a7dd29f73",
    "SequenceNumber": "15005013108127170566"
}
{
    "MD5OfMessageBody": "4470c5a3c22ebd058e6614e5e523e9db",
    "MessageId": "4ab6c9cd-68dc-4d80-8d68-99aa0af1c079",
    "SequenceNumber": "15005013108127170567"
}
{
    "MD5OfMessageBody": "850da42707592a6bfebc5b111d9f8aba",
    "MessageId": "539c93b6-7fdd-4b48-9551-5fc44bd8059d",
    "SequenceNumber": "15005013108127170568"
}
Received messages:
{
    "Messages": []
}

I'm using the localstack/localstack:latest docker image

Expected Behavior

We should see messages each time they are put into the queue

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run localstack/localstack

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

awslocal s3 mb s3://mybucket

Environment

- OS: OS X Sonoma 14.6.1
- LocalStack:
  Loc
8000
alStack version: latest
  LocalStack Docker image sha:
  LocalStack build date:
  LocalStack build git hash:

Anything else?

No response
@patrickomatic patrickomatic added type: bug Bug report status: triage needed Requires evaluation by maintainers labels May 9, 2025
@localstack-bot
Copy link
Collaborator

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Support if you are a licensed user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines.

@brunodmartins
Copy link
Contributor

I will work on this issue!

@Anze1508 Anze1508 added aws:sqs Amazon Simple Queue Service status: backlog Triaged but not yet being worked on and removed status: triage needed Requires evaluation by maintainers labels May 12, 2025
@brunodmartins
Copy link
Contributor

@patrickomatic and @Anze1508 there is no bug, this is an expected behavior due to the FIFO and visibility. As explained on the AWS Docs:

For FIFO queues, messages with the same message group ID are processed in a strict sequence. When a message with a message group ID is in-flight, subsequent messages in that group are not made available until the in-flight message is either deleted or the visibility timeout expires. However, this doesn’t "lock" the group indefinitely– each message is processed in sequence, and only when each message is deleted or becomes visible again will the next message in that group be available to consumers. This approach ensures order 7144 ed processing within the group without unnecessarily locking the group from delivering messages.

I could prove this also debugging and changing the script a bit to validate the scenario:

#!/bin/bash
set -e

queue=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/fifo-test.fifo
echo "Creating queue"
awslocal sqs create-queue --queue-name fifo-test.fifo --attributes '{"FifoQueue": "true", "VisibilityTimeout": "10", "ContentBasedDeduplication": "true" }'

for i in `seq 3`; do
  echo "Sending messages"
  awslocal sqs send-message --queue-url $queue --message-body HELLO-1 --message-group-id grp1
  awslocal sqs send-message --queue-url $queue --message-body HELLO-2 --message-group-id grp1
  awslocal sqs send-message --queue-url $queue --message-body HELLO-3 --message-group-id grp1

  echo "Received messages:"
  awslocal sqs receive-message --queue-url $queue --attribute-names All --max-number-of-messages 10
  sleep 20
done

This script will correctly display all messages, since the visibility timeout is lowered to 10 seconds, and we wait 20 seconds upon each receive-message execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
aws:sqs Amazon Simple Queue Service status: backlog Triaged but not yet being worked on type: bug Bug report
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@patrickomatic @brunodmartins @localstack-bot @Anze1508
0