Context manager to acquire Postgres advisory locks #138
Conversation
This Postgres feature is invaluable when dealing with synchronisations, especially when importing concurrently records from a system. When we export a record, we are able to acquire a lock on the exported record to prevent 2 jobs to export it at the same time. This is different when we import a record for the first time and with several jobs running in parallel, chances are high that 2 jobs will import the same record at the same moment. The Postgres advisory lock comes handy there for they allow to acquire an application lock. Usually we'll acquire the lock at the beginning of an import (beginning of ``Importer.run()``) and we'll throw a ``RetryableJobError`` if the lock cannot be acquired so the job is retried later. The lock will remain in place until the end of the transaction. Example: - Job 1 imports Partner A - Job 2 imports Partner B - Partner A has a category X which happens not to exist yet - Partner B has a category X which happens not to exist yet - Job 1 import category X as a dependency - Job 2 import category X as a dependency Since both jobs are executed concurrently, they both create a record for category X. With this lock: - Job 1 imports Partner A, it puts a lock for this partner - Job 2 imports Partner B, it puts a lock for this partner - Partner A has a category X which happens not to exist yet - Partner B has a category X which happens not to exist yet - Job 1 import category X as a dependency, it puts a lock for this category - Job 2 import category X as a dependency, try to put a lock but can't, Job 2 is retried later, and when it is retried, it sees the category X created by Job 1 See http://topopps.com/implementing-postgres-advisory-locks/ for the article where I learned about the computation of the hash for this purpose.
|
👍 |
|
@guewen This new functionality looks really great! I already see a lot of cases where I need it. As usual, the code looks fine and comes with tests and documentation. |
|
Another contribution to make the framework more robust 👍 |
Context manager to acquire Postgres advisory locks
| be retried when the lock cannot be acquired. | ||
| """ | ||
| if pg_try_advisory_lock(self.env, lock): | ||
| yield |
There was a problem hiding this comment.
retrospectively I wonder if the context manager is a good choice as the lock is acquired until the end of the transaction anyway... Maybe the following is more clear?
def try_advisory_lock(self, lock, retry_seconds=1):
if not pg_try_advisory_lock(self.env, lock):
raise RetryableJobError('Could not acquire advisory lock',...)There was a problem hiding this comment.
retrospectively I wonder if the context manager is a good choice as the lock is acquired until the end of the transaction anyway... Maybe the following is more clear?
@lmignon @damdam-s @pedrobaeza opinions?
There was a problem hiding this comment.
Yeah, it can be clearer, but it isn't also very confused this way. What you prefer.
There was a problem hiding this comment.
Good catch @guewen ! The expected behavior of the contextmanager is to 'scope' the changes to the context to the code enclosed by the with statement. In this case the lock is acquired until the end of the transaction not the end of the enclosed lines.
There was a problem hiding this comment.
But does it matter? They both end at the same more or less, isn't it?
There was a problem hiding this comment.
Exactly what I thought. I'll make a PR tomorrow.
|
@pedrobaeza IMO yes. With the 'with' statement I expect that the lock is released at the end of the block and it's not the case. |
|
OK, I understand. @guewen, please go ahead with the PR, please. |
It uses the new OCA/connector#138, OCA/connector#139 feature.
It uses the new OCA/connector#138, OCA/connector#139 feature.
It uses the new OCA/connector#138, OCA/connector#139 feature.
It uses the new OCA/connector#138, OCA/connector#139 feature.
It uses the new OCA/connector#138, OCA/connector#139 feature.
It uses the new OCA/connector#138, OCA/connector#139 feature.
This Postgres feature is invaluable when dealing with synchronisations,
especially when importing concurrently records from a system.
When we export a record, we are able to acquire a lock on the exported record
to prevent 2 jobs to export it at the same time. This is different when we
import a record for the first time and with several jobs running in parallel,
chances are high that 2 jobs will import the same record at the same moment.
The Postgres advisory lock comes handy there for they allow to acquire an
application lock. Usually we'll acquire the lock at the beginning of an import
(beginning of
Importer.run()) and we'll throw aRetryableJobErrorifthe lock cannot be acquired so the job is retried later. The lock will remain
in place until the end of the transaction.
Example:
Since both jobs are executed concurrently, they both create a record for category X.
With this lock:
is retried later, and when it is retried, it sees the category X created by
Job 1
See http://topopps.com/implementing-postgres-advisory-locks/ for the article
where I learned about the computation of the hash for this purpose.