Added ConcurrentOrdereredMap to avoid synchronization on GetQueues #117
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AbstractFrontierService will use it instead of a synchronized LinkedHashMap to avoid contention during long lasting operations such as ListURLs or CountURLs. Signed-off-by: Laurent Klock <Laurent.Klock@arhs-cube.com>
jnioche
requested changes
Mar 8, 2025
service/src/main/java/crawlercommons/urlfrontier/service/ConcurrentInsertionOrderMap.java
Show resolved
Hide resolved
service/src/main/java/crawlercommons/urlfrontier/service/ConcurrentOrderedMap.java
Show resolved
Hide resolved
service/src/test/java/crawlercommons/urlfrontier/service/ConcurrentOrderedMapTest.java
Show resolved
Hide resolved
Signed-off-by: Laurent Klock <Laurent.Klock@arhs-cube.com>
|
Regarding the performance comparison, I setup locally a stress test which does the following:
Each thread switches between two crawls containing 350257 and 697743 URLs With 10 threads doing 10 times the loop, I get the following numbers:
|
jnioche
approved these changes
Mar 10, 2025
|
Looks good. Maybe change the title of this PR so that it explains succinctly what it achieves? |
rzo1
approved these changes
Mar 11, 2025
Signed-off-by: Laurent Klock <Laurent.Klock@arhs-cube.com>
Done |
jnioche
approved these changes
Mar 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The purpose of this PR is to avoid the synchronization on the internal queue map.
(Avoid synchronized blocks around GetQueues calls)
The synchronized(getQueues()) call causes contention during long queue traversals which are very likely to occur during calls to countURLs & ListURLs.
This change is based on the ConcurrentOrdereredMap class which uses Guava striped Locks to provide fine
fine-grained locking and internal ConcurrentHashMap and ConcurrentSkipListMap for maintaining insertion order.