8000 Switch to `orjson` as faster JSON implementation by Swatinem · Pull Request #772 · codecov/worker · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on May 5, 2025. It is now read-only.

Switch to orjson as faster JSON implementation #772

Merged
merged 2 commits into from
Oct 15, 2024
Merged

Switch to orjson as faster JSON implementation #772

merged 2 commits into from
Oct 15, 2024

Conversation

Swatinem
Copy link
Contributor
@Swatinem Swatinem commented Oct 9, 2024 •
edited
Loading

orjson is supposed to be a faster JSON parser/serializer, with benchmark claiming its ~3x faster at parsing, and up to ~10x faster serializing. (See https://github.com/ijl/orjson?tab=readme-ov-file#performance)

Here I mostly changed the loads codepaths, and most of the dumps ones that do not define a custom serializer implementation.

@Swatinem Swatinem requested a review from a team October 9, 2024 09:14
@Swatinem Swatinem self-assigned this Oct 9, 2024
@Swatinem
Copy link
Contributor Author
Swatinem commented Oct 9, 2024

In Sentry we switched some of the JSON implementations to orjson as well, but hit some problems there:

  • the default python JSON implementation serialized NaN, which is not valid however and will fail in orjson.
  • orjson serializes datetime types using timezones, which some parsers do not expect.

I don’t expect to see any problems related to that, as I doubt we are serializing any datetime objects. The NaN case is more interesting however, and is something to keep an eye on.

@codecov Codecov
Copy link
codecov bot commented Oct 9, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.03%. Comparing base (e373a03) to head (42ea1b5).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #772      +/-   ##
==========================================
- Coverage   98.03%   98.03%   -0.01%     
==========================================
  Files         442      442              
  Lines       36613    36602      -11     
==========================================
- Hits        35895    35883      -12     
- Misses        718      719       +1
Flag Coverage Δ
integration 98.03% <100.00%> (-0.01%) ⬇️
unit 98.03% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 95.92% <100.00%> (-0.01%) ⬇️
OutsideTasks 97.99% <100.00%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
database/utils.py 96.10% <100.00%> (ø)
services/processing/loading.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.20% <100.00%> (ø)
services/report/languages/coveralls.py 100.00% <100.00%> (ø)
services/report/languages/gap.py 97.22% <100.00%> (-2.78%) ⬇️
services/report/parser/legacy.py 100.00% <ø> (ø)
...eport/parser/tests/unit/test_version_one_parser.py 100.00% <100.00%> (ø)
services/report/parser/types.py 100.00% <100.00%> (ø)
services/report/parser/version_one.py 100.00% <100.00%> (ø)
services/report/report_processor.py 95.49% <100.00%> (ø)
... and 6 more

@codecov-staging
Copy link
codecov-staging bot commented Oct 9, 2024 •
edited by codecov-notifications bot
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #772      +/-   ##
==========================================
- Coverage   98.03%   98.03%   -0.01%     
==========================================
  Files         442      442              
  Lines       36613    36602      -11     
==========================================
- Hits        35895    35883      -12     
- Misses        718      719       +1
Flag Coverage Δ
integration 98.03% <100.00%> (-0.01%) ⬇️
unit 98.03% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 95.92% <100.00%> (-0.01%) ⬇️
OutsideTasks 97.99% <100.00%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
database/utils.py 96.10% <100.00%> (ø)
services/processing/loading.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.20% <100.00%> (ø)
services/report/languages/coveralls.py 100.00% <100.00%> (ø)
services/report/languages/gap.py 97.22% <100.00%> (-2.78%) ⬇️
services/report/parser/legacy.py 100.00% <ø> (ø)
...eport/parser/tests/unit/test_version_one_parser.py 100.00% <100.00%> (ø)
services/report/parser/types.py 100.00% <100.00%> (ø)
services/report/parser/version_one.py 100.00% <100.00%> (ø)
services/report/report_processor.py 95.49% <100.00%> (ø)
... and 6 more

@codecov-qa
Copy link
codecov-qa bot commented Oct 9, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.03%. Comparing base (e373a03) to head (42ea1b5).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #772      +/-   ##
==========================================
- Coverage   98.03%   98.03%   -0.01%     
==========================================
  Files         442      442              
  Lines       36613    36602      -11     
==========================================
- Hits        35895    35883      -12     
- Misses        718      719       +1
Flag Coverage Δ
integration 98.03% <100.00%> (-0.01%) ⬇️
unit 98.03% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 95.92% <100.00%> (-0.01%) ⬇️
OutsideTasks 97.99% <100.00%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
database/utils.py 96.10% <100.00%> (ø)
services/processing/loading.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.20% <100.00%> (ø)
services/report/languages/coveralls.py 100.00% <100.00%> (ø)
services/report/languages/gap.py 97.22% <100.00%> (-2.78%) ⬇️
services/report/parser/legacy.py 100.00% <ø> (ø)
...eport/parser/tests/unit/test_version_one_parser.py 100.00% <100.00%> (ø)
services/report/parser/types.py 100.00% <100.00%> (ø)
services/report/parser/version_one.py 100.00% <100.00%> (ø)
services/report/report_processor.py 95.49% <100.00%> (ø)
... and 6 more

@codecov-public-qa Codecov Public QA
Copy link
codecov-public-qa bot commented Oct 9, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.03%. Comparing base (e373a03) to head (42ea1b5).

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #772      +/-   ##
==========================================
- Coverage   98.03%   98.03%   -0.01%     
==========================================
  Files         442      442              
  Lines       36613    36602      -11     
==========================================
- Hits        35895    35883      -12     
- Misses        718      719       +1
Flag Coverage Δ
integration 98.03% <100.00%> (-0.01%) ⬇️
unit 98.03% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 95.92% <100.00%> (-0.01%) ⬇️
OutsideTasks 97.99% <100.00%> (-0.01%) ⬇️
Files Coverage Δ
database/utils.py 96.10% <100.00%> (ø)
services/processing/loading.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.20% <100.00%> (ø)
services/report/languages/coveralls.py 100.00% <100.00%> (ø)
services/report/languages/gap.py 97.22% <100.00%> (-2.78%) ⬇️
services/report/parser/legacy.py 100.00% <ø> (ø)
...eport/parser/tests/unit/test_version_one_parser.py 100.00% <100.00%> (ø)
services/report/parser/types.py 100.00% <100.00%> (ø)
services/report/parser/version_one.py 100.00% <100.00%> (ø)
services/report/report_processor.py 95.49% <100.00%> (ø)
... and 6 more

giovanni-guidini
giovanni-guidini approved these changes Oct 15, 2024
View reviewed changes
Copy link
Contributor
@giovanni-guidini giovanni-guidini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@Swatinem
Switch to orjson as faster JSON implementation
4098131 
`orjson` is supposed to be a faster JSON parser/serializer, with benchmark claiming its ~3x faster at parsing, and up to ~10x faster serializing.

Here I mostly changed the `loads` codepaths, and most of the `dumps` ones that do not define a custom serializer implementation.
@Swatinem
change parsedreport.contents
42ea1b5
@Swatinem Swatinem added this pull request to the merge queue Oct 15, 2024
Merged via the queue into main with commit 3853a7d Oct 15, 2024
26 of 27 checks passed
@Swatinem Swatinem deleted the swatinem/orjson branch October 15, 2024 09:19
Swatinem added a commit to codecov/shared that referenced this pull request Oct 21, 2024
@Swatinem
Switch to orjson for Report de/serialization
9e7cd0c 
This is the same as codecov/worker#772, and for the same reasons:
`orjson` is advertised as being 3x faster for deserialization, and up to 10x faster for serialization.
@Swatinem Swatinem mentioned this pull request Oct 21, 2024
Merged
Swatinem added a commit to codecov/shared that referenced this pull request Oct 21, 2024
@Swatinem
Switch to orjson for Report de/serialization
93bf100 
This is the same as codecov/worker#772, and for the same reasons:
`orjson` is advertised as being 3x faster for deserialization, and up to 10x faster for serialization.
github-merge-queue bot pushed a commit to codecov/shared that referenced this pull request Oct 21, 2024
@Swatinem
Switch to orjson for Report de/serialization (#401)
0d27239 
This is the same as codecov/worker#772, and for the same reasons:
`orjson` is advertised as being 3x faster for deserialization, and up to 10x faster for serialization.
matt-codecov pushed a commit to codecov/umbrella that referenced this pull request Apr 25, 2025
@Swatinem
Switch to orjson for Report de/serialization (#401)
03a9fef 
This is the same as codecov/worker#772, and for the same reasons:
`orjson` is advertised as being 3x faster for deserialization, and up to 10x faster for serialization.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@giovanni-guidini giovanni-guidini giovanni-guidini approved these changes

Assignees

@Swatinem Swatinem

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Swatinem @giovanni-guidini
0