Enable parallel build #2040
Enable parallel build #2040
Conversation
Support for parallel compilation of *.erl file was dropped before 3.0 release. However, our tests for a project containing ~500 source files show substantial gain, lowering compilation time from 58 seconds to 18 on a MacBook Pro 15" (4 cores, 8 threads), and to just 10 seconds on Xeon-D machine.
This patch does two things:
1. it broadens the interface for the compiler module so that
non-first-file modules can possibly be parallelized. This is done by
dynamically switching on `[ListOfFiles]`, which remains sequential as
before, or `{[SeqPriority], [Parallel]}`, which divides regular files
between higher priority ones and those that can be in parallel
2. implements this mechanism in the rebar compiler, based on the erl
file digraph. If a file has an in-neighbour, it is depended on by
another file. The mechanism therefore makes it so all files that have
dependants get compiled in their strict relative sequential order
first, and then the undepended-on files get compiled together in
parallel.
By running:
./rebar3 ct --suite test/rebar_compile_SUITE.erl --case \
recompile_when_parse_transform_inline_changes --repeat 50
the previous iteration of this would rapidly fail, and this one succeeds
every time.
94201c8 to
9f81a57
Compare
|
Running this branch on Rebar3 itself had low impact on compile performance from scratch (deps cached, but not in _build):
The slight slowdown on a small-enough project might certainly be worth the gain on a large one. |
|
I like this one. Performance results: PS: for comparison, my naive version is real 0m14.290s. |
|
@tsloughter re: your comment on the other thread. I think things are not too bad considering there's a check for I had limited slow downs here as well compared to the massive speedup Max has reported, but we might want to check with smaller projects to find a cutoff point. |
|
Maybe it should not spawn any workers if the |
|
Now even cheap desktops have 16+ threads, and parallel build with 8 files (< NumSchedulers) benefits from using 8 workers. In fact, spawning a worker does not not cost much, spawn rate of 8,000 per second is fine on a laptop. |
|
After testing a bit more, I realised that there is also a bottleneck in init_dag, which is single-threaded, does some file i/o, and is quite slow as it performs full-scale *.erl file parsing (rebar_compiler_erl, line 242). == |
|
Got the OK from Tristan on IRC. |
This PR supercedes #2039, which had the following description:
This patch does two things on top of @max-au's PR:
[ListOfFiles], which remains sequential asbefore, or
{[SeqPriority], [Parallel]}, which divides regular files between higher priority ones and those that can be in parallelBy running:
the previous iteration of this PR would rapidly fail, and this one succeeds every time.