Forgot to mention, and this is tangentially related to my comments from yesterday:
A paper from 2020 showed that Cranelift was an order of magnitude faster than LLVM, while producing code that was approximately twice as slow on some benchmarks. Cranelift was still slower than the paper’s authors’ custom copy-and-patch JIT compiler, however.
Cranelift is itself written in Rust, making it possible to use as a benchmark to compare itself to LLVM. A full debug build of Cranelift itself using the Cranelift backend took 29.6 seconds on my computer, compared to 37.5 with LLVM (a reduction in wall-clock time of 20%).
Notes:
It’s easy to gloss over the “order of magnitude” part in the presence of concrete and current numbers mentioned later.
It’s actually “orders of magnitude” faster.
But the numbers only show a 20% speed increase!
The unattended reader will be left with the impression that Cranelift compiles 20% faster for a 2x slowdown. Some comments below the article confirms that.
What the article author missed (again) is that the biggest Cranelift wins come when used in release/optimized/multi-pass mode. I mention multi-pass because the author should have noticed that the (relatively old) 2020 research paper he linked to tested Cranelift twice, with one mode having the single-pass tag attached to it.
Any Rust user knows that slow builds (sometimes boringly so) are actually release builds. These are the builds where the slowness of LLVM optimizing passes is felt. And these are the builds where Cranelift shines, and is indeed orders of magnitude faster than LLVM.
The fact that Cranelift manages to build non-optimized binaries 20% faster than LLVM is actually impressively good for Cranelift, or impressively bad for LLVM, however you want to look at it.
And that is the problem with researches/authors with no direct field expertise. They can easily miss some very relevant subtleties, leading the readers to make grossly wrong conclusions.
Yeah, I’m no compiler engineer, but testing both release and debug builds is the minimum I’d do. That doesn’t even get into classes of optimizations, like loop unrolling, binary size, macros, or function inlining, which I also expect to be in a compiler comparison.
Forgot to mention, and this is tangentially related to my comments from yesterday:
Notes:
But the numbers only show a 20% speed increase!
The unattended reader will be left with the impression that Cranelift compiles 20% faster for a 2x slowdown. Some comments below the article confirms that.
What the article author missed (again) is that the biggest Cranelift wins come when used in release/optimized/multi-pass mode. I mention multi-pass because the author should have noticed that the (relatively old) 2020 research paper he linked to tested Cranelift twice, with one mode having the single-pass tag attached to it.
Any Rust user knows that slow builds (sometimes boringly so) are actually release builds. These are the builds where the slowness of LLVM optimizing passes is felt. And these are the builds where Cranelift shines, and is indeed orders of magnitude faster than LLVM.
The fact that Cranelift manages to build non-optimized binaries 20% faster than LLVM is actually impressively good for Cranelift, or impressively bad for LLVM, however you want to look at it.
And that is the problem with researches/authors with no direct field expertise. They can easily miss some very relevant subtleties, leading the readers to make grossly wrong conclusions.
Yeah, I’m no compiler engineer, but testing both release and debug builds is the minimum I’d do. That doesn’t even get into classes of optimizations, like loop unrolling, binary size, macros, or function inlining, which I also expect to be in a compiler comparison.