Calling method with splat arguments is slower than MRI

[gogainda]

Hi there,
I was comparing the results from fast_ruby repo using Truffle Ruby vs MRI.
I notice that this code shows different results. I am not talking about instructions per seconds but difference between 'slow' and 'fast' methods. Ideally it should be the same as in MRI

truffleruby 1.0.0-rc12, like ruby 2.4.4, GraalVM CE Native [x86_64-linux]
Warming up --------------------------------------
Function with single Array argument
                       827.269k i/100ms
Function with splat arguments
                       423.000  i/100ms
Calculating -------------------------------------
Function with single Array argument
                        743.500M (± 2.6%) i/s -      3.460B in   4.978265s
Function with splat arguments
                          4.775k (±28.8%) i/s -     22.419k in   5.055312s

Comparison:
Function with single Array argument: 743500303.4 i/s
Function with splat arguments:     4775.3 i/s - 155697.35x  slower

vs

ruby -v code/general/array-argument-vs-splat-arguments.rb
ruby 2.6.1p33 (2019-01-30 revision 66950) [x86_64-linux]
Warming up --------------------------------------
Function with single Array argument
                       121.426k i/100ms
Function with splat arguments
                         3.837k i/100ms
Calculating -------------------------------------
Function with single Array argument
                          3.070M (± 1.8%) i/s -     15.421M in   5.025579s
Function with splat arguments
                         46.596k (± 6.8%) i/s -    234.057k in   5.048882s

Comparison:
Function with single Array argument:  3069532.9 i/s
Function with splat arguments:    46596.0 i/s - 65.88x  slower

[nirvdrum]

Thanks for the report! It certainly looks like something odd is going on that we need to look into. As a minor quibble though, I think the method naming convention adopted in fast_ruby is misleading. I've raised this point and got buy in to go through and rename the methods. I just haven't found the time to do it yet. In the time since I originally filed the issue, MRI has gotten its own JIT and I expect to see the roles of "fast" and "slow" to change for some of these methods.

All of this is a long-winded way of saying that we shouldn't necessarily have the same performance profile as MRI. In an ideal world, there'd be no measurable difference between the two forms of method calls, so you could then make the choice of one over the other by taste, rather than compromising for performance. And in that regard, we've missed the mark, even if we're 240x faster than MRI for the single array argument case.

[eregon]

The interesting part in this benchmark, is whether M.func(*M::ITEMS) creates copies of the Array, and how many copies and how fast.

[aardvark179]

Yeah. In MRI the splatted array is simply passed as a Ruby array (after enabling copy on write), so it doesn't actually copy any of the elements. Our argument passing code turns it into a Java Object[] which means we have to copy all the arguments on every call.

[eregon]

MRI still seems to do at least one copy or walk, as it's >10x faster with ITEMS 10x smaller.

As a note, the original benchmark code is also slightly incorrect because args doesn't contain the same value in both cases.

I'm also unsure of how representative is this benchmark.
Do people often call methods with a rest argument with 10 000 arguments?
If we make it to a more realistic 10 elements, then the difference is much smaller, and we could easily optimize that one better.

OTOH, optimizing splats by passing a copy-on-write Ruby Array would be a more general optimization.
FWIW, it cannot be the original array. A different Array object is needed for semantics as shown by this code:

ITEMS = (1..10).to_a.freeze

def self.func(*args)
  args << -1
end

p func(*ITEMS) # => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, -1]

So this is a long way for me to say I'd rather optimize this if it happens in real-world Ruby code than just a micro-benchmark for the sake of it.
The conclusion that using a rest argument with a very large Array is slower holds on both MRI and TruffleRuby currently, and I think it's generally a good idea to pass large arrays as single arguments rather than splatting them.

@gogainda Do you think splatting a large Array is a common pattern in Ruby?
Do you know Ruby code using this?

[gogainda]

Don't think that this is common pattern, at least I haven't seen it in the wild. If you think that this benchmark doesn't represent real life application we can close this issue.

[nirvdrum]

I agree that handling smaller cases makes more sense. I suspect someone was looking at O(1) and O(n) and pumping the n value makes the resulting discrepancy easier to spot.

[gogainda]

Latest results (much better now, but still not comparable to mri implementation)

truffleruby 20.1.0-dev-c24c4074, like ruby 2.6.5, GraalVM CE Native [x86_64-linux]


Warming up --------------------------------------
Function with single Array argument
                         2.503k i/100ms
Function with splat arguments
                       145.000  i/100ms
Calculating -------------------------------------
Function with single Array argument
                         43.477M (± 4.2%) i/s -    194.701M in   4.711732s
Function with splat arguments
                          3.717k (±50.0%) i/s -     10.295k in   5.145532s

Comparison:
Function with single Array argument: 43477047.1 i/s
Function with splat arguments:     3717.3 i/s - 11695.95x  slower