J.Benchmark

Benchmark is a basic comparison test tool

JVM technology makes it difficult to get absolute performance data

Benchmark allows to run several targets side by side, comparing their performance against each other

Let's try to benchmark 'add' element performance for List, Vector and Buffer

J.Benchmark(
 ("scala.List  ", () => { var v: List[Int]   = Nil;          for (i <- 1 <> 1000) v = i :: v; v.size }),
 ("scala.Vector", () => { var v: Vector[Int] = Vector.empty; for (i <- 1 <> 1000) v = v :+ i; v.size }),
 ("Val.Buffer",   () => { val v: Buffer[Int] = Buffer();     for (i <- 1 <> 1000) v += i;     v.size }),
)

// Output
Final Result. Total length is about 12 secs
--- ------------ ------- --- ------- --- ---------
Num Name         Ops/Sec %   Memory  %   Avg Value
--- ------------ ------- --- ------- --- ---------
1   scala.List   144.6k  67  36.7kB  28  1000.0
2   scala.Vector 45.6k   21  128.6kB 100 1000.0
3   Val.Buffer   213.0k  100 22.1kB  17  1000.0
--- ------------ ------- --- ------- --- ---------

The results present three values:

• 'Ops/Sec' is the number of times the target executed per second
• 'Memory' is the average memory increase for one execution
• 'Avg Value' is the average of target returns for validity check. Naturally, all targets should arrive to the same value in different ways

Note. Absolute values are less informative than following them percentages of the largest value

From the above example, the following conclusions can be made:

• List is significantly and Vector is five times slower than Buffer
• Vector is seriously heavier on memory

Comparison to JMH

JMH targets absolute precision benchmarking, where plus/minus 1 percent is a big deal.

J.Benchmark is used to compare targets, with performance differences in tens and hundreds percent, where 1-2 percent is a margin of error. In the example above, we do not care about 2 percent error, when establishing that scala.List is about 300% faster than scala.Vector.

Precision Test

To get a feeling on how precise the tests are, we can run the above example with each line trippled:

J.Benchmark(
  ("scala.List  ", () => { var v: List[Int]   = Nil;          for (i <- 1 <> 1000) v = i :: v; v.size }),
  ("scala.List  ", () => { var v: List[Int]   = Nil;          for (i <- 1 <> 1000) v = i :: v; v.size }),
  ("scala.List  ", () => { var v: List[Int]   = Nil;          for (i <- 1 <> 1000) v = i :: v; v.size }),
  ("scala.Vector", () => { var v: Vector[Int] = Vector.empty; for (i <- 1 <> 1000) v = v :+ i; v.size }),
  ("scala.Vector", () => { var v: Vector[Int] = Vector.empty; for (i <- 1 <> 1000) v = v :+ i; v.size }),
  ("scala.Vector", () => { var v: Vector[Int] = Vector.empty; for (i <- 1 <> 1000) v = v :+ i; v.size }),
  ("Val.Buffer",   () => { val v: Buffer[Int] = Buffer();     for (i <- 1 <> 1000) v += i;     v.size }),
  ("Val.Buffer",   () => { val v: Buffer[Int] = Buffer();     for (i <- 1 <> 1000) v += i;     v.size }),
  ("Val.Buffer",   () => { val v: Buffer[Int] = Buffer();     for (i <- 1 <> 1000) v += i;     v.size }),
)

// Output
Final Result. Total length is about 12 secs
--- ------------ ------- --- ------- --- ---------
Num Name         Ops/Sec %   Memory  %   Avg Value
--- ------------ ------- --- ------- --- ---------
1   scala.List   143.4k  54  41.5kB  30  1000.0
2   scala.List   142.7k  54  34.5kB  25  1000.0
3   scala.List   144.1k  54  35.7kB  26  1000.0
4   scala.Vector 46.3k   17  135.0kB 100 1000.0
5   scala.Vector 46.2k   17  132.2kB 97  1000.0
6   scala.Vector 45.3k   17  134.7kB 99  1000.0
7   Val.Buffer   263.7k  100 21.2kB  15  1000.0
8   Val.Buffer   253.4k  96  23.2kB  17  1000.0
9   Val.Buffer   263.0k  99  20.0kB  14  1000.0
--- ------------ ------- --- ------- --- ---------