Skip to content

Commit

Permalink
Merge pull request #2 from jbowes/hashi-bench
Browse files Browse the repository at this point in the history
Add benchmarks for hashicorp, too
  • Loading branch information
jbowes authored Sep 7, 2021
2 parents 63ad67b + 793ed7b commit 12eee94
Show file tree
Hide file tree
Showing 8 changed files with 134 additions and 14 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
bench/ours.txt
bench/masterminds.txt
bench/hashicorp.txt
35 changes: 35 additions & 0 deletions BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
Compared to [Masterminds/semver](https://github.com/Masterminds/semver):
```
go run golang.org/x/perf/cmd/benchstat -geomean masterminds.txt ours.txt
name old time/op new time/op delta
Expand Down Expand Up @@ -38,4 +39,38 @@ ParseConstraint/wildcard-8 14.0 ± 0% 4.0 ± 0% -71.43% (p=0.000 n=
ParseConstraint/range-8 22.0 ± 0% 4.0 ± 0% -81.82% (p=0.000 n=10+10)
ParseConstraint/union-8 21.0 ± 0% 7.0 ± 0% -66.67% (p=0.000 n=10+10)
[Geo mean] 8.96 3.51 -60.81%
```

Compared to [hashicorp/go-version](https://github.com/hashicorp-go-version) (note that `go-version` does not have feature parity):
```
go run golang.org/x/perf/cmd/benchstat -geomean hashicorp.txt ours.txt
name old time/op new time/op delta
Parse/simple-8 1.10µs ± 2% 0.06µs ± 7% -94.09% (p=0.000 n=10+9)
Parse/pre-8 1.26µs ± 3% 0.18µs ± 4% -85.57% (p=0.000 n=10+9)
Parse/build-8 1.35µs ±13% 0.10µs ± 6% -92.63% (p=0.000 n=10+9)
Parse/pre+build-8 1.59µs ± 3% 0.24µs ±12% -84.61% (p=0.000 n=9+9)
ParseConstraint/equal-8 2.14µs ± 2% 0.22µs ± 9% -89.94% (p=0.000 n=9+10)
ParseConstraint/tilde-8 2.64µs ± 6% 0.21µs ± 3% -91.94% (p=0.000 n=10+8)
ParseConstraint/range-8 1.28µs ± 4% 0.25µs ±13% -80.75% (p=0.000 n=10+10)
[Geo mean] 1.55µs 0.19µs -87.67%
name old alloc/op new alloc/op delta
Parse/simple-8 539B ± 0% 96B ± 0% -82.19% (p=0.002 n=8+10)
Parse/pre-8 539B ± 0% 148B ± 0% -72.54% (p=0.002 n=8+10)
Parse/build-8 539B ± 0% 101B ± 0% -81.26% (p=0.002 n=8+10)
Parse/pre+build-8 539B ± 0% 160B ± 0% -70.30% (p=0.000 n=10+10)
ParseConstraint/equal-8 1.06kB ± 0% 0.30kB ± 0% -71.19% (p=0.000 n=10+10)
ParseConstraint/tilde-8 1.05kB ± 0% 0.30kB ± 0% -70.95% (p=0.000 n=10+10)
ParseConstraint/range-8 113B ± 1% 304B ± 0% +169.98% (p=0.000 n=10+10)
[Geo mean] 521B 220B -57.79%
name old allocs/op new allocs/op delta
Parse/simple-8 5.00 ± 0% 1.00 ± 0% -80.00% (p=0.000 n=10+10)
Parse/pre-8 5.00 ± 0% 4.00 ± 0% -20.00% (p=0.000 n=10+10)
Parse/build-8 5.00 ± 0% 2.00 ± 0% -60.00% (p=0.000 n=10+10)
Parse/pre+build-8 5.00 ± 0% 5.00 ± 0% ~ (all equal)
ParseConstraint/equal-8 11.0 ± 0% 4.0 ± 0% -63.64% (p=0.000 n=10+10)
ParseConstraint/tilde-8 10.0 ± 0% 4.0 ± 0% -60.00% (p=0.000 n=10+10)
ParseConstraint/range-8 5.00 ± 0% 4.00 ± 0% -20.00% (p=0.000 n=10+10)
[Geo mean] 6.18 3.51 -43.13%
```
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@
and testing.

There are many semver packages. This one aims to be the fastest at parsing
and comparing values, with low memory usage. On average, this package is over
and comparing values, with low memory usage. On average, this package is roughly
ten times faster at parsing versions and constraints than the popular
`Masterminds/semver` package. View more stats in the [benchmarks][bench].
`Masterminds/semver` and `hashicorp-go-version` packages. View more stats in
the [benchmarks][bench].

Versions can be compared with one another to determine which is newer.
Constraints specify inclusions and exclusions of semver ranges that a given
Expand Down
40 changes: 29 additions & 11 deletions bench/Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
benchstat: ours.txt masterminds.txt
benchstat: benchstat-masterminds benchstat-hashicorp

benchstat-hashicorp: ours.txt hashicorp.txt
go run golang.org/x/perf/cmd/benchstat -geomean hashicorp.txt ours.txt

benchstat-masterminds: ours.txt masterminds.txt
go run golang.org/x/perf/cmd/benchstat -geomean masterminds.txt ours.txt

BENCH=go test -bench . . | grep "^Bench"
Expand All @@ -18,13 +23,26 @@ ours.txt:

masterminds.txt:
touch $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
$(BENCH) >> $@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@
cd masterminds && $(BENCH) >> ../$@

hashicorp.txt:
touch $@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
cd hashicorp && $(BENCH) >> ../$@
1 change: 1 addition & 0 deletions bench/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ go 1.17

require (
github.com/Masterminds/semver/v3 v3.1.1
github.com/hashicorp/go-version v1.3.0
golang.org/x/perf v0.0.0-20210220033136-40a54f11e909
)
2 changes: 2 additions & 0 deletions bench/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ github.com/gonum/internal v0.0.0-20181124074243-f884aa714029/go.mod h1:Pu4dmpkhS
github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9/go.mod h1:XA3DeT6rxh2EAE789SSiSJNqxPaC0aE9J8NTOI0Jo/A=
github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9/go.mod h1:0EXg4mc1CNP0HCqCz+K4ts155PXIlUywf0wqN+GfPZw=
github.com/googleapis/gax-go v0.0.0-20161107002406-da06d194a00e/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/hashicorp/go-version v1.3.0 h1:McDWVJIU/y+u1BRV06dPaLfLCaT7fUTJLp5r04x7iNw=
github.com/hashicorp/go-version v1.3.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down
62 changes: 62 additions & 0 deletions bench/hashicorp/hashicorp_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Copyright (c) 2021 James Bowes. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package hashicorp_test

import (
"testing"

"github.com/hashicorp/go-version"
)

// Benchmarks are defined as slices so they run in the same order every time,
// for easier visual comparison.

func BenchmarkParse(b *testing.B) {
bcs := []struct {
name string
in string
}{
{"simple", "1.2.3"},
{"pre", "1.2.3-beta"},
{"build", "1.2.3+build"},
{"pre+build", "1.2.3-rc.1+build.info"},
}

for _, bc := range bcs {
b.Run(bc.name, func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, _ = version.NewVersion(bc.in)
}
})
}
}

func BenchmarkParseConstraint(b *testing.B) {
bcs := []struct {
name string
in string
}{
{"equal", "=2.2"},
{"tilde", "~>2.2.0"},
{"range", ">=2.2.x, <3.2.0"},

// unsupported by go-version
// {"wildcard", "1.x"},
// {"caret", "^2.2.0"},
// {"union", "~2.2.0 || =3.3.0"},
}

for _, bc := range bcs {
b.Run(bc.name, func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, _ = version.NewConstraint(bc.in)
}
})
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package bench_test
package masterminds_test

import (
"testing"
Expand Down

0 comments on commit 12eee94

Please sign in to comment.