[sferik]

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against master and the branch side by side, in alternating worker processes, and fails the build if the branch is slower or allocates more. It judges CPU time and instruction counts with confidence intervals, and counts allocations exactly. When something regresses, it profiles both versions with StackProf and says where the extra time went.

There are 44 benchmarks, and most of them stay away from the server on purpose, so that what's timed is the driver and not PostgreSQL. Results are fetched once and then read in memory with values, getvalue, each (with string and symbol keys), each_row, tuple, column_values and the field name and type lookups. The type maps decode the same 100-row result through BasicTypeMapForResults in text and binary format and through a TypeMapByColumn, and BasicTypeMapForQueries picks encoders for a mixed set of parameters and feeds embed_params. The coders get benchmarks of their own, 100 fixed values a call: the Integer, Float, Date and both Timestamp text decoders (with the recent work on those in mind), integer and quoted-string arrays, Bytea, the matching encoders and the binary Int8, Float8 and TimestampUtc coders. Then there's escape_string, escape_literal, quote_ident and the bytea round trip, the CopyRow encoder and decoder, and parse_connect_args with a URL, a keyword string and a Hash. The rest do go to a server: exec, exec_params, exec_prepared, 100 rows with and without a result type map, and COPY in and out with coders. They use the duel-postgresql plugin, which starts a PostgreSQL server for each worker on a Unix socket and also judges each benchmark on the queries and rows the server ran.

The tricky part was pg being the plugin's own dependency. duel-postgresql needs a released pg, and it used to require it as .duel.rb loaded, before Duel loaded the version. So every worker was quietly measuring pg 1.7.0 from the bundle instead of either version. I checked this with a probe that wrote out each worker's $LOADED_FEATURES. I fixed that in duel-postgresql 0.1.4, which requires pg only once the worker's version has loaded, so .duel.rb needs no workaround. With it, the probe showed each worker with its own pg.rb and pg_ext.bundle and nothing else. The released pg also broke rake spec once it was in the bundle, because its fat binary gem has a lib/3.4/pg_ext that pg.rb tries first. So Duel lives in an optional duel group in the Gemfile, with the released pg as the source gem, and bundle install and the specs are exactly as they were unless you opt in. With the group installed the specs still pass (1163 examples, 0 failures).

A full run takes 20 to 25 seconds on my laptop, and the Duel job took about two and a half minutes on GitHub Actions, most of it setup and compiling both versions.

The Duel job only reads. The comment on a pull request comes from a second workflow, perf-comment.yml, which runs none of the pull request's code, only duel comment from Duel's gem, and hides Duel's earlier comments as outdated so the newest one is the one in view. If a change is slower on purpose, a duel: accept label lets it through, and adding it runs the Duel job again.

[larskanis]

Thank you for your proposal! It is interesting. Linux is the platform, which I use the most, so that I'm propably aware of performance changes. But not so much on Windows and Macos. Does Duel work there too?

[sferik]

Thank you for your proposal! It is interesting.

Thanks for considering it!

Linux is the platform, which I use the most, so that I'm propably aware of performance changes. But not so much on Windows and Macos. Does Duel work there too?

Duel runs on Linux and macOS. It doesn't run on Windows, because it controls its workers through Unix sockets and process groups. The Perf job here runs on Ubuntu, so it would catch a regression in the shared C code (the decoders, type maps, COPY and so on), but not one that only shows up in a macOS code path, but I could add that to the CI matrix. I’d like to add Windows support to Duel in the future, but that would be a major undertaking.