ankerl::nanobench::Bench Class Reference

Main entry point to nanobench's benchmarking facility. More...

#include <nanobench.h>

Collaboration diagram for ankerl::nanobench::Bench:

Public Member Functions
	Bench ()
	Creates a new benchmark for configuration and running of benchmarks. More...
	Bench (Bench &&other)
Bench &	operator= (Bench &&other)
	Bench (Bench const &other)
Bench &	operator= (Bench const &other)
	~Bench () noexcept
template<typename Op >
Bench &	run (char const *benchmarkName, Op &&op)
	Repeatedly calls op() based on the configuration, and performs measurements. More...
template<typename Op >
Bench &	run (std::string const &benchmarkName, Op &&op)
template<typename Op >
Bench &	run (Op &&op)
	Same as run(char const* benchmarkName, Op op), but instead uses the previously set name. More...
Bench &	title (char const *benchmarkTitle)
	Title of the benchmark, will be shown in the table header. More...
Bench &	title (std::string const &benchmarkTitle)
ANKERL_NANOBENCH(NODISCARD) std Bench &	name (char const *benchmarkName)
	Name of the benchmark, will be shown in the table row. More...
Bench &	name (std::string const &benchmarkName)
template<typename T >
ANKERL_NANOBENCH(NODISCARD) std Bench &	batch (T b) noexcept
	Sets the batch size. More...
	ANKERL_NANOBENCH (NODISCARD) double batch() const noexcept
Bench &	unit (char const *unit)
	Sets the operation unit. More...
Bench &	unit (std::string const &unit)
ANKERL_NANOBENCH(NODISCARD) std Bench &	timeUnit (std::chrono::duration< double > const &tu, std::string const &tuName)
	Sets the time unit to be used for the default output. More...
ANKERL_NANOBENCH(NODISCARD) std ANKERL_NANOBENCH(NODISCARD) std Bench &	output (std::ostream *outstream) noexcept
	Set the output stream where the resulting markdown table will be printed to. More...
ANKERL_NANOBENCH(NODISCARD) std Bench &	clockResolutionMultiple (size_t multiple) noexcept
	Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds. More...
	ANKERL_NANOBENCH (NODISCARD) size_t clockResolutionMultiple() const noexcept
Bench &	epochs (size_t numEpochs) noexcept
	Controls number of epochs, the number of measurements to perform. More...
	ANKERL_NANOBENCH (NODISCARD) size_t epochs() const noexcept
Bench &	maxEpochTime (std::chrono::nanoseconds t) noexcept
	Upper limit for the runtime of each epoch. More...
ANKERL_NANOBENCH(NODISCARD) std Bench &	minEpochTime (std::chrono::nanoseconds t) noexcept
	Minimum time each epoch should take. More...
ANKERL_NANOBENCH(NODISCARD) std Bench &	minEpochIterations (uint64_t numIters) noexcept
	Sets the minimum number of iterations each epoch should take. More...
	ANKERL_NANOBENCH (NODISCARD) uint64_t minEpochIterations() const noexcept
Bench &	epochIterations (uint64_t numIters) noexcept
	Sets exactly the number of iterations for each epoch. More...
	ANKERL_NANOBENCH (NODISCARD) uint64_t epochIterations() const noexcept
Bench &	warmup (uint64_t numWarmupIters) noexcept
	Sets a number of iterations that are initially performed without any measurements. More...
	ANKERL_NANOBENCH (NODISCARD) uint64_t warmup() const noexcept
Bench &	relative (bool isRelativeEnabled) noexcept
	Marks the next run as the baseline. More...
	ANKERL_NANOBENCH (NODISCARD) bool relative() const noexcept
Bench &	performanceCounters (bool showPerformanceCounters) noexcept
	Enables/disables performance counters. More...
	ANKERL_NANOBENCH (NODISCARD) bool performanceCounters() const noexcept
template<typename Arg >
ANKERL_NANOBENCH(NODISCARD) std Bench &	doNotOptimizeAway (Arg &&arg)
	Retrieves all benchmark results collected by the bench object so far. More...
template<typename T >
Bench &	complexityN (T b) noexcept
	ANKERL_NANOBENCH (NODISCARD) double complexityN() const noexcept
std::vector< BigO >	complexityBigO () const
template<typename Op >
BigO	complexityBigO (char const *name, Op op) const
	Calculates bigO for a custom function. More...
template<typename Op >
BigO	complexityBigO (std::string const &name, Op op) const
Bench &	render (char const *templateContent, std::ostream &os)
Bench &	render (std::string const &templateContent, std::ostream &os)
Bench &	config (Config const &benchmarkConfig)
	ANKERL_NANOBENCH (NODISCARD) Config const &config() const noexcept
template<typename T >
Bench &	batch (T b) noexcept
template<typename Arg >
Bench &	doNotOptimizeAway (Arg &&arg)

Private Attributes
Config	mConfig {}
std::vector< Result >	mResults {}

Detailed Description

Main entry point to nanobench's benchmarking facility.

It holds configuration and results from one or more benchmark runs. Usually it is used in a single line, where the object is constructed, configured, and then a benchmark is run. E.g. like this:

ankerl::nanobench::Bench().unit("byte").batch(1000).run("random fluctuations", [&] {
    // here be the benchmark code
});

In that example Bench() constructs the benchmark, it is then configured with unit() and batch(), and after configuration a benchmark is executed with run(). Once run() has finished, it prints the result to std::cout. It would also store the results in the Bench instance, but in this case the object is immediately destroyed so it's not available any more.

Definition at line 616 of file nanobench.h.

Constructor & Destructor Documentation

Bench() [1/3]

ankerl::nanobench::Bench::Bench

(

)

Creates a new benchmark for configuration and running of benchmarks.

Bench() [2/3]

ankerl::nanobench::Bench::Bench

(

Bench &&

other

)

Bench() [3/3]

ankerl::nanobench::Bench::Bench

(

Bench const &

other

)

~Bench()

ankerl::nanobench::Bench::~Bench ( )

noexcept

Member Function Documentation

ANKERL_NANOBENCH() [1/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [2/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [3/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const &

noexcept

ANKERL_NANOBENCH() [4/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [5/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [6/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [7/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [8/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [9/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

ANKERL_NANOBENCH() [10/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const

noexcept

batch() [1/2]

template<typename T >

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::batch ( T  b )

noexcept

Sets the batch size.

E.g. number of processed byte, or some other metric for the size of the processed data in each iteration. If you benchmark hashing of a 1000 byte long string and want byte/sec as a result, you can specify 1000 as the batch size.

Template Parameters

T	Any input type is internally cast to double.

Parameters

b	batch size

Here is the caller graph for this function:

batch() [2/2]

template<typename T >

Bench& ankerl::nanobench::Bench::batch ( T  b )

noexcept

Definition at line 1207 of file nanobench.h.

clockResolutionMultiple()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::clockResolutionMultiple ( size_t  multiple )

noexcept

Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds.

This is the main trick nanobech to be so fast: we find out how accurate the clock is, then run the benchmark only so often that the clock's accuracy is good enough for accurate measurements.

The default is to run one epoch for 1000 times the clock resolution. So for 20ns resolution and 11 epochs, this gives a total runtime of

To be precise, nanobench adds a 0-20% random noise to each evaluation. This is to prevent any aliasing effects, and further improves accuracy.

Total runtime will be higher though: Some initial time is needed to find out the target number of iterations for each epoch, and there is some overhead involved to start & stop timers and calculate resulting statistics and writing the output.

Parameters

multiple

Target number of times of clock resolution. Usually 1000 is a good compromise between runtime and accuracy.

complexityBigO() [1/3]

std::vector<BigO> ankerl::nanobench::Bench::complexityBigO

(

)

const

Calculates Big O of the results with all preconfigured complexity functions. Currently these complexity functions are fitted into the benchmark results:

, , , , , .

If we e.g. evaluate the complexity of std::sort, this is the result of std::cout << bench.complexityBigO():

|   coefficient |   err% | complexity
|--------------:|-------:|------------
|   5.08935e-09 |   2.6% | O(n log n)
|   6.10608e-08 |   8.0% | O(n)
|   1.29307e-11 |  47.2% | O(n^2)
|   2.48677e-15 |  69.6% | O(n^3)
|   9.88133e-06 | 132.3% | O(log n)
|   5.98793e-05 | 162.5% | O(1)

So in this case provides the best approximation.

embed:rst
See the tutorial :ref:`asymptotic-complexity` for details.

Returns

Evaluation results, which can be printed or otherwise inspected.

Here is the caller graph for this function:

complexityBigO() [2/3]

template<typename Op >

BigO ankerl::nanobench::Bench::complexityBigO	(	char const *	name,
		Op	op
	)		const

Calculates bigO for a custom function.

E.g. to calculate the mean squared error for , which is not part of the default set of complexityBigO(), you can do this:

auto logLogN = bench.complexityBigO("O(log log n)", [](double n) {
    return std::log2(std::log2(n));
});

The resulting mean squared error can be printed with std::cout << logLogN. E.g. it prints something like this:

2.46985e-05 * O(log log n), rms=1.48121

Template Parameters

Op	Type of mapping operation.

Parameters

name	Name for the function, e.g. "O(log log n)"
op	Op's operator() maps a double with the desired complexity function, e.g. log2(log2(n)).

Returns

BigO Error calculation, which is streamable to std::cout.

Definition at line 1195 of file nanobench.h.

Here is the call graph for this function:

complexityBigO() [3/3]

template<typename Op >

BigO ankerl::nanobench::Bench::complexityBigO	(	std::string const &	name,
		Op	op
	)		const

Definition at line 1200 of file nanobench.h.

Here is the call graph for this function:

complexityN()

template<typename T >

Bench & ankerl::nanobench::Bench::complexityN ( T  b )

noexcept

embed:rst

Sets N for asymptotic complexity calculation, so it becomes possible to calculate `Big O
<https://en.wikipedia.org/wiki/Big_O_notation>`_ from multiple benchmark evaluations.

Use :cpp:func:`ankerl::nanobench::Bench::complexityBigO` when the evaluation has finished. See the tutorial
:ref:`asymptotic-complexity` for details.

Template Parameters

T	Any type is cast to double.

Parameters

b	Length of N for the next benchmark run, so it is possible to calculate bigO.

Definition at line 1214 of file nanobench.h.

Here is the caller graph for this function:

config()

Bench& ankerl::nanobench::Bench::config

(

Config const &

benchmarkConfig

)

doNotOptimizeAway() [1/2]

template<typename Arg >

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::doNotOptimizeAway

(

Arg &&

arg

)

Retrieves all benchmark results collected by the bench object so far.

Each call to run() generates a Result that is stored within the Bench instance. This is mostly for advanced users who want to see all the nitty gritty detials.

Returns

All results collected so far.

embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::doNotOptimizeAway`.

doNotOptimizeAway() [2/2]

template<typename Arg >

Bench& ankerl::nanobench::Bench::doNotOptimizeAway

(

Arg &&

arg

)

Definition at line 1221 of file nanobench.h.

Here is the call graph for this function:

epochIterations()

Bench& ankerl::nanobench::Bench::epochIterations ( uint64_t  numIters )

noexcept

Sets exactly the number of iterations for each epoch.

Ignores all other epoch limits. This forces nanobench to use exactly the given number of iterations for each epoch, not more and not less. Default is 0 (disabled).

Parameters

numIters

Exact number of iterations to use. Set to 0 to disable.

Here is the caller graph for this function:

epochs()

Bench& ankerl::nanobench::Bench::epochs ( size_t  numEpochs )

noexcept

Controls number of epochs, the number of measurements to perform.

The reported result will be the median of evaluation of each epoch. The higher you choose this, the more deterministic the result be and outliers will be more easily removed. Also the err% will be more accurate the higher this number is. Note that the err% will not necessarily decrease when number of epochs is increased. But it will be a more accurate representation of the benchmarked code's runtime stability.

Choose the value wisely. In practice, 11 has been shown to be a reasonable choice between runtime performance and accuracy. This setting goes hand in hand with minEpocIterations() (or minEpochTime()). If you are more interested in median runtime, you might want to increase epochs(). If you are more interested in mean runtime, you might want to increase minEpochIterations() instead.

Parameters

numEpochs

Number of epochs.

Here is the caller graph for this function:

maxEpochTime()

Bench& ankerl::nanobench::Bench::maxEpochTime ( std::chrono::nanoseconds  t )

noexcept

Upper limit for the runtime of each epoch.

As a safety precausion if the clock is not very accurate, we can set an upper limit for the maximum evaluation time per epoch. Default is 100ms. At least a single evaluation of the benchmark is performed.

See also

minEpochTime(), minEpochIterations()

Parameters

t	Maximum target runtime for a single epoch.

minEpochIterations()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::minEpochIterations ( uint64_t  numIters )

noexcept

Sets the minimum number of iterations each epoch should take.

Default is 1, and we rely on clockResolutionMultiple(). If the err% is high and you want a more smooth result, you might want to increase the minimum number or iterations, or increase the minEpochTime().

See also

minEpochTime(), maxEpochTime(), minEpochIterations()

Parameters

numIters

Minimum number of iterations per epoch.

Here is the caller graph for this function:

minEpochTime()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::minEpochTime ( std::chrono::nanoseconds  t )

noexcept

Minimum time each epoch should take.

Default is zero, so we are fully relying on clockResolutionMultiple(). In most cases this is exactly what you want. If you see that the evaluation is unreliable with a high err%, you can increase either minEpochTime() or minEpochIterations().

See also

maxEpochTime(), minEpochIterations()

Parameters

t	Minimum time each epoch should take.

Here is the caller graph for this function:

name() [1/2]

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::name

(

char const *

benchmarkName

)

Name of the benchmark, will be shown in the table row.

Here is the caller graph for this function:

name() [2/2]

Bench& ankerl::nanobench::Bench::name

(

std::string const &

benchmarkName

)

operator=() [1/2]

Bench& ankerl::nanobench::Bench::operator=

(

Bench &&

other

)

operator=() [2/2]

Bench& ankerl::nanobench::Bench::operator=

(

Bench const &

other

)

output()

ANKERL_NANOBENCH (NODISCARD) std ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::output ( std::ostream *  outstream )

noexcept

Set the output stream where the resulting markdown table will be printed to.

The default is &std::cout. You can disable all output by setting nullptr.

Parameters

outstream

Pointer to output stream, can be nullptr.

performanceCounters()

Bench& ankerl::nanobench::Bench::performanceCounters ( bool  showPerformanceCounters )

noexcept

Enables/disables performance counters.

On Linux nanobench has a powerful feature to use performance counters. This enables counting of retired instructions, count number of branches, missed branches, etc. On default this is enabled, but you can disable it if you don't need that feature.

Parameters

showPerformanceCounters

True to enable, false to disable.

relative()

Bench& ankerl::nanobench::Bench::relative ( bool  isRelativeEnabled )

noexcept

Marks the next run as the baseline.

Call relative(true) to mark the run as the baseline. Successive runs will be compared to this run. It is calculated by

• 100% means it is exactly as fast as the baseline
• >100% means it is faster than the baseline. E.g. 200% means the current run is twice as fast as the baseline.
• <100% means it is slower than the baseline. E.g. 50% means it is twice as slow as the baseline.

See the tutorial section "Comparing Results" for example usage.

Parameters

isRelativeEnabled

True to enable processing

render() [1/2]

Bench& ankerl::nanobench::Bench::render	(	char const *	templateContent,
		std::ostream &	os
	)

embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::render`.

render() [2/2]

Bench& ankerl::nanobench::Bench::render	(	std::string const &	templateContent,
		std::ostream &	os
	)

run() [1/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run	(	char const *	benchmarkName,
		Op &&	op
	)

Repeatedly calls op() based on the configuration, and performs measurements.

This call is marked with noinline to prevent the compiler to optimize beyond different benchmarks. This can have quite a big effect on benchmark accuracy.

embed:rst
.. note::

  Each call to your lambda must have a side effect that the compiler can't possibly optimize it away. E.g. add a result to an
  externally defined number (like `x` in the above example), and finally call `doNotOptimizeAway` on the variables the compiler
  must not remove. You can also use :cpp:func:`ankerl::nanobench::doNotOptimizeAway` directly in the lambda, but be aware that
  this has a small overhead.

Template Parameters

Op	The code to benchmark.

Definition at line 1183 of file nanobench.h.

run() [2/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run

(

Op &&

op

)

Same as run(char const* benchmarkName, Op op), but instead uses the previously set name.

Template Parameters

Op	The code to benchmark.

Definition at line 1161 of file nanobench.h.

Here is the call graph for this function:

run() [3/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run	(	std::string const &	benchmarkName,
		Op &&	op
	)

Definition at line 1189 of file nanobench.h.

timeUnit()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::timeUnit	(	std::chrono::duration< double > const &	tu,
		std::string const &	tuName
	)

Sets the time unit to be used for the default output.

Nanobench defaults to using ns (nanoseconds) as output in the markdown. For some benchmarks this is too coarse, so it is possible to configure this. E.g. use timeUnit(1ms, "ms") to show ms/op instead of ns/op.

Parameters

tu	Time unit to display the results in, default is 1ns.
tuName	Name for the time unit, default is "ns"

title() [1/2]

Bench& ankerl::nanobench::Bench::title

(

char const *

benchmarkTitle

)

Title of the benchmark, will be shown in the table header.

Changing the title will start a new markdown table.

Parameters

benchmarkTitle

The title of the benchmark.

title() [2/2]

Bench& ankerl::nanobench::Bench::title

(

std::string const &

benchmarkTitle

)

unit() [1/2]

Bench& ankerl::nanobench::Bench::unit

(

char const *

unit

)

Sets the operation unit.

Defaults to "op". Could be e.g. "byte" for string processing. This is used for the table header, e.g. to show ns/byte. Use singular (byte, not bytes). A change clears the currently collected results.

Parameters

unit	The unit name.

Here is the caller graph for this function:

unit() [2/2]

Bench& ankerl::nanobench::Bench::unit

(

std::string const &

unit

)

warmup()

Bench& ankerl::nanobench::Bench::warmup ( uint64_t  numWarmupIters )

noexcept

Sets a number of iterations that are initially performed without any measurements.

Some benchmarks need a few evaluations to warm up caches / database / whatever access. Normally this should not be needed, since we show the median result so initial outliers will be filtered away automatically. If the warmup effect is large though, you might want to set it. Default is 0.

Parameters

numWarmupIters

Number of warmup iterations.

Member Data Documentation

mConfig

Config ankerl::nanobench::Bench::mConfig {}

private

Definition at line 971 of file nanobench.h.

mResults

std::vector<Result> ankerl::nanobench::Bench::mResults {}

private

Definition at line 972 of file nanobench.h.

---

The documentation for this class was generated from the following file:

• /work/src/bench/nanobench.h