Add a subsection for microbenchmarks to the profiling page.

[Ivorforce]

• Addresses Add custom test command line option godot-proposals#12319 (comment)

We currently have no documented way of performing microbenchmarks.

There is no perfectly suitable page to add this to, but I think the profiling page is a well suited candidate even though it's not technically profiling, because people are somewhat likely to find it here.
But I'm open to moving the information elsewhere!

[lawnjelly]

More things to mention (specially for c++):

• do the first test runs without timing
• first run to commit pages that have been allocated
• first run to warm the cache
• run the main loop multiple times, the smallest measurement is often most useful
• don't run anything else on your PC while benchmarking (especially e.g. browsers)

These are all very sensitive to other things using your OS.

As an alternative for microbenching you can use RDTSC / RDTSCP (forces processor to finish previous instructions) to measure CPU clock ticks. This used to be easier, but on modern multicore PCs which scale speeds there's more considerations, but most have invariant TSC on the CPU regardless of stepping (check this though for your CPU).

(in fact I must revisit using RDTSC for a recent couple of benchmarks in PRs)

More possibles:

• if you run the same data each loop, it won't stress the cache, but it may do e.g. perfect branch prediction
• good practice often to create a random data set before benchmarking
• the size of the data set will determine how much cache thrashing will go on
• might be worth mentioning websites dedicated to benchmarking (sometimes you can do the benchmark online, with pre-built helpers, especially for code which is localised and doesn't need to pull in the rest of Godot)
• one I discovered today - don't use rand() within a benchmark, it's not O(1), and might do locking for threads

Obviously we shouldn't need to be a full reference, but we should probably include some of the major pitfalls, otherwise we'll get new benchmarkers and have to explain the same artifacts each time. And emphasize how difficult it can be to get correct results, I regularly get artifacts even after having done benchmarks here and there for many years 😁 .

[Ivorforce]

More things to mention (specially for c++):

• do the first test runs without timing
• first run to commit pages that have been allocated
• first run to warm the cache
• run the main loop multiple times, the smallest measurement is often most useful
• don't run anything else on your PC while benchmarking (especially e.g. browsers)

Those are all good tips, but I deliberately didn't include any in my PR.

If we wanted to be a "best benchmarking practices" reference, we'd have to include a lot more info, for example about how to avoid dead code elimination or why cold checks are often more important than hot checks. I don't think this is in scope for the docs, so I opted for saying "hey be careful about it and read a guide somewhere" instead.

Obviously we shouldn't need to be a full reference, but we should probably include some of the major pitfalls, otherwise we'll get new benchmarkers and have to explain the same artifacts each time. And emphasize how difficult it can be to get correct results, I regularly get artifacts even after having done benchmarks here and there for many years 😁 .

I'd rather find a blessed guide to link to somewhere and not give any tips here at all. I don't have one at hand, unfortunately, but it sounds like you might be interested in writing one 😄

[lawnjelly]

but it sounds like you might be interested in writing one

Absolutely not 🤣
Yeah, agreed maybe we can just link to some good benchmarking websites.

https://quick-bench.com
https://github.com/google/benchmark

Unfortunately most tutorials seem to be directing you to use pre-existing libraries / tools for benchmarking instead of doing it from scratch, which is a bit sad, but maybe we'll find one.

Maybe it's because it's so easy to get wrong (even for the pros!), beginners get steered away from it.

[Ivorforce]

I had an AI research credible stable resources we can link, found 3:

• https://www.brendangregg.com/activebenchmarking.html: High level overview, good to link at the top. Site has existed for 15+ years so unlikely to go anywhere. Well known performance engineer.
• https://github.com/dendibakh/perf-book: An in-depth guide on performance analysis and testing.
• https://www.agner.org/optimize/: We all know this one, Agner Fog's guide on CPU optimization.

I've only heard of the last one, but the first two also look like incredible fits, so I think they are all worth linking. I'll adjust in a push.
Regarding your links, I'd rather avoid linking anything that's using libraries. Most benchmarks will probably be hand written and a library will be too much mental overhead for a single PR.

[lawnjelly]

Regarding your links, I'd rather avoid linking anything that's using libraries. Most benchmarks will probably be hand written and a library will be too much mental overhead for a single PR.

I mostly feel the same way, although I'm trying to begrudgingly recognise that for most people spending hours diagnosing artifacts in their benchmarks isn't their idea of fun (like it might be for you or I 😁 ), and I know some quite advanced programmers will still just use off the shelf libs rather than write their own, just because of the pitfalls.

I did check brendangregg's website earlier but didn't spot that particular article (most were quite in depth).

The Bakhvalov book I may have looked through before at some point, seems like a good one to link covering optimization and benchmarking. 👍

[Ivorforce]

If you think some people would prefer a library for it we could mention that and perhaps link one or two examples.

[lawnjelly]

Looks good.

Something may be worth adding that gdscript benchmarking can be difficult to use / not hugely enlightening when the cost of the function under test (e.g. c++) is very cheap in relation to the overheads of gdscript.

In many situations you get so much variation from the gdscript overheads, these might be 1000x as e.g. a c++ function that you are trying to optimize. But still I've seen people making PRs and trying to use a gdscript benchmark where it is clearly inappropriate.

Not quite sure how to word it.

[lawnjelly]

Although there's a few things we could add, this is fine to get started imo.

[AThousandShips]

LGTM