processor cache and Visual C++

Question

processor cache and Visual C++

Batorek 1

Please provide advice on writing Visual C++ IEC/ISO 14882:1998 functions that would be stored in the processor cache.

0 comments

2 answers

Your answer

Answer 1

Hi @Batorek

Short version first: no C++ standard, including C++98 (ISO/IEC 14882:1998), gives you a way to pin a function into the processor cache. The CPU decides what lives in cache based on what's executed recently and how close the code is in memory. What you can do is write and build the code so the hot functions stay small and sit next to each other, which is what keeps them resident in practice.

The approach that gives the biggest effect in Visual C++:

Keep hot functions compact: no large local arrays, few branches, no exceptions in the hot path, and loops that walk memory sequentially.
Group them in memory with __declspec(code_seg("hot")) on each hot function so the linker places them together in one section, and mark the small ones __forceinline so calls disappear.
Build with /O2 /GL and link with /LTCG, then use Profile-Guided Optimization (/GENPROFILE, run a representative workload, then /USEPROFILE). PGO reorders your functions and basic blocks by actual execution frequency, which is the closest thing to "cache-aware layout" the toolchain offers.
For the data those functions touch, keep it contiguous and aligned (__declspec(align(64))) so each cache line carries useful bytes.

Are you targeting a specific CPU, and is your concern the instruction cache or the data your functions read? The answer changes a bit depending on which one is the bottleneck.

If this helped, please click Accept Answer so others with the same question can find it.

References: https://learn.microsoft.com/en-us/cpp/build/profile-guided-optimizations https://learn.microsoft.com/en-us/cpp/cpp/code-seg-declspec

Answer 2

Processor cache contents cannot be controlled directly from Visual C++ or ISO/IEC 14882:1998 C++ code. The practical approach is to write time-critical code so it is more likely to benefit from cache locality and to avoid patterns that increase cache misses and page faults.

Useful guidance:

Keep related data together. Data structures with good locality of reference reduce missed cache hits and page faults.
Prefer arrays over dynamically allocated linked lists when practical. Traversing linked lists can miss the cache or cause page faults because each link may be in a different memory location. In some cases, a simple array-based implementation is faster.
Be careful with hash tables that use dynamically allocated linked lists. These can perform substantially worse because of poor locality. An array-based hash table (closed hashing) can have superior performance.
Minimize dynamic allocation overhead in time-critical paths. For MFC code, CString uses dynamic allocation. A simple char array on the stack can be faster for some scenarios, and constant strings should use const char *.
If using CArray, size it up front. Use CArray::SetSize and specify growth behavior so repeated insertions do not cause frequent reallocations and copies, which can fragment memory and increase cache misses and page faults.
Avoid unnecessary memory overhead in list structures. CList is a doubly linked list. If a doubly linked list is not required, a singly linked list reduces pointer-update overhead and memory use, which also reduces opportunities for cache misses and page faults.
In parallel code, avoid false sharing. False sharing happens when separate tasks write to variables on the same cache line, causing repeated cache invalidation and reloads. One mitigation is to place frequently written variables on separate cache lines. When sharing data among tasks, concurrency::combinable is recommended because it creates thread-local variables in a way that makes false sharing less likely.
Measure before and after changes. Use Performance Monitor (perfmon.exe) to gather information about performance.

Also, cache correctness is different from cache performance. On multiprocessor systems, if shared values are accessed concurrently, use proper synchronization. The documented example shows using volatile in /volatile:ms mode or InterlockedExchange to ensure visibility and ordering between processors.