The reorder buffer (ROB) is a queue of instructions in program order that tracks the status of the instructions currently in the instruction window. Instructions are enqueued in program order after register renaming. These instructions can be in various states of execution as the instructions in the window are selected for execution. Completed instructions leave the reorder buffer in program order when committed. The capacity of the reorder buffer limits how far ahead in the instruction stream the processor can search for independent instructions. This has been steadily increasing over many processor generations.

The instruction window can also be limited by exhausting resources on the processor other than reorder buffer entries. In physical register file microarchitectures (Pentium 4, Intel Sandy Bridge and newer), the effective instruction window can also be limited by running out of physical registers for renaming.

In this article, I describe a microbenchmark to measure the size of the instruction window and present results for several processor microarchitectures. I also use the same microbenchmark to measure the number of available rename registers on PRF microarchitectures, and show some special cases where certain instructions are handled without consuming physical rename registers.

Measuring Instruction Window Size

Fig. 1: Microbenchmark example showing two iterations of the microbenchmark inner loop for ROB size of 4 instructions, with three instructions per load (left) and four instructions per load (right). Loop latency nearly doubles when enough nops are inserted so that the next load does not fit in the ROB.

The instruction window constrains how many instructions ahead of the most recent unfinished instruction a processor can look at to find instructions to execute in parallel. Thus, I can create a microbenchmark that measures the size of the instruction window if it has these two characteristics: It uses a suitable long-latency instruction to block instruction commit, and it is able to observe how many instructions ahead the processor is able to execute once instruction commit is stalled.

A good long-latency instruction to use is a load that misses the cache. A typical cache miss latency is greater than 200 clock cycles, enough time to fill the reorder buffer with other instructions. One way to get a single load to consistently miss in the cache is to do pointer chasing in an appropriately-initialized array, like the method used to measure cache system behaviours.

The method I used to detect how far ahead of a cache miss a processor can examine is to place another cache miss some number of instructions after a first cache miss. When the first cache miss is blocking instruction commit, if the second cache miss is within the instruction window, then the two cache misses are overlapped (memory-level parallelism), but if it is outside the instruction window, the two memory accesses are serialized. NOP is a good filler instruction in between the memory loads because it executes quickly, has no data dependencies, and occupies a reorder buffer entry.

Figure 1 shows an example of the behaviour of two inner-loop iterations of this microbenchmark for a reorder buffer size of 4. When there are only three instructions per load (mov, nop, nop, mov, nop, nop, ...), the processor is able to search ahead to the next mov while stalled on the first, so that two cache misses can be partially overlapped, leading to executing one iteration of the loop for every cache miss time (plus a few nops). When enough nops are inserted so that the next cache miss cannot fit into the instruction window (mov, nop, nop, nop, mov, nop, nop, nop, ...), the two cache misses are serialized and one loop iteration takes nearly twice as long, executing one iteration in the time of two cache misses (plus a few nops).

Reorder Buffer Capacity

Fig 2: Reorder buffer size for Ivy Brige (168 entries) and Lynnfield (128 entries).

Figure 2 shows the results of using the microbenchmark to measure the reorder buffer size of Intel’s Ivy Bridge and Lynnfield (similar to Nehalem) microarchitectures. The measured values (168 and 128, respectively) agree with the published numbers. The stepwise increase in execution time caused by serializing the two cache misses is clearly visible in the graphs. The slight slope upwards reflects the execution throughput of NOPs.

Figure 3 shows results for other microarchitectures, with the curves cropped to the region of interest, for clarity. Again, the measurements agree with published numbers for all of the microarchitectures.

Fig 3: Reorder buffer capacities for several other microarchitectures: Ivy Bridge (168), Sandy Bridge (168), Lynnfield (128), Northwood (126), Yorkfield (96), Palermo (72), and Coppermine (40).

Hyper-Threading

On processors with Hyper-Threading, the reorder buffer is partitioned between two threads. It is known that this partitioning is static (and it makes sense to partition statically anyway), so each thread gets half of the ROB entries. One interesting question is whether this static partitioning occurs only when both thread contexts are active, or whether it’s permanently partitioned whenever Hyper-Threading is enabled.

Tests on Lynnfield (1st generation Core i7) and Northwood (Pentium 4) show that this partitioning occurs only when both thread contexts are active. Two copies of the microbenchmark pinned to two thread contexts on the same core shows half the ROB capacity when both thread contexts are active, but full capacity when one thread context is idle. Good.

Physical Register File Size

Fig. 4: Measuring the size of the speculative portion of the physical register file. Northwood (around 115), Sandy Bridge and Ivy Bridge (around 130-135).

Measuring the reorder buffer size uses nop instructions to separate the memory loads. This is ideal because nop instructions consume reorder buffer entries, but not destination registers. On microarchitectures that rename registers using a physical register file (PRF), there are often fewer registers than reorder buffer entries (because some instructions don’t produce a register result, such as branches). If the filler instruction is changed from nop to one that produces a register result (ADD), I can measure the size of the speculative portion of the PRF.

Note that since the PRF holds both speculative and committed register values, I cannot measure the size of the entire PRF, only the portion holding speculative values. However, if we assume that published PRF sizes are correct, the difference would indicate the number of registers used for architectural state, which includes both x86 ISA-visible state as well as some extra architectural registers for internal use.

Figure 4 shows the results for microarchitectures that use a physical register file. The Pentium 4 Northwood appears to have 115 speculative PRF entries. Since the PRF size is expected to be 128 entries, that means 13 registers are used for non-speculative state. Sandy Bridge and Ivy Bridge appear to have 131. The noisy region between 118 and 131 registers hints at some non-ideal behaviour in reclaiming unused registers (in exchange for simpler hardware?), resulting in fewer registers being available for use some but not all of the time. Assuming the expected 160-entry PRF leaves 29 registers of non-speculative state. This seems somewhat high even considering that SNB/IVB are 64-bit (16 architectural integer registers) and Northwood is 32-bit (8 architectural integer registers).

Fig. 5: FP/SSE Physical Register File Size. Sandy Bridge and Ivy Bridge have 120 available speculative entries (32-bit mode). Northwood has 87.

The same experiment (Figure 5) can be repeated with a 128-bit SSE (or 256-bit AVX) instruction to measure the size of the floating-point and SSE PRF. As before, there are more registers used for non-speculative state than expected. Sandy Bridge and Ivy Bridge should have 144 registers (120 available for speculative state), and Northwood should have 128 registers (87 available for speculative state). Interestingly, there are 9 fewer speculative registers (111 vs. 120) in 64-bit mode than 32-bit, probably for YMM8-15 that are inaccessible in 32-bit mode. I can’t explain why so many registers are needed for non-speculative state (24 to 41), since the ISA-visible state mainly consists of 8 x87 floating-point registers, and 8 or 16 XMM or YMM registers.

x86 Zeroing Idiom and MOV Optimization

Fig. 6a: Ivy Bridge executes MOV instructions without consuming destination registers (limited by 168-entry ROB), while Sandy Bridge and Northwood do (limited by speculative portion of PRF).

A common x86 idiom for zeroing registers is to XOR a register with itself. Processors usually recognize this sequence and break the dependence on the source register value. In PRF microarchitectures, it may be possible to optimize this even further and point the renamer entry for that register to a special “zero” register, or augment the renamer with a special value indicating the register holds a value of zero, and not require any execution resources for the instruction. I can test for the zeroing idiom by repeating the above test using XOR reg, reg instructions and observing whether those instructions consume destination registers. Both Sandy Bridge and Ivy Bridge can zero registers without consuming destination registers, while Northwood does consume destination registers. (Graphs not shown)

Fig. 6b: Ivy Bridge MOV optimization for 256-bit MOVDQA instructions (128-bit MOVDQA for Northwood). Fewer registers are used, but there is something other than the ROB size that limits the instruction window.

Repeating the test with MOV instructions can show whether register to register moves are executed by manipulating pointers in the register renamer rather than copying values. This is a known new feature in Ivy Bridge. Figure 6 shows that it is indeed new in Ivy Bridge. Oddly, when the source and destination registers are the same, this optimization does not happen, nor is the operation treated as a NOP. Figure 6b shows the results for using 256-bit AVX MOVDQA. These look strange. It is also strange that Ivy Bridge’s instruction window is lower than the ROB size of 168 entries, as though registers were still consumed, though fewer.

Unresolved Puzzle

Fig. 7: Sandy Bridge executing interleaved 256-bit AVX and 64-bit addition instructions is limited to a 147-instruction window, which is greater than both PRFs, but smaller than the ROB

We know that in both Sandy/Ivy Bridge and Northwood, there are separate floating-point/SSE/AVX and integer register files. If I use instructions that do not produce a destination register (nop), the instruction window is limited by the ROB size. Similarly, if I use instructions that produce an integer or AVX/SSE result, the instruction windows becomes limited by the integer or FP/SSE register file instead. If I alternate between integer and SSE/AVX instructions, I would expect that the instruction window becomes ROB-size limited again, since the demand for registers is spread across two register files.

Figure 7 shows the results of this experiment. Ivy Bridge and Northwood behave as expected: AVX or SSE interleaved with an integer addition are limited by the ROB size. Sandy Bridge AVX or SSE interleaved with integer instructions seems to be limited to looking ahead ~147 instructions by something other than the ROB. Having tried other combinations (e.g., varying the ordering and proportion of AVX vs. integer instructions, inserting some NOPs into the mix), it seems as though both SSE/AVX and integer instructions consume registers from some form of shared pool, as the instruction window is always limited to around 147 regardless of how many of each type of instruction are used, as long as neither type exhausts its own PRF supply on its own. That is, when AVX instructions < ~110 and integer instructions < ~130, the instruction window becomes limited to ~147, regardless of how many of each type are in the instruction window. I can't think of any reason why SSE/AVX and integer register allocations should be linked. This strange behaviour appears to be fixed in Ivy Bridge, but Figure 6b shows Ivy Bridge still has some strangeness in its AVX register allocation...

Conclusion

Using two chains of pointer chasing with filler instructions is a powerful tool to find out certain details about a processor’s microarchitecture. I’ve shown how to use it to measure the reorder buffer size, the number of available speculative registers in physical register files, and measure the existence of several related optimizations. This is not an exhaustive list, but I need to get back to doing real work now…

Assuming data that has been recently accessed will likely be accessed again soon usually works well. However, an access pattern that repeatedly cycles through a working set larger than the cache results in 100% cache miss: The most recently used cache line won’t be reused for a long time. Adaptive Insertion Policies for High Performance Caching (ISCA 2007) and a follow-on paper High performance cache replacement using re-reference interval prediction (RRIP) (ISCA 2010) describe similar cache replacement policies aimed at improving this problem. The L3 cache on Intel’s Ivy Bridge appears to use an adaptive policy resembling these, and is no longer pseudo-LRU.

Measuring Cache Sizes

Figure 1: Cache access latencies for four Intel microarchitectures (Stride = 64 bytes) [png][pdf]

The behaviour of LRU replacement policies with cyclic access patterns is useful for measuring cache sizes and latencies. The access pattern used to generate Figure 1 is a random cyclic permutation, where each cache line (64 bytes) in an array is accessed exactly once in a random order before the sequence repeats. Each access is data-dependent so this measures the full access latency (not bandwidth) of the cache. Using a cyclic pattern results in sharp changes in latency between cache levels.

Figure 1 clearly shows two levels of cache for the Yorkfield Core 2 (32 KB and 6 MB) and three levels for the other three. All of these transitions are fairly sharp, except for the L3-to-memory transition for Ivy Bridge (“3rd Generation Core i5″). There is new behaviour in Ivy Bridge’s L3 cache compared to the very similar Sandy Bridge. Curiosity strikes again.

Ivy Bridge vs. Sandy Bridge

Figure 2: Varying the stride of the random cyclic permutation access pattern for Ivy Bridge and Sandy Bridge. Sharp corners appear when stride >= cache line size, except for Ivy Bridge L3. [png][pdf]

Figure 2 shows a comparison of Sandy Bridge and Ivy Bridge, for varying stride. The stride parameter affects which bytes in the array are accessed by the random cyclic permutation. For example, a stride of 64 bytes will only access pointers spaced 64 bytes apart, accessing only the first 4 bytes of each cache line, and accessing each cache line exactly once in the cyclic sequence. A stride less than the cache line size results in accessing each cache line more than once in the random sequence, leading to some cache hits and transitions between cache levels that are not sharp. Figure 2 shows that Sandy Bridge and Ivy Bridge have the same behaviour except for strides at least as big as the cache line size for Ivy Bridge’s L3.

There are several hypotheses that can explain an improvement in L3 cache miss rates. Only a changed cache replacement policy agrees with observations:

• Prefetching: An improved prefetcher capable of prefetching near-random accesses would benefit accesses of any stride. Figure 2 shows no improvement over Sandy Bridge for strides smaller than a cache line.
• Changed hash function: This would show a curve with a strange shape, as some parts of the array see a smaller cache size while some other parts see a bigger size. This is not observed.
• Changed replacement policy: Should show apparent cache size unchanged, but transitions between cache levels may not show the sharp transition seen with LRU policies. This agrees with observations.

Figure 3 shows two plots similar to Figure 2 for larger stride values (512 bytes to 128 MB). The curved shape of the plots for Ivy Bridge is clearly visible for many stride values.

Figure 3a: Sandy Bridge, larger strides [png][pdf]

Figure 3b: Ivy Bridge [png][pdf]

Adaptive replacement policy?

An interesting paper from UT Austin and Intel from ISCA 2007 (Adaptive Insertion Policies for High Performance Caching) discussed improvements to the LRU replacement policy for cyclic access patterns that don’t fit in the cache. The adaptive policy tries to learn whether the access pattern reuses the cache lines before eviction and chooses an appropriate replacement policy (LRU vs. Bimodal Insertion Policy, BIP). BIP places new cache lines most of the time in the LRU position, the opposite behaviour of LRU.

Testing for an adaptive policy can be done by attempting to defeat it. The idea is to trick the cache into thinking that cached data is reused, by modifying the access pattern to reuse each cache line before eviction. Instead of a single pointer chase by repeating p = *(void**)p, the inner loop was changed to do two pointer chases p = *(void**)p; q = *(void**)q;, with one pointer lagging behind the other by some number of iterations, designed to touch each line fetched into the L3 cache exactly twice before eviction. Figure 4 plots the same parameters as Figure 3 but with the dual pointer chase access pattern. The Ivy Bridge plots closely resemble Sandy Bridge, showing that the replacement policy is adaptive, and has been mostly defeated.

Figure 4a: Sandy Bridge, dual pointer chase [png][pdf]

Figure 4b: Ivy Bridge, dual pointer chase [png][pdf]

Since Ivy Bridge uses an adaptive replacement policy, it is likely that the replacement policy is closely related to the one proposed in the paper. Before we probe for more detail, we need to look a little more closely at the L3 cache.

Ivy Bridge L3 Cache

The L3 cache (also known as LLC, last level cache) is organized the same way for both Sandy Bridge and Ivy Bridge. The cache line size is 64 bytes. The cache is organized as four slices on a ring bus. Each slice is either 2048 sets * 16-way (2 MB for Core i7) or 2048 sets * 12-way (1.5 MB for Core i5). Physical addresses are hashed and statically mapped to each slice. (See http://www.realworldtech.com/sandy-bridge/8/) Thus, the way size within each slice is 128 KB, and a stride of 128 KB should access the same set of all four slices using traditional cache hash functions.

Figure 3 reveals some information about the hash function used. Here are two observations:

• Do bits [16:0] (128KB) of the physical address map directly to sets? I think so. If not, the transition at 6 MB would be spread out somewhat, with latency increasing over several steps, with the increase starting below 6 MB.
• Is the cache slice chosen by exactly bits [18:17] of the physical address? No. Higher-order address bits are also used to select the cache slice. In Figure 3, the apparent cache size with 256 KB stride has doubled to 12 MB. It continues to double at 512, 1024, and 2048 KB strides. This behaviour resembles a 48-way cache. This can happen if the hash function equally distributes 256 through 2048 KB strides over all four slices. Thus, some physical address bits higher than bit 20 are used to select the slice, not just bits [20:17]. Paging with 2 MB pages limits my ability to test physical address strides greater than 2 MB.

Choosing a Policy using Set Dueling

Figure 5: Dynamic Insertion Policy, Set Duel [png][pdf]

Adaptive cache replacement policies choose between two policies depending on which one is appropriate at the moment. Set dueling proposes to dedicate a small portion of the cache sets to each policy to detect which policy is performing better (dedicated sets), and the remainder of the cache as follower sets that follow the better policy. A single saturating counter compares the number of cache misses occurring in the two dedicated sets.

This test attempts to show set-dueling behaviour by finding which sets in the cache are dedicated and which are follower sets. This test uses a 256 KB stride (128 KB works equally well), which maps all accesses onto one set in each of the four cache slices (due to the hash function). By default, this would mean all accesses only touch the first set in each slice if the low address bits [16:0] are zero. Thus, I introduce a new parameter, offset, which adds a constant to each address so that I can map all accesses to the second, third, etc. sets in each slice instead of always the first. This parameter is swept both up and down, and the cache replacement policy used is observed.

Since the adaptive policy chooses between two different replacement policies, I chose to distinguish between the two by observing the average latency at one specific array size, 4/3 of the L3 cache size (e.g., compare Sandy Bridge vs. Ivy Bridge at 8 MB in Figure 2). A high latency indicates the use of an LRU-like replacement policy, while a lower latency indicates the use of a thrashing-friendlier BIP/MRU type policy. Figure 5 plots the results.

It appears that most of the cache sets can use both cache replacement policies, except for two 4 KB regions at 32-36 and 48-52 KB (4 KB = 64 cache sets). These two regions always use LRU and BIP/MRU policies, respectively. The plot is periodic every 128 KB because there are 2048 sets per cache slice.

The global-counter learning behaviour is seen in Figure 5 by observing the different policies used while sweeping offset ascending vs. descending. Whenever the offset causes memory accesses to land on a dedicated set, it accumulates cache misses, causing the rest of the cache to flip to using the other policy. Cache misses on follower sets do not influence the policy, so the policy does not change until offset reaches the next dedicated set.

DIP or DRRIP?

The later paper from Intel (ISCA 2010) proposes a replacement policy that is improved over DIP-SD by also being resistant to scans. DIP and DRRIP are similar in that both use set dueling to choose between two policies (SRRIP vs. BRRIP in DRRIP, LRU vs. BIP in DIP).

Figure 6: Scan resistance of Sandy Bridge LRU, Ivy Bridge “LRU-like”, and Ivy Bridge “BIP-like”. Pointer chasing through one huge array, and one array whose size is swept (x-axis). Replacement policy is scan-resistant if it preferentially keeps the small array in the cache rather than split 50-50. [png][pdf]

One characteristic of RRIP is that it proposes four levels of re-reference prediction. True LRU has as many levels as the associativity, while NRU (not recently used) has two levels (recently used, or not). Intel’s presentation at Hot Chips 2012 hinted at using four-level RRIP (“Quad-Age LRU” on slide 46). Thus, I would like to measure whether Ivy Bridge uses DIP or DRRIP.

Scans are short bursts of cache accesses that are not reused. SRRIP (but not LRU) attempts to be scan tolerant by predicting new cache lines to have a “long” re-reference interval and allowing recently-used existing cache lines to age slowly before being evicted, preferentially evicting the newly-loaded but not reused cache lines. This behaviour provides an opportunity to detect scan resistance by pointer-chasing through two arrays, one of which is much larger than the cache (scanning), one of which fits in the cache (working set).

Figure 6 plots the average access time for accesses that alternate between a huge array (cache miss) and a small array (possible cache hit) whose size is on the x-axis. The size of the working set that will fit in the cache in the presence of scanning is related to scan resistance. Memory accesses are serialized using lfence. Replacement policy is selected by choosing a stride and offset that coincides with a dedicated set (See Figure 5).

With half of the accesses hitting each array, an LRU replacement policy splits the cache capacity evenly between the two arrays, causing L3 misses when the small array size exceeds ~half the cache. BIP (or BRRIP) is scan-resistant and keeps most of the cache for the frequently-reused small array. The LRU/SRRIP policy in Ivy Bridge behaves very similar to Sandy Bridge and does not seem to be scan-resistant, thus it is likely not SRRIP as proposed. It could, however, be a variant on SRRIP crafted to behave like LRU.

The four-level RRIP family of replacement policies are an extension of NRU, using two bits per cache line to encode how soon each cache line should be evicted (RRPV). On eviction, one cache line with RRPV=3 (LRU position) is chosen to be evicted. If no cache lines have RRPV=3, all RRPV values are incremented until a cache line can be evicted. Whenever a cache line is accessed (cache hit), its RRPV value is set to 0 (MRU position). The various insertion policies as proposed in the paper are as follows:

• BRRIP: New cache lines are inserted with RRPV=3 with high probability, RRPV=2 otherwise.
• SRRIP: New cache lines are inserted with RRPV=2

I made some modifications for my simulations:

• BRRIP: Modified the low-probability case to insert with RRPV=0 instead of RRPV=2
• LRU-like RRIP: New cache lines are inserted with RRPV=0. This is intended to approximate LRU.

Figure 7: Cache simulations of scan resistance of several replacement policies. Ivy Bridge (Figure 6) matches very well with LRU and BRRIP policies, but does not show the scan-resistance property of SRRIP. [png][pdf]

Figure 7 shows L3-only cache simulation results of running the same access pattern through simulations of several replacement policies. The simulated cache is configured to match the Core i5 (four 1.5 MB slices, 12-way). The two policies used by Ivy Bridge (Figure 6) match very closely to LRU and BRRIP. Ivy Bridge matches less closely to a modified four-level RRIP configured to behave like LRU, and also matches slightly less well to BIP (not shown) than BRRIP. The simulation of SRRIP verifies its scan-resistance properties, which are not observed in the experimental measurement of Ivy Bridge.

I am not able to measure whether Ivy Bridge has really changed from the pseudo-LRU that was used in Sandy Bridge to a four-level RRIP. Given Intel’s hint in the slide and that Ivy Bridge behaves slightly differently from Sandy Bridge, I am inclined to believe that Ivy Bridge uses RRIP, despite the experimental measurements matching more closely to LRU than LRU-like four-level RRIP. However, it is fairly clear that Ivy Bridge lacks the scan-resistance property proposed in the ISCA 2010 paper.

Conclusion

Although the cache organization between Sandy Bridge and Ivy Bridge are essentially identical, Ivy Bridge’s L3 cache has an improved replacement policy. The policy appears to be similar to a hybrid between “Dynamic Insertion Policy — Set Duel” (DIP-SD) and “Dynamic Re-Reference Interval Prediction” (DRRIP), using four-level re-reference predictions and set dueling, but without scan resistance. For each 2048-set cache slice, 64 sets are dedicated to each of the LRU-like and BRRIP policies, with the remaining 1920 cache sets being follower sets that follow the better policy.

Acknowledgements

Many thanks to my friend Wilson for pointing me to the two ISCA papers on DIP and DRRIP.

Each new generation of CMOS manufacturing processes brings about a new set of trade-offs. Intel’s recent tradition of manufacturing the same processor microarchitecture across two processes provides an opportunity to look at some of the voltage-delay-power scaling trends. Intel’s new Ivy Bridge processor manufactured on a 22nm tri-gate CMOS process, which is a significant change from the planar transistors used in previous processes. Intel’s previous-generation Sandy Bridge processor made on their 32nm planar CMOS process uses a similar architecture, and can be used as a point of comparison.

In a complete system, a processor’s power consumption, voltage, temperature, and operating frequency can be observed, while the latter three can be controlled. Using those tools, we can measure static and dynamic power as a function of temperature, frequency, and voltage, create shmoo plots (voltage vs. operating frequency), and compare overall thermal resistance.

There have been some rumblings that Ivy Bridge does not overclock as well as Sandy Bridge. On the other hand, Intel claims the 22nm process improves performance over 32nm. Another difference between the two processors is the switch from using solder thermal interface material (STIM) to polymer (PTIM), resulting in increased thermal resistance and higher junction temperatures on Ivy Bridge for the same power. A comparison of the measurements across Sandy Bridge and Ivy Bridge can quantify some of these observations.

Methodology

• Core i5-2500K (32nm Sandy Bridge) and i5-3570K (22nm Ivy Bridge)
• Biostar TZ77MXE motherboard

The TZ77MXE motherboard allows adjustment of processor frequency (multiplier) and voltage, although it does not allow manual adjustment of voltage below 1.0V, or negative voltage offsets.

Power consumption is measured using multimeters on the 12V power connector to measure current and voltage. On modern ATX motherboards, CPU and GPU power regulators are powered by the 12V power connector, which gives a convenient place to measure the current and voltage consumed by the processor package excluding the rest of the system. Power is measured before the processor’s voltage converter. DC-DC converters are typically efficient, so I make no attempt to compensate for it. Voltage is measured at the connector after the ammeter’s voltage drop to reduce the skew caused by the ammeter’s resistance. Switching converters are generally tolerant of varying input voltages (10.9V to 12.2V observed) with minimal impact on efficiency.

The processor’s operating voltage is measured using the on-board IT8728F chip. Temperature is measured using coretemp (CPU on-die temperature sensors), reporting the temperature of the hottest core (usually core 2, cores numbered 0-3).

To control processor operating frequency, I changed only the multiplier while leaving BCLK at 100 MHz. Core voltage is controlled by setting a fixed voltage in the BIOS. I rely on the measured voltage rather than the voltage setting because the actual voltage can vary based on the load line (a mechanism that lowers supply voltage under high load to reduce the peak voltage swing) or “load line calibration” (a mechanism to defeat the load line). Processor temperature is controlled by lowering the cooling fan speed to raise the temperature.

Power consumption (switching activity?) depends strongly on the choice of workload. Power and temperature measurements are made when all four cores of the processor are active running the Prime95 torture test. Prime95 is able to sanity-check its own calculations, so it is also used to check for processor stability when generating a shmoo plot.

Results

Power and Temperature

Fig. 1a: SNB Power vs. Temperature
1.26 V, 1.6 GHz and 2.4 GHz

Fig. 1b: IVB Power vs. Temperature
1.26 V, 1.6 GHz and 2.4 GHz

I measure power consumption vs. temperature first, since its results can be used to compensate for varying temperature in later measurements. For both processors, I measure total power at 1.26V at both 1.6 GHz and 2.4 GHz. Total power can be broken down into two components: Static power that does not vary with switching frequency, and dynamic power that varies with switching frequency. Assuming dynamic power scales linearly with frequency, measuring at two frequencies allows extrapolating power consumption down to 0 Hz to separate out dynamic power and static power.

Figure 1a and 1b shows power vs. temperature for Sandy Bridge (SNB) and Ivy Bridge (IVB), respectively. Total power is plotted, as well as the extrapolated static power. Figure 1b plots both Ivy Bridge’s and Sandy Bridge’s static power for comparison. Dynamic power does not depend on temperature, since the 1.6 GHz and 2.4 GHz curves are parallel. The extrapolated static power curve includes data points from both curves translated downwards by twice and three times the difference in power between the two curves. The extrapolated static power data points fits an exponential function very well, which agrees with theory that says leakage power typically grows exponentially with temperature. Ivy Bridge shows a significant improvement in static (leakage) power. One of the claimed benefits of multi-gate transistors is better channel control resulting in a better subthreshold slope and lower subthreshold leakage, and this measurement agrees.

Dynamic Power vs. Frequency

Fig. 2a: SNB Power vs. Frequency
1.26 V, variable temperature

Fig. 2b: Power vs. Frequency
1.26 V, variable temperature

Another classic textbook result is that dynamic power scales linearly with frequency. Figures 2a and 2b show measurements of total power and dynamic power for Sandy Bridge and Ivy Bridge, at 1.26V. Total power consumption is measured, while dynamic power is calculated by subtracting out the temperature-dependent static power found in the previous section.

The dynamic power curve fits a linear trendline very well. The intercept of the dynamic power trendline is expected to be zero (no dynamic power when no switching activity). A non-zero intercept for the trendline indicates some amount of experimental error, around half a watt in these plots. The red curves of total power has a slight upwards curve because total power (static power, but not dynamic power) increases with temperature.

Figure 2b includes the dynamic power curves for both processors for comparison. At 1.26V (an arbitrary voltage somewhat higher than the typical operating point), dynamic power for Ivy Bridge is only slightly lower (~6%). The main objective of this graph was to show that dynamic power increases linearly with frequency. The next section shows how dynamic power scales with processor supply voltage.

Power vs. Supply Voltage

Fig. 3a: SNB Power vs. Voltage
1.6 GHz and 2.4 GHz, 90°C

Fig. 3b: IVB Power vs. Voltage
1.6 GHz and 2.4 GHz, 90°C

Fig. 3c: Power vs. Voltage Comparison
2.4 GHz, 90°C

The textbook formula says that dynamic power should be proportional to the square of the supply voltage. This section describes the same measurement. I vary the processor supply voltage while keeping frequency and temperature constant. Like earlier, dynamic and static power is separated by measuring power consumption and 1.6 and 2.4 GHz. I keep temperature constant at 90°C because it is easy to raise the operating temperature by slowing down the cooling fan, but very difficult to lower it. The resulting measurements will show how dynamic power scales with supply voltage and how static power scales with supply voltage at a fixed 90°C temperature.

Figures 3a and 3b show the results of these measurements for Sandy Bridge and Ivy Bridge, respectively.

The top two curves in each figure are direct measurements of total processor power at 1.6 and 2.4 GHz. Since total power includes both static and dynamic power, we need to break total power into static and dynamic components before curve fitting. Because temperature is kept constant, each pair of data points at a given voltage have the same static power, so static power can be computed as above, by taking the difference between total power at 1.6 and 2.4 GHz, independently for each voltage, giving the green static power curve. Dynamic power is then computed by subtracting static power from the total power.

For Sandy Bridge (Fig. 3a), the dynamic power fits a power curve well, and comes surprisingly close to the expected quadratic relation, P_dynamic ∝ V². Static power also fits a power curve (although I’m not aware of theory that requires it), where static power increases roughly as the cube of the voltage.

On Ivy Bridge (Fig. 3b), the curve fits are somewhat unexpected. Static power grows much slower than on Sandy Bridge (roughly P_static ∝ V^1.85 instead of V³), but dynamic power grows slightly more quickly with voltage (P_dynamic ∝ V^2.3 compared to V²). A comparison of just the 2.4 GHz dynamic power and static power is plotted in Fig. 3c. Dynamic power on Ivy Bridge is lower for all practical voltages (the curve fit suggests Ivy Bridge dynamic power will exceed Sandy Bridge above 1.9V).

I speculate that these differences (slower static power increase, but slightly higher dynamic power increase with voltage) are properties of tri-gate processes, but I don’t know enough about the differences between planar and tri-gate to know whether these observations match with theory.

Voltage-Frequency Shmoo Plot

Fig 4a: SNB Voltage-Frequency Shmoo

Fig 4b: IVB Voltage-Frequency Shmoo

Fig 4c: Voltage-Frequency Shmoo Comparison

The primary knob for increasing the frequency of a processor is increasing its operating voltage. A shmoo plot characterizes the voltage-frequency relationship by testing a processor at various voltage and frequencies and recording which points function correctly (“pass”) and which do not (“fail”). The boundary between the pass and fail points indicate the lowest voltage at a given frequency (or, alternatively, highest frequency at a given voltage) at which that the processor can still operate, which would correlate to how easily one can overclock the processor.

Unlike the rest of the measurements, the shmoo plots are made while only using one processor core with three cores idle. Prime95 was run on the slowest of the four cores, and a particular voltage and frequency is considered “pass” if Prime95 runs for around 10 minutes without error. The shmoo plots are slightly optimistic: A real-world usage scenario with four active cores instead of one usually requires higher voltage and causes higher temperatures, further reducing achievable frequency. Although running just one active core reduces the effect of temperature (by reducing the temperature change), I do not measure or compensate for the impact of temperature on maximum frequency.

Figures 4a and 4b show the shmoo plots for Sandy Bridge and Ivy Bridge, respectively. Additionally, a line was drawn that connects the lowest voltage that passes at each frequency, which approximates the boundary between the “pass” and “fail” points. Figure 4c shows a comparison of Sandy Bridge and Ivy Bridge. The two boundary lines from Figures 4a and 4b are plotted in Figure 4c. It is interesting that the slope of the Ivy Bridge curve (blue) is higher than for Sandy Bridge. Although Ivy Bridge is significantly faster than Sandy Bridge at low voltages, increasing the operating frequency requires a larger voltage increase on Ivy Bridge, such that the two chips require the same voltage (1.32V) to run at 4.5 GHz. This would suggest that overclocking Ivy Bridge beyond this point is somewhat more difficult, even though Ivy Bridge is faster/lower voltage at the lower non-overclocked frequency (below 3.8-3.9 GHz).

Fig. 5: 22nm Performance improvements, from Intel's presentations, extended.

One might recall Intel’s initial presentations on their 22nm process showing charts showing performance and/or voltage improvements over their 32nm process. One such graph is reproduced in the left half of Figure 5. Intel’s chart is interesting: The performance and voltage gains claimed are indeed impressive, but the gain decreases at nigher voltages (37% faster at 0.7V, 18% faster at 1.0V), but the typical operating point for the desktop processors is beyond the right edge of the chart (even before overclocking). Is there something unpleasant about the higher (typical!) voltages that Intel didn’t want to mention?

Subject to a few important caveats, Intel’s chart of voltage vs. gate delay is equivalent to a shmoo plot. One caveat is that Intel’s chart shows low-level transistor delays, while a shmoo plot shows the delay of a more complex circuit. In addition, a complex circuit consists of both transistor delay and interconnect delay, so it is expected that performance gains seen at the transistor level will be smaller when applied to a whole processor because interconnect delays are expected to become worse with each process shrink.

Given the above caveats, I have attempted to transform the shmoo plot (by plotting delay instead of its reciprocal, frequency) and overlay that onto Intel’s chart in Figure 5. Notice that the voltage range I was able to test is actually entirely off the right edge of Intel’s chart. My shmoo plot seems to match up reasonably well with Intel’s plot. Although performance improvements at low-voltage are high, the improvement shrinks to around 5 percent at typical operating voltages, and performance improve even turns into a performance loss at higher voltages seen when overclocking.

Thermal Resistance

Fig. 6: Thermal Resistance

The ability to cool a processor is determined by its thermal resistance. Power is dissipated at the bottom side of the chip, with most of the heat being dissipated through the top side. Most of the heat must pass through the silicon die, heat spreader, heatsink, then out to air, with some form of thermal interface material in the interface between each of those. The overall thermal resistance can be measured by measuring the power dissipation and total temperature difference between the on-die temperature sensors and ambient air.

There are two main reasons why Sandy Bridge and Ivy Bridge may have different thermal resistance. First, as chips are scaled smaller, power dissipation does not scale as much, leading to higher power density. Ivy Bridge’s die size (160 mm²) is 26% smaller than Sandy Bridge’s (216 mm²), reducing the contact surface area between the die and heat spreader. Second, Intel has switched from using solder between the die and heat spreader (solder thermal interface material, STIM, ~87 W/mK) to a polymer material (PTIM, 3-4 W/mK), presumably because Ivy Bridge’s reduced power dissipation is now comfortably within the range suitable for using PTIM (See Figure 16).

Thermal resistance is measured with all four cores active (fewer active cores results in a hot spot). The stock thermal paste, heatsink, and cooling fan are used on both processors. The cooling fan is kept at its maximum constant speed (around 2050 RPM), and power dissipation is varied by changing the CPU supply voltage.

Figure 6 shows a measurement of the thermal resistance on both processors. On both processors, thermal resistance improves somewhat at higher power. The thermal resistance of Ivy Bridge is around 0.15 °C/W worse than Sandy Bridge. Although it’s not possible to break down the contribution of the two reasons, it seems likely that most of the increase in thermal resistance is due to the change in TIM. An increase of 0.15 °C/W roughly corresponds to the bulk thermal resistance of a ~90 μm layer of PTIM over the die area of 160 mm².

Summary

The above measurements attempt to characterize some of the changes when moving from Intel’s 32nm planar to 22nm Tri-Gate process. The 22nm Ivy Bridge significantly improves on static (leakage) power over 32nm Sandy Bridge, but only shows small reductions in dynamic power. Ivy Bridge also requires higher voltage increases for the same frequency increase, leading to more difficult overclocking but power savings at lower (standard) speeds.

In addition to the CMOS process changes, the thermal resistance of Ivy Bridge increased over Sandy Bridge, likely due to the change from solder to polymer thermal interface material between the die and heat spreader.

The problem with RAID mode is that standard AHCI drivers cannot be used. A JMicron-specific driver is available only for Windows. Some OSes (Linux, drivers/pci/quirks.c) set a PCI configuration register to put the controller into AHCI+IDE mode after booting and use standard drivers. Other OSes (Mac OS X) do nothing and can’t use the card.

I attempt to solve this problem by hacking the JMB363 option ROM to put the card into AHCI mode, with partial success. The two SATA ports work in OS X, including hot-plugging, but I could not get the parallel ATA enabled, and I had to bypass loading the option ROM (thus, can’t boot from the SATA ports anymore). The patched ROM might also be useful for other OSes that don’t have Linux-style PCI quirks.

Option ROMs

• Original Option ROM 1.07.24
• Patch three uses of PCI configuration register 0xdf[1:0] to read 2′b10. This seems to put the controller in AHCI mode.
  • Patched Option ROM
• Also patch two uses of 0xdf[6] to read 1′b1 and writes 0xa1 to configuration register 0x41. This bypasses the option ROM and keeps the SATA ports enabled.
  • Patched Option ROM

Hacking the Option ROM

I used the newest version of the option ROM (1.07.24) posted at JMicron’s FTP site. Some hints about the configuration of the JMB363 can be found in JMicron’s release notes, particularly about PCI configuration register 0xdf. The release notes hints at the existence of a newer 1.07.28, but it wasn’t posted. Linux’s drivers/pci/quirks.c gave some hints about the meaning of PCI configuration register 0x40-43.

Configuration Register 0xDF

PCI configuration register 0xdf seems to be intended for main BIOS to communicate its settings to the option ROM, rather than having a direct effect on the hardware. Bits [1:0] seem to indicate controller mode. Bits [7:6] and [5:4] look like they control two instances of the same thing because the code that parses them is structured the same way. The release notes say bit [6] is used to put the chip into “multi-function” mode with the option ROM disabled, which feels a little bit like a hack to me. Bits [3:2] don’t appear to be used.

It seems like bits [1:0] select the card mode (RAID, IDE, AHCI, IDE?), with the value 2′b11 probably not intended to be used (the option ROM code only distinguishes between 2′b00, 2′b01, and other).

Since setting df[6] causes “jump out of Option ROM, do nothing” (See release notes for v1.03), it is no longer possible to boot from any disks attached to the chip.

In no case did setting 0xdf[6] cause the JMB363 to become a multi-function device. I speculate its purpose is to disable the option ROM, allowing the main BIOS to set up the rest of the device configuration without interference.

• Register df[6] is used by the option ROM code at offset 0x3517.
• Registers df[7:6] and df[5:4] are used at offsets 0x372d and 0x3756, respectively. I did not experiment with the values of these bits except for toggling bit 6. They seem important, causing PCI config register bits 0xed[5:2] and 0xcd[5:2] to be set when df[7:6] and df[5:4], respectively, is 2′b01 or 2′b10.
• Register df[1:0] is used in the option ROM at offset 0x3553, 0x3574, and 0x37c1

Configuration Register 0x40-43

These configuration registers seem to control the hardware directly. The option ROM sets them but doesn’t read them.

Linux (drivers/pci/quirks.c) modifies this configuration register, which results in 0x80c2a1bf on my card. I tried modifying the option ROM to also configure register 0x40-43 to this value, but had many problems booting. Either I’m getting some of the other configuration bytes wrong or this value is intended only to be used after booting the system.

Problems

Only modifying register df[1:0] to value 2′b10 seems to put the controller into AHCI mode (with no IDE PCI function, but PATA drives are still detected by the option ROM). The option ROM loads, correctly reports connected drives, and allows booting from them. However, OS X does not detect hard drives that were detected by the option ROM at boot time. If SATA drives are not attached until after the option ROM loads, they are detected by the OS and can be hot-plugged. Therefore, I chose to set register df[6] to cause the option ROM to quit without detecting drives. A side-effect is that without the option ROM, the computer can’t boot from the disks.

The problems with configuration register 0x40-43 are more problematic.

Linux sets register 0x41 to 0xa1, while the option ROM will set it to 0xf1 or 0x51. 0x51 appears to disable hot-plugging the disks on the SATA channels entirely. 0xa1 causes a hang for several minutes at boot, seemingly trying and failing to find a disk, while using 0xf1 boots normally. Trying with 0xb1 and 0xe1 shows the same behaviour as for 0xa1, but hangs only if a disk is not present on SATA0 or SATA1, respectively. Thus it seems like setting 0x41[7:6] or 0x41[5:4] to 2′b10 causes the option ROM to assert that a disk must be present at boot time for that channel. The hang does not occur if the option ROM is skipped by setting 0xdf[6], so I use 0xa1 to be closer to what Linux does, although I notice no other differences between 0xf1 and 0xa1.

Linux sets register 40 bit 2 (to enable the IDE port?). Trying to do that in the option ROM causes a several-minute hang during boot when loading the option ROM (again, seemingly waiting for a disk and giving up), even when a PATA disk is present. With df[6] set, the option ROM does not detect any disks, despite spending several minutes.

Final configuration

• PATA not enabled because turning it on causes a hang during boot.
• Option ROM not enabled by setting df[6], so the disks are not bootable.
• SATA drives working in OS X with the standard AHCI driver, including hot-plugging.

PCICFG v1.24  (c) Copyright 1997,1998 Ralf Brown
Modified by Datapath based on V1.19

-----------------------------------------------------------
PCI bus 04 device 00 function 00:  Header Type 'non-bridge' (single-func)
Vendor:	197B	???                                               
Device:	2363	???                                               
Class:	  01	disk                	Revision:	10
SubClass: 06	???                 	ProgramI/F:	01
CommandReg:   0007 = I/O-on mem-on busmstr
Status Reg:  0010 = CapList (fast)
CacheLine:      08	Latency:	00	BIST:	     00
SubsysVendor:    197B	SubsysDevice: 2363
Base Addresses:
	(0) 00009001 = I/O base=00009000 len=8
	(1) 00009401 = I/O base=00009400 len=4
	(2) 00009801 = I/O base=00009800 len=8
	(3) 00009C01 = I/O base=00009C00 len=4
	(4) 0000A001 = I/O base=0000A000 len=16
	(5) D9000000 = mem base=D9000000 len=512
CardBus:     00000000	ExpansionROM: 00000000 (64K,disabled)
INTline:	   0A	INTpin:       01
MinGrant:	   00	MaxLatency:   00
Device-Specific Data:
 40: 80C2A1BD  E4FF0808  40F00060  00000000  00110010  00008000 
 58: 000A2000  00036C11  10110000  00000000  00000000  00000000 
 70: 00000000  00000000  00000000  00000000  10000000  00000000 
 88: 00000000  40035001  00000000  00000000  00000000  00000000 
 A0: 00000000  00000000  00000000  00000000  00000000  00000000 
 B8: 00000000  80000000  00000000  00000000  00000000  00000000 
 D0: 80000018  1000001C  00EB0000  00000000  00000000  00000000 
 E8: 00000000  00000000  00100058  00000000  00000000  00100000 
Capabilities List:
	ID @8C = 01 PCI Power Management
	 PMC    =  PME#-D3hot
		DynClk = 0, PCI_PM version = 3
	 PMCSR  = 0000, data-select=0 unknown/unimplemented
		state=D0   
	 PMCSRX = -- -- -- --
	 Data   = 00
	ID @50 = 10 (unknown)

I tested the same C code through the following compilers, both 32- and 64-bit, on the same Core i7-3770K system:

• OS X Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn)
• OS X gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)
• Linux gcc version 4.7.1 (GCC)
• Linux Intel C Compiler Version 12.0.3

The execution speed is normalized to the geometric mean of all the results.

The two sets of bars show the single-threaded performance as well as the performance when running eight copies of the routine (i.e., with Hyper-Threading). The performance gap between compilers narrows when Hyper-Threading is used, because a second thread can absorb the effects of poor instruction scheduling.

Even for a simple loop, there is a surprising difference in the execution speed of the compiled code. Because the loop is so simple and repetitive, a human can figure out roughly what the optimal output is and compare that to what the compilers generate.

C Source Code

In the above code, i is the loop counter and also used as a non-changing integer value in the computations. There are 5 independent chains of ALU operations, alternating between addition and logical AND. counter is a volatile integer variable that is incremented once per iteration. This routine should be able to provide enough data-independent instructions to execute 5 instructions per clock, which is more than sufficient to saturate today’s processors. The highest value I measured is 2.94 instructions per clock including loop and counting overhead. It’s 3.00 instructions per clock once the loop overhead of 2 (fused) instructions every iteration of 100 ALU operations is removed.

The optimal assembly code after compilation should roughly follow the structure of the C code. It would alternate between 5 (independent) ADD operations and 5 AND, with a few loop-counter instructions inserted somewhere within the loop. The general principle is that data-dependent operations should be spaced out to make it easier for the processor to find instruction-level parallelism.

Mac OS X Compilers

Here is the disassembly of the loop compiled on the Mac OS X compilers. The disassembly is in AT&T syntax, so the destination operand is the one on the right. The instructions operating on each “chain” of instructions is colour-coded the same as in the C source code.

It appears that for both Clang and GCC, the Apple compilers use LLVM for optimization and code generation, with Clang and GCC only being front-ends. This might explain the similarity in instruction scheduling strategies between the two compilers: There is a tendency to group dependent chains of instructions together, making it hard for the processor to extract ILP. Only the 64-bit LLVM-Clang has some amount of interleaving of instructions, which leads to a significant performance improvement compared to the other three compilers.

Given the poor instruction scheduling of 32-bit LLVM-Clang and both 32- and 64-bit LLVM-GCC, it’s no surprise these three perform worst. So what distinguishes between these three?

In 32-bit LLVM-Clang, the first 6 instructions in the loop have many long-latency data dependencies. The value of eax is modified at the bottom of the loop (loop-carried dependency at instruction #105), then stored to memory (instruction #1), loaded again (#4), and is immediately consumed (#6). There wasn’t even an attempt to space out the long-latency store-load-use operations, and this hurts performance.

32-bit LLVM-GCC generates nearly the same code with the same problems as 32-bit LLVM-Clang, but backing up the register value (instruction #1 in the LLVM-Clang code) is done in the middle of the loop (instruction #24).

64-bit LLVM-GCC’s output is significantly better. There are more registers in x86-64 so register spilling is no longer necessary, and the long-latency memory operations are no longer data-dependent with the ALU operations. It still suffers from poor instruction scheduling.

Linux Compilers

The Linux compilers perform quite well, and the disassembly clearly shows that instruction scheduling is much improved compared to the LLVM-generated code.

32-bit GCC (real GCC this time, not an LLVM backend) shows some irregularity in the instruction interleaving. It spills one register to memory, and does that fairly intelligently. The register is spilled at instructions #87-88 and not consumed until instruction #105. Likewise, the same register is spilled again at instruction #1 but not consumed until instruction #11. Despite needing to spill a register to memory, its performance impact is fairly small.

64-bit code allows GCC to have enough registers to avoid spilling to memory…maybe too many. In instructions #1 and #92, it needlessly switches between using edx and r15d to hold the same variable. Like 32-bit GCC, 64-bit GCC also unnecessarily splits up the counter increment into three instructions (load #87, add #97, store #104). Because it does not need to spill registers to memory, GCC’s instruction scheduling spaces out dependent instructions well, although seemingly with some randomness.

Both the 32-bit and 64-bit Intel compilers generate nearly identical code, except for register assignment, and therefore, code size. Instructions are perfectly interleaved, with maximal spacing between dependent instructions. Incrementing the volatile counter uses a pointer stored in a register (instruction #97) with the destination operand in memory, with no register needed to temporarily store the counter value. This saves one register, making 32-bit ICC to be the only 32-bit compiler that does not spill any registers to memory for this routine. The use of dec-jne (instructions #102-103) to terminate the loop also allows macro-op fusion to work.

Register Spilling

LLVM-Clang and LLVM-GCC seem to interpret the volatile differently from the GCC and Intel C compilers. The counter variable was declared as volatile unsigned int &counter. LLVM-Clang and LLVM-GCC interpret this as meaning that both the pointer itself is volatile, as well as the value to which it points. This leads to code which loads the value of the pointer from memory, then increments the integer located in memory using that pointer. GCC and Intel’s compiler interprets the volatile declaration as meaning only the final integer is volatile. It keeps the pointer in a register, and simply increments the integer located where the pointer points. This reduces register usage by one. GCC wastes one register by splitting up counter++ into three instructions instead of using a destination operand in memory, so the 32-bit Intel compiler is the only compiler that generates 32-bit code that does not spill any registers to memory.

Instruction Scheduling

It seems like this version of LLVM gets it wrong (it tends to group dependent operations together), while GCC and ICC do it right.

System

• Core i7-3770K 3.5 GHz, 4 cores, 8 threads (2-way SMT)
• Mageia Linux, kernel 3.4.4, CFQ scheduler
• Mac OS X 10.7 (Update: Also 10.8)
• Workload: Independent ALU instructions (does not improve with SMT)

Update

The thread scheduler on Mac OS X 10.8 behaves identically.

Test Result

As more concurrent processes are run, the total throughput increases up to 4 threads on 4 cores. Due to the nature of the workload, SMT offers no additional throughput improvement. A good SMT-aware process scheduler would first assign threads to separate cores before doubling up threads on the same core. The CFQ scheduler is able to do that, but the OS X scheduler does not. It makes sub-optimal scheduling decisions when there are between 4 and 6 threads, choosing to assign two threads on the same core while leaving another core idle, causing a performance loss (11% loss at 4 threads).

Is the OS X process scheduler SMT-aware? Yes. Does it make good scheduling decisions? Not really, but still better than the BFS scheduler.

The Intel High Definition Audio Specification defines defines a standard 48-pin package for the codec chip, although it is “not a compliance requirement for all codecs”. It appears that Both Realtek and VIA codecs usually follow the standard package, so it’s likely they’re (almost?) pin-compatible. Update: The pin-outs of the ALC885 and VT2021 are the same.

Datasheets:

• ALC885
• VT2021 (Thanks, David!)

Since I’m building a Hackintosh with a GA-Z77M-D3H with a VIA VT2021 chip, I decided to work around the audio issues by swapping the VT2021 with a Realtek ALC885 chip.

Procedure

1. Desolder the existing codec from the motherboard
2. Clean off any bridged pads on the motherboard (oops)
3. Add flux and solder new chip
4. Inspect, then power on

Here are close-up photographs of the removed VT2021 chip and newly-installed ALC885.

Results

Surprisingly, the replaced chip appears to work fine. It was auto-detected and configured automatically in Linux, and also works in OS X with the usual amount of headache. I have not rigorously tested whether all of the output ports work. Those that I tested (rear microphone and rear line out) work fine.

The codec chip and its accompanying 12V-to-5V linear regulator (AS78L05, left of the codec) get suspiciously hot (around 50 C?). I do not know if it’s normal for the ALC885 to dissipate enough power to reach that temperature, nor did I think of checking the VT2021′s temperature before replacing it. Since the audio output is working, I currently have no proof that this is abnormal…

• AppleIntelGraphicsHD4000.kext
• AppleIntelGraphicsFramebufferCapri.kext
• …and maybe some others I don’t know about

As pointed out by ElNono and proteinshake, the critical bit missing to get the HD4000 drivers to load is the AAPL,ig-platform-id device property for the graphics device. Of course, on anything other than Apple hardware, this property wouldn’t exist, and would need to be added.

The AAPL,ig-platform-id property is a 32-bit number that must be one of the values listed in the following table. Thanks to ElNono for figuring it out.

The value of AAPL,ig-platform-id selects which graphics configuration the driver will use (er, blindly assume). Setting AAPL,ig-platform-id to any of the 12 values will load the HD4000 driver, but there are some other constraints.

First, the setting affects which ports are enabled. Configurations with zero ports should be avoided (no output?). Configurations with just one port should probably be avoided because it’s probably not the port you’re looking for. On my GA-Z77M-D3H, for configurations with one port, the enabled port is not the DVI port. For configurations with two or more ports, DVI was available. The VGA port is not enabled for any of the configurations. I did not test the HDMI port.

Also, the graphics memory size for the configuration must match the setting in BIOS. If they don’t match, the driver may crash (kernel panic at gen7_memory.cpp:721), or the display may be corrupted. For example, using configuration 0x01660000 (96 MB, 3 pipes, 4 ports), set the Graphics Memory Size to 96 MB to match. It appears that OS X doesn’t obey the DVMT Total Memory Size setting (always 512 MB with 8 GB system RAM), so I just left it as MAX to match.

Results

ioreg -l -p IODeviceTree can be used to verify that the device property was correctly added to the list of properties for the graphics device. The end result is that the AAPL,ig-platform-id property shows up in ioreg output. A correct value causes AppleIntelGraphicsHD4000.kext to load.

Setting AAPL,ig-platform-id

The goal is to inject a device property named AAPL,ig-platform-id with a 32-bit value that is one of the table entries (e.g., 01660000). There are many ways to inject a device property, and I know very few of them. The method I used was to modify the Chameleon bootloader’s GraphicsEnabler code to add the property whenever GraphicsEnabler=y (gma.c). Here are a few options I’ve heard of:

• Modify the bootloader’s GraphicsEnabler code. I modified Chimera 1.10 (rev 1999, i386/libsaio/gma.c) to add a new device property value.
• Device property injection by the bootloader using “device-properties” in org.chameleon.Boot.plist. I don’t know how to generate such a string, but there are a few examples posted in this thread. Something about gfxutil.
• Editing the DSDT to add the property for the graphics device.

Modifying the Chameleon Boot Loader

I chose to modify the GraphicsEnabler code from the Chameleon/Chimera bootloader. The patches below can be applied to the source code from svn (http://forge.voodooprojects.org/svn/chameleon). I’ve also compiled patched versions of Chameleon (rev. 2012) and Chimera (rev. 1999) for convenience.

• Chameleon svn r2012
  • Patch for gma.c
  • Patched Chameleon r2012 [pkg] [md5]
• Update July 20, 2012: Chimera 1.11
  • Chimera 1.11 has been updated with something similar to what’s described here (AAPL,ig-platform-id set to 01660009 and device-id set to 0166).
• Chimera 1.10.0
  • Patch for gma.c
  • Patched Chimera 1.10.0 [pkg] [md5]

Settings that don’t matter

It appears that AAPL,ig-platform-id is the only critical setting to enable hardware acceleration. I’ve tried changing a few other things and none of them seem to matter.

device-id and revision. The driver doesn’t care whether this is mobile (0166) or desktop (0162). I suspect it needs to be one of the two in order to load the kext, but it’s not important to match it with the high-order 16 bits of AAPL,ig-platform-id. I left mine unchanged at 0162:0009 using configuration 01660000.

GFX0 vs. IGPU in DSDT. This doesn’t seem to matter either. Both work the same, so I left mine as GFX0 without issues.

SMBios version/productname. I tried a few, and also tried without smbios.plist. The driver loaded fine in all cases. (Mac Pro, iMac, MacBook Pro were all ok)

Settings that do matter

AAPL,ig-platform-id. Yes, it matters.

Graphics memory size in BIOS. This must match the amount of graphics memory for the chosen configuration.

Connectors. Not all output connectors are enabled. Getting a garbled display could be a symptom the connector currently used is disabled. Changing the value of AAPL,ig-platform-id might change which connectors are enabled/disabled.

Test Setup

Current was measured by multimeter at the 12V connector. Power is calculated assuming voltage is 12V. Measuring power at the 12V CPU power connector isolates the power to the CPU, GFX, and power converter losses, without measuring the power consumed by the rest of the system. I don’t attempt to adjust for power conversion losses.

CPU core voltage was measured by the sensor on the motherboard. The GA-Z77M-D3H does not have core voltage adjustments, so I’m limited to observing what Intel’s voltage algorithm decides (which is interesting too).

In addition to frequency and voltage, the number of active cores was also varied. This is accomplished by setting processor affinity to constrain the test workload to run on fewer cores. The data presented here involve running Prime95 on both thread contexts of an active core, except the series testing four active cores without Hyper-Threading. Leaving one of each pair of thread contexts idle measures no-Hyper-Threading power, and leaving both thread contexts idle results in an idle processor core.

Limitations

The most obvious limitation is that I don’t have control over VCore, so I can’t characterize the power consumption of the processor at all operating points, nor can I produce a frequency-voltage schmoo plot.

Temperature affects power consumption (higher temperature causes higher leakage power). In these tests, I used Intel’s stock cooler with PWM set to default, so the core temperature isn’t constant. The stock cooler also limited the maximum frequency I could test without thermal throttling. I did let temperature stabilize before taking voltage and power measurements. I haven’t characterized the dependence of CPU power on core temperature (at the same voltage and frequency).

Results

Power vs. Frequency

Intel’s voltage control algorithm seems fairly simple:

• Below 3 GHz: 0.96 V
• 3 GHz to 4 GHz: Linear increase
• Above 4 GHz: 1.20 V

The voltage is slightly higher when more cores are utilized. In the scatterplot of VCore points, the highest voltage for each frequency is for 4 loaded cores, while the bottom voltage is for 1 loaded core (3 idle cores).

This voltage control algorithm is interesting. Below 3 GHz, VCore is set at 0.96 V, suggesting that the processor isn’t able to run reliably below that voltage (with some safety margin). This is in contrast to Intel’s PR campaign for their 22 nm process which touted leakage reductions at 0.7 V (-37%) through 1.0 V (-18%) compared to their 32 nm process. Ivy Bridge’s actual operating voltage range (0.96 – 1.2 V) is almost entirely outside (and above) the range of voltages presented in Intel’s data. Suspicious.

A second interesting feature is that the voltage increases end at 4 GHz, slightly above the highest turbo frequency bin. Clearly, Intel doesn’t cater to people who overclock beyond the highest turbo bin. Ideally, the voltage-frequency curve should approximate the shmoo plot, which typically has linearly increasing voltage with frequency. This stepped curve is sub-optimal when using a voltage offset (shifting this curve upwards or downwards) when overclocking, where a more desirable curve would be to (super?-)linearly extend the voltage curve beyond 4 GHz. Of course, using a fixed voltage (flat curve) is even less optimal.

Below 3 GHz, where VCore is roughly constant, power consumption increases fairly linearly with frequency, as expected. Power increases roughly cubic with respect to frequency in the region between 3 and 4 GHz because of the linearly-increasing voltage, and the voltage increases returns to ~linear once voltage stops increasing past 4 GHz.

Idle power is an impressive 4.4 W (including voltage converter losses).

Hyper-Threading Power

One of the data series in the plot involves loading all 4 cores with one thread each. This measures the power consumption impact of Hyper-Threading. Comparing 4 active cores with one thread vs. two threads each, Hyper-Threading consumes an extra 8% of power. However, at an average of 22% performance improvement, Hyper-Threading is a power-efficient method of improving performance, provided the workload has enough threads to use it.

I will run the workloads that I ran on various older systems and compare them with the new processor. Since I don’t have a Sandy Bridge processor, the comparisons will be against the previous microarchitecture (Lynnfield/Gulftown). I expect Sandy Bridge would be very similar to (but slightly slower) than Ivy Bridge.

See also:

• Core2, Nehalem, FPGA CAD
• Hyper-Threading Performance

FPGA CAD

The first set of benchmarks are an extension of the earlier FPGA CAD benchmarks. Most of the results have already been presented there, but will be copied here for convenience.

Hardware

Workloads

Results

Memory Latency and Bandwidth

The memory latency on Ivy Bridge is essentially unchanged from the previous microarchitecture (Core i7 Lynnfield/Gulftown), but bandwidth has increased significantly, despite using the same memory (DDR3 ~1600).

FPGA CAD

All around per-clock performance improvements of nearly 15% in Ivy Bridge. Stock clock speeds are up about 15% vs. the top-binned Lynnfield (Core i7-880) too. It’s strange how VPR seems to behave differently from Quartus. Quartus placement improves more than clustering and routing over several processor generations, but VPR placement improves less.

Other Benchmarks

These tests were run on the same systems as before, but clock speeds are lower. These are the same benchmarks used in the next section on simultaneous multithreading.

The per-clock performance of Ivy Bridge is 20% better than both Lynnfield and Yorkfield. Surprisingly, Lynnfield doesn’t seem much better than its preceding generation Yorkfield (2% CPI) on these workloads.

The runtime comparisons include the impact of the processors’ clock speeds. The Core 2 (Yorkfield) and Core i7 (Lynnfield) systems are at stock speed, while Ivy Bridge is overclocked by 11%. The graph shows Ivy Bridge a good 60% faster than both earlier chips, which would still be near 50% if all chips were not overclocked. At stock clock speeds, 20% of this gain comes from microarchitectural improvements, and 25% from increased clock speeds.

Hyper-Theading Performance

This section repeats the tests done earlier for Lynnfield’s Hyper-Threading: Hyper-Threading Performance. The Lynnfield results are taken from the measurements made for the earlier tests.

Hardware

Workloads

Throughput Scaling with Multiple Threads

With the exception of the kernel compile workload, all of these tests start multiple instances of the same task and measures the total throughput of the processor (number of tasks/average runtime for task).

As expected, total throughput increases near linearly with the number of cores used up to 4 (cores are relatively independent), throughput increases slowly between 4 and 8 thread contexts used (Hyper-threading thread contexts are not equivalent to full processors), and is roughly flat beyond 8 thread contexts (time-slicing by the OS does not improve throughput.

Hyper-Threading Throughput Scaling

The first chart focuses on comparing the throughput at 8 threads vs. 4 threads for the different workloads. The median geometric mean improvement for HT is 23%. The pathological Dhrystone workload has improved: Although Dhrystone still does not benefit from Hyper-threading, it is no longer slower. It seems like Ivy Bridge gains slightly less from Hyper-threading than Lynnfield. This is not necessarily a bad thing: It could be a symptom that Ivy Bridge is doing a better job utilizing the pipeline with just one thread, reducing the performance gain available for two threads.

The second chart compares the throughput at 4 threads vs. 1 thread. Ivy Bridge seems to be noticeably worse at this than Lynnfield.

There seems to be no correlation between workloads that scale well on real cores and those that scale well under Hyper-threading. The correlation between the two microarchitectures is higher: Workloads that scale well on Lynnfield tend to also scale well on Ivy Bridge, and vice versa.