Why GDDR Memory Might Be the Next Big Step in Computer Architecture for AI

No, this is not a gaming blog, but gaming is where we’ll start. If you happen to be into PC gaming, then you’ll know that the best graphics cards for gaming usually contain a class of dynamic access memory called GDDR (e.g. GDDR6). A single graphics card can have tens of gigabytes of local GDDR6 memory that is connected to the GPU and that is independent of the main memory on the motherboard. In this article, we describe what GDDR is and why it’s so important.

Spoiler alert: the answer is not because of gaming.

What Is GDDR Memory?

GDDR stands for Graphics Double Data Rate, so from the name, you can see that it’ is a special kind of Double Data Rate memory. It has an array of memory cells for storing bits (ones and zeros), and it’s organized in rows and columns – very much like a regular DDR memory found in a PC motherboard or high-performance laptop. The difference lies in how fast the cells can be accessed internally inside the memory IC and also how fast the data can be transferred from the memory IC to the host processor that is connected to it. This is illustrated in Figure 1, where we show a high-level block diagram of the architecture of a GDDR memory.

As can be seen, the device contains the memory bank or banks (storing the 1s and 0s). The process of reading from or writing to a location in the memory bank is enabled by a command and address decoding block, and this block works in conjunction with certain row access and column access circuitry. Specifically, when a certain address of the memory is being read from by the host controller, these row access and column access circuits are activated very rapidly in order to pinpoint a single coordinate (i.e. an address that is very much like the address you enter on your favorite Maps App on your smartphone) or a set of coordinates in the memory. Then, the readout circuitry is used to fetch the data in these locations and send it through the data input/output logic block.

The above is essentially how a regular DDR memory works as well – hence the similarity between GDDR and DDR. The difference with GDDR is that it’ is designed to be blisteringly fast. The contents of the memory could be vectors for rendering a high-resolution image for example. The process of rendering very high-performance gaming graphics requires repetitive access to large amounts of data and performing repetitive computations on this data at very high speeds.

A typical connection between a Graphics Processing Unit (GPU) and a GDDR memory is shown in Figure 2.

To render graphics on the user’s display, the GPU performs repetitive numerical computations on large sets of data that represent the images being displayed, and this data is stored in the GDDR memory. The computations themselves are less flexible than general-purpose calculations in a general CPU, and this is exactly why they can run at much higher speeds than in a regular CPU. 

The result of performing repetitive and similar calculations on large sets of data translates into a series of read and write transactions to the memory. And these transactions occur through the transfer of high-speed digital data over these small interconnection wires between the GPU and the memory. For example, a modern GDDR6 link can easily rely on 16 Gbps SerDes-like speeds but across a large number of pins.

Technical Differences Between GDDR and DDR Memories

So, GDDR memory is similar to DDR memory, but it’s not the same. Here, we show three technical differences between the two types of memories.

Bandwidth

We already alluded to this above. A typical GDDR6 DRAM device will run at 16 Gbps per pin. And with two memory channels per DRAM, each containing 16 DQ pins, this translates to a total throughput of 512 Gbps on a single DRAM device. By contrast, a state of the art DDR5 DRAM device can operate at 6.4 Gbps per pin. 

Routing Style and Connection Scheme

Apart from internal access architecture differences between the two memory classes, one advantage that GDDR memories have over DDR memories is that they are usually connected in a point-to-point configuration instead of a multi-drop configuration. That is, each host controller port is connected to only one single memory device. Figure 3 shows an explanation of the difference between the two connection schemes.

Because DDR memories are connected in a multi-drop configuration, the PCB trace between the memory controller and any given DRAM device is a lot less pristine than a corresponding trace connected between a memory controller and a GDDR memory. The DDR trace often has discontinuities due to the inevitable impedance mismatches in all the loads that are attached to it, and this results in signal reflections and waveform distortions. This in turn limits the maximum clock speed that the DDR trace can be operated at.

On the other hand, a GDDR memory’s address bus is only connected to its corresponding pins on the GPU. So, at least in principle, there’ is a larger opportunity to optimize the signal integrity of the trace and ensure that there are no discontinuities on it. On the flip side, the much higher bandwidth of the trace in GDDR makes it more sensitive to even small geometry imperfections. Even seemingly insignificant layout errors like forgetting to remove non-functional pads in a via can result in performance degradation for GDDR links.

Protocol Differences

The DDR protocol is different from the GDDR protocol, and this means that the host controller design is different. The details of these differences are beyond the scope of this article, but they include the following:

• Different numbers and types of forward clocks
• Different command and address bus widths
• Different data and strobe arrangements

What Makes GDDR so Important?

If you’ve taken a neural network class at university, the end-of-year assignment would have probably been performed on a graphics card. This is exactly why GDDR is becoming so important these days. As it turns out, the computations required to train or run a neural network are very similar in concept to the computations required for video processing in a gaming application. Namely, repetitive and similar computations must be performed on large sets of data and at a very fast pace. This means two things:

1. GPU architectures are best suited for artificial intelligence (AI) applications and machine learning (ML); not CPU architectures
2. GDDR memories (with their high bandwidth and optimized architecture) are best suited for AI and ML; not DDR memories

Additionally, with a tremendous push to add AI processing even on mobile terminals, the computer architecture is changing drastically. While there are other ways to create high-bandwidth memory implementations, GDDR stands to be a great low-cost alternative (compared to the other competing solutions).

Apart from AI, many networking and automotive applications now require faster memory architectures. To put it simply, processing power in CPUs and GPUs has far outpaced the ability to increase transfer speeds to/from memories. So, the memory architecture is the bottleneck now in several applications such as automotive, networking, and industrial. GDDR memories represent a great potential solution to address this gap. They offer the best of both worlds:

• They are much faster than DDR memory
• They are much more cost effective than exotic memory solutions such as High-Bandwidth Memory (HBM)

Are you developing some of the latest high speed digital applications? At Introspect, we design tools to accelerate the creation of upcoming technologies. To learn more, send us an email at info@introspect.ca.