Memory hierarchy

The hierarchical arrangement of storage in current computer architectures is called the memory hierarchy. It is designed to take advantage of memory locality in computer programs. Each level of the hierarchy has the properties of higher bandwidth, smaller size, and lower latency than lower levels.

Most modern CPUs are so fast that for most program workloads, the locality of reference of memory accesses and the efficiency of the caching and memory transfer between different levels of the hierarchy are the practical limitation on processing speed
As a result, the CPU spends much of its time idling, waiting for memory I/O to complete. This is sometimes called the space cost, as a larger memory object is more likely to overflow a small/fast level and require use of a larger/slower level.

The memory hierarchy in most computers is:

Diagram of the computer memory hierarchy

Processor registers – fastest possible access (usually 1 CPU cycle), only hundreds of bytes in size
Level 1 (L1) cache – often accessed in just a few cycles, usually tens of kilobytes
Level 2 (L2) cache – higher latency than L1 by 2× to 10×, often 512 KiB or more
Level 3 (L3) cache – higher latency than L2, often 2048 KiB or more
Main memory (DRAM) – may take hundreds of cycles, but can be multiple gigabytes. Access times may not be uniform, in the case of a NUMA machine.
Disk storage – millions of cycles latency, but very large
Tertiary storage – several seconds latency, can be huge

The various major units in a typical memory system can be viewed as forming a hierarchy of memories (m1,m2,...,mn) in which each member mi is in a sense subordinate to the next highest member mi-1 of the hierarchy.

Management

Modern programming languages mainly assume two levels of memory, main memory and disk storage, though in assembly language, and in inline assembler in languages such as C, registers can be directly accessed. Taking optimal advantage of the memory hierarchy requires the cooperation of programmers, hardware, and compilers (as well as underlying support from the operating system):
Programmers are responsible for moving data between disk and memory through file I/O.
Hardware is responsible for moving data between memory and caches.
Optimizing compilers are responsible for generating code that, when executed, will cause the hardware to use caches and registers efficiently.

http://en.wikipedia.org/wiki/Memory_hierarchy