0%

Redis之内存分配器

Redis在编译时便会指定内存分配器,内存分配器可以是libc、jemalloc、tcmalloc,默认是libc(Mac下)。

一、基础

  1. libc:泛指,凡是符合实现了C标准规定的内容都是一种libc,如glic。

    • Redis默认内存分配器(Mac下)
    • glibc是Linux下C标准库的实现,全称GNU C Library。glibc本身是GNU旗下的C标准库,后来逐渐成为了Linux的标准C库,而Linux原来的标准C库libc逐渐不再被维护。
      • GNU计划,有译为“革奴计划”,是由理查德·斯托曼在1983年9月27日公开发起的自由软件集体协作计划,它的目标是创建一套完全自由的操作系统GNU。
  2. jemalloc:FreeBSD的内存分配器,最大优势在于多线程情况下的高性能以及内存碎片的减少,在Facebook有广泛应用。

    • 安装时可指定make MALLOC=jemalloc
  3. tcmalloc:google开发的内存分配器,据称它的内存分配速度是glibc2.3中实现的malloc的数倍。

    • 安装时可指定(Mac下)
      • brew install gperftools
      • make MALLOC=tcmalloc

源码README.md

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Allocator
---------

Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.

To force compiling against libc malloc, use:

% make MALLOC=libc

To compile against jemalloc on Mac OS X systems, use:

% make MALLOC=jemalloc

二、延伸

  1. 同时安装三个类型分配器,端口依次为6379、7379、8379
  2. 使用info memory查看信息(6.2.5),传送门
参数 libc jemalloc tcmalloc
used_memory 1125632 1068768 1074984
used_memory_human 1.07M 1.02M 1.03M
used_memory_rss 2105344 2621440 2367488
used_memory_rss_human 2.01M 2.50M 2.26M
used_memory_peak 1186016 1128640 1139000
used_memory_peak_human 1.13M 1.08M 1.09M
used_memory_peak_perc 94.91% 94.70% 94.38%
used_memory_overhead 1080032 1027264 1033472
used_memory_startup 1062592 1006664 1012976
used_memory_dataset 45600 41504 41512
used_memory_dataset_perc 72.34% 66.83% 66.95%
allocator_allocated 1080512 1237680 1033952
allocator_active 2067456 1581056 2329600
allocator_resident 2067456 4722688 2329600
total_system_memory 17179869184 17179869184 17179869184
total_system_memory_human 16.00G 16.00G 16.00G
used_memory_lua 37888 37888 37888
used_memory_lua_human 37.00K 37.00K 37.00K
used_memory_scripts 0 0 0
used_memory_scripts_human 0B 0B 0B
number_of_cached_scripts 0 0 0
maxmemory 0 0 0
maxmemory_human 0B 0B 0B
maxmemory_policy noeviction noeviction noeviction
allocator_frag_ratio 1.91 1.28 2.25
allocator_frag_bytes 986944 343376 1295648
allocator_rss_ratio 1.00 2.99 1.00
allocator_rss_bytes 0 3141632 0
rss_overhead_ratio 1.02 0.56 1.02
rss_overhead_bytes 37888 -2101248 37888
mem_fragmentation_ratio 1.95 2.55 2.29
mem_fragmentation_bytes 1024832 1593696 1333536
mem_not_counted_for_evict 0 0 0
mem_replication_backlog 0 0 0
mem_clients_slaves 0 0 0
mem_clients_normal 17440 20496 20496
mem_aof_buffer 0 0 0
mem_allocator libc jemalloc-5.1.0 tcmalloc-2.9
active_defrag_running 0 0 0
lazyfree_pending_objects 0 0 0
lazyfreed_objects 0 0 0
  • used_memory:由Redis分配器分配的内存总量,包含了redis进程内部的开销和数据占用的内存,以字节为单位。
  • used_memory_human:以人类更直观的单位展示分配的内存总量。
  • used_memory_rss:向操作系统申请的内存大小,值一般大于used_memory,因为Redis的内存分配器会产生内存碎片。
  • used_memory_rss_human:以人类更直观的单位展示向操作系统申请的内存大小。
  • used_memory_peak:Redis的内存消耗峰值,以字节为单位。
  • used_memory_peak_human:以人类更直观的单位展示内存消耗峰值。
  • used_memory_peak_perc:使用内存达到峰值内存的百分比,即(used_memory/used_memory_peak)*100%
  • used_memory_overhead:Redis维护数据集的内部机制所需的内存开销,包括所有客户端输出缓冲区、查询缓冲区、AOF重写缓冲区和主从复制backlog等。
  • used_memory_startup:服务启动时消耗的内存。
  • used_memory_dataset:数据占用的内存大小,即used_memory-used_memory_overhead
  • used_memory_dataset_perc:数据占用的内存大小的百分比,(used_memory_dataset/(used_memory-used_memory_startup))*100%
  • allocator_allocated:Total bytes allocated form the allocator, including internal-fragmentation. Normally the same as used_memory.
  • allocator_active:Total bytes in the allocator active pages, this includes external-fragmentation.
  • allocator_resident:Total bytes resident (RSS) in the allocator, this includes pages that can be released to the OS (by MEMORY PURGE, or just waiting).
  • total_system_memory:系统内存总量,以字节为单位。
  • total_system_memory_human:以人类更直观的单位展示系统内存总量。
  • used_memory_lua:Lua脚本存储占用的内存,以字节为单位。
  • used_memory_lua_human:Lua脚本存储占用的内存,以MB/KB为单位。
  • used_memory_scripts:Number of bytes used by cached Lua scripts.
  • used_memory_scripts_human:Human readable representation of previous value
  • number_of_cached_scripts:
  • maxmemory:Redis实例的最大内存配置,设置的最大内存。
  • maxmemory_human:Redis实例的最大内存配置,设置的最大内存。
  • maxmemory_policy:当达到maxmemory时的淘汰策略。
  • allocator_frag_ratio:Ratio between allocator_active and allocator_allocated. This is the true (external) fragmentation metric (not mem_fragmentation_ratio).
  • allocator_frag_bytes:Delta between allocator_active and allocator_allocated. See note about mem_fragmentation_bytes.
  • allocator_rss_ratio:Ratio between allocator_resident and allocator_active. This usually indicates pages that the allocator can and probably will soon release back to the OS.
  • allocator_rss_bytes:Delta between allocator_resident and allocator_active
  • rss_overhead_ratio:Ratio between used_memory_rss (the process RSS) and allocator_resident. This includes RSS overheads that are not allocator or heap related.
  • rss_overhead_bytes:Delta between used_memory_rss (the process RSS) and allocator_resident.
  • mem_fragmentation_ratio:碎片率,约等于used_memory_rss/used_memory
  • mem_fragmentation_bytes: Delta between used_memory_rss and used_memory. Note that when the total fragmentation bytes is low (few megabytes), a high ratio (e.g. 1.5 and above) is not an indication of an issue.
  • mem_not_counted_for_evict:
  • mem_replication_backlog:
  • mem_clients_slaves:
  • mem_clients_normal:
  • mem_aof_buffer:
  • mem_allocator:内存分配器。
  • active_defrag_running:0表示没有活动的defrag任务正在运行,1则相反(defrag意为内存碎片整理)。
  • lazyfree_pending_objects:0表示不存在延迟释放的挂起对象,1则相反。
  • lazyfreed_objects:
  1. 使用redis-benchmark压测对比内存碎片率

三、参考

  1. 参考一
  2. 参考二
  3. 参考三
  4. 参考四
  5. 参考五
  6. 参考六