平平淡淡从从容容

0%

Redis服务监控和压力测试

发表于 分类于 2.1k 2 分钟

通常来讲，缓存的命中率越高则表示使用缓存的收益越高，应用的性能越好(响应时间越短、吞吐量越高)，抗并发的能力越强。

一、常用工具

  1. redis-stat，几年前的老项目
  2. redmon，几年前的老项目
  3. redis-faina，几年前的老项目
  4. redis-broswer，几年前的老项目
  5. RedisLive，几年前的老项目，由python编写的并且开源的图形化监控工具，核心服务部分只包含一个web服务和一个基于redis自带的info及monitor命令的监控服务。
  6. zabbix
  7. promethuse
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
# Server
redis_version:6.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:cd01c408c5307915
redis_mode:standalone
os:Darwin 19.6.0 x86_64
arch_bits:64
multiplexing_api:kqueue
atomicvar_api:atomic-builtin
gcc_version:4.2.1
process_id:40616
run_id:f0c38cff282eea9724af6dd9a63646150fa8cd90
tcp_port:6379
uptime_in_seconds:85774
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:51027
executable:/usr/local/redis-6.0.7/./redis-server
config_file:/usr/local/redis-6.0.7/6379.conf
io_threads_active:0

# Clients
connected_clients:2
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:1162800
used_memory_human:1.11M
used_memory_rss:1429504
used_memory_rss_human:1.36M
used_memory_peak:1207648
used_memory_peak_human:1.15M
used_memory_peak_perc:96.29%
used_memory_overhead:1114108
used_memory_startup:1079728
used_memory_dataset:48692
used_memory_dataset_perc:58.61%
allocator_allocated:1116192
allocator_active:1391616
allocator_resident:1391616
total_system_memory:8589934592
total_system_memory_human:8.00G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.25
allocator_frag_bytes:275424
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.03
rss_overhead_bytes:37888
mem_fragmentation_ratio:1.28
mem_fragmentation_bytes:313312
mem_not_counted_for_evict:104
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:33972
mem_aof_buffer:104
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1627357908
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:232
aof_base_size:91
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:7
total_commands_processed:16690
instantaneous_ops_per_sec:0
total_net_input_bytes:234900
total_net_output_bytes:65226538
instantaneous_input_kbps:0.01
instantaneous_output_kbps:2.40
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:589
evicted_keys:0
keyspace_hits:97
keyspace_misses:9
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:467
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:16700
total_writes_processed:16694
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_replid:b1dd50279f85b1339917f1ae030f9dd1aca6db3d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:107.256007
used_cpu_user:28.073143
used_cpu_sys_children:0.006693
used_cpu_user_children:0.001512

# Modules
module:name=panda,ver=1,api=1,filters=0,usedby=[],using=[],options=[]

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=6,expires=0,avg_ttl=0

二、使用

  1. redis-stat

    • 安装

      • 通过Ruby安装
      • 通过Jar方式安装

    • 使用

      • 通过web页面
      • 通过终端命令行redis-stat --auth=your_password，无法启动运行，看报错信息意思是有的方法已经废弃了，只能通过web端查看
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      /Library/Ruby/Gems/2.6.0/gems/sinatra-1.3.6/lib/sinatra/base.rb:1070: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/sinatra-1.3.6/lib/sinatra/base.rb:1070: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/sinatra-1.3.6/lib/sinatra/base.rb:1070: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/sinatra-1.3.6/lib/sinatra/base.rb:1070: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/sinatra-1.3.6/lib/sinatra/base.rb:1070: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/si-0.1.4/lib/si/patch.rb:5: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/si-0.1.4/lib/si/patch.rb:9: warning: constant ::Bignum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/option_initializer-1.5.1/lib/option_initializer.rb:154: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/option_initializer-1.5.1/lib/option_initializer.rb:198: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/redis-3.0.7/lib/redis/client.rb:386: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/si-0.1.4/lib/si/module.rb:4: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/si-0.1.4/lib/si/module.rb:21: warning: constant ::Fixnum is deprecated
      /Library/Ruby/Gems/2.6.0/gems/si-0.1.4/lib/si/module.rb:10: warning: constant ::Fixnum is deprecated
      undefined method `display_width' for #<String:0x00007fe7c5225678>
      Did you mean?  display

  2. redmon

    • 安装sudo gem install redmon
    • 启动redmon
    • 查看http://0.0.0.0:4567/

  3. redis-faina

  4. redis-broswer

  5. RedisLive(较早的工具，后期已不见维护，调试时因为版本问题各种出错，暂时放弃)

    • 安装RedisLive,clone或直接下载zip压缩包

    • 安装依赖

      • tornado pip install tornado
      • redis.py pip install redis
      • python-dateutil pip install python-dateutil
        • 指定目录pip install tornado --target /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
        • 指定版本pip install -v tornado==2.1.1

    • cd RedisLive(zip解压后不叫此名字)

    • cp redis-live.conf.example redis-live.conf

    • 编辑redis-live.conf

      • RedisServers是要监控的实例
      • RedisStatsServer则是监控信息临时写入的实例
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    {
    	"RedisServers":
    	[ 
    		{
      			"server": "154.17.59.99",
      			"port" : 6379
    		},
    		
    		{
      			"server": "localhost",
      			"port" : 6380,
      			"password" : "some-password"
    		}		
    	],

    	"DataStoreType" : "redis",

    	"RedisStatsServer":
    	{
    		"server" : "ec2-184-72-166-144.compute-1.amazonaws.com",
    		"port" : 6385
    	},
    	
    	"SqliteStatsStore" :
    	{
    		"path":  "to your sql lite file"
    	}
    }

    • 启动./redis-monitor.py --duration=120

      • 报错，一般是由版本兼容导致的修改内容如下redisprovider.py，改完后重试正常或直接修改文件开头#! /usr/bin/env python3.8
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      #30行
      # self.conn.zadd(server + ":memory", str(timeutils.convert_to_epoch(timestamp)), data)
      self.conn.zadd(server + ":memory", data)

      #60行
      command_count_key = server + ":CommandCount:" + epoch
      # pipeline.zincrby(command_count_key, command, 1)
      pipeline.zincrby(command_count_key, 1, command)

      command_count_key = server + ":DailyCommandCount:" + current_date
      # pipeline.zincrby(command_count_key, command, 1)
      pipeline.zincrby(command_count_key, 1, command)

      key_count_key = server + ":KeyCount:" + epoch
      # pipeline.zincrby(key_count_key, keyname, 1)
      pipeline.zincrby(key_count_key, 1, keyname)

      key_count_key = server + ":DailyKeyCount:" + current_date
      # pipeline.zincrby(key_count_key, keyname, 1)
      pipeline.zincrby(key_count_key, 1, keyname)
      • 扩展
      1
      2
      3
      4
      5
      6
      7
      python与redis旧版本数据库的交互：
      zadd: db.zadd(REDIS_KEY, score, member)
      zincrby: db.zincrby(REDIS_KEY, member, increment)

      python与redis新版本数据库交互：
      zadd：db.zadd(REDIS_KEY, {member:score})
      zincrby：db.zincrby(REDIS_KEY, increment, menber)

    • 启动./redis-live.py

      • 报错
        • 直接修改文件开头#! /usr/bin/env python3.8
      1
      2
      3
      4
      5
      6
      7
      Traceback (most recent call last):
        File "./redis-live.py", line 3, in <module>
          import tornado.ioloop
        File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tornado/ioloop.py", line 67
          def fileno(self) -> int:
                           ^
      SyntaxError: invalid syntax
      • 报错
        • 加入以下代码
          1
          2
          3
          import sys
          sys.path.append("api/controller")
          sys.path.append("dataprovider")
      1
      2
      3
      4
      5
      6
      Traceback (most recent call last):
        File "./redis-live.py", line 15, in <module>
          from api.controller.ServerListController import ServerListController
        File "/usr/local/RedisLive/src/api/controller/ServerListController.py", line 1, in <module>
          from BaseController import BaseController
      ModuleNotFoundError: No module named 'BaseController'

      • 报错TabError: inconsistent use of tabs and spaces in indentation

        • 修改api/util/timeutils.py
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        def convert_to_epoch(timestamp):
            if (type(timestamp) is datetime.date):
                timestamp = datetime.datetime.fromordinal(timestamp.toordinal())
                timestamp = timestamp.replace(tzinfo=None)
                diff = (timestamp - datetime.datetime(1970, 1, 1))
                seconds = int(total_seconds(diff))
                return seconds

        def convert_to_epoch1(timestamp):
            if (type(timestamp) is datetime.date):
                timestamp = datetime.datetime.fromordinal(timestamp.toordinal())
                timestamp = timestamp.replace(tzinfo=None)
            diff = (timestamp - datetime.datetime(1970, 1, 1))
            seconds = int(total_seconds(diff))
            return seconds

      • 报错TypeError: '>' not supported between instances of 'NoneType' and 'int'

  6. zabbix

    • 安装MySQ，建议5.7版本

    • 安装zabbix server

      • 下载源码包wget https://cdn.zabbix.com/zabbix/sources/stable/4.0/zabbix-4.0.32.tar.gz
      • 解压tar -zxvf zabbix-4.0.32.tar.gz
      • cd zabbix-4.0.32

  7. promethuse

三、参考

  1. 参考一
  2. 参考二
  3. 参考三
  4. 参考四
  5. 参考五

一、概念

      传统上所谓压力测试(stress testing)是指将整个金融机构或资产组合置于某一特定的（主观想象的）极端市场情况下，如假设利率骤升100个基本点，某一货币突然贬值30%，股价暴跌20%等异常的市场变化，然后测试该金融机构或资产组合在这些关键市场变量突变的压力下的表现状况，看是否能经受得起这种市场的突变。

      在软件测试中，压力测试(Stress Test)，也称为强度测试、负载测试，通过模拟实际应用的软硬件环境及用户使用过程的系统负荷，长时间或超大负荷地运行测试软件，来测试被测系统的性能、可靠性、稳定性等。

redis做压测可以用自带的redis-benchmark工具，用法如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]

 -h <hostname>      Server hostname (default 127.0.0.1)
 -p <port>          Server port (default 6379)
 -s <socket>        Server socket (overrides host and port)
 -a <password>      Password for Redis Auth
 --user <username>  Used to send ACL style 'AUTH username pass'. Needs -a.
 -c <clients>       Number of parallel connections (default 50)
 -n <requests>      Total number of requests (default 100000)
 -d <size>          Data size of SET/GET value in bytes (default 3)
 --dbnum <db>       SELECT the specified db number (default 0)
 --threads <num>    Enable multi-thread mode.
 --cluster          Enable cluster mode.
 --enable-tracking  Send CLIENT TRACKING on before starting benchmark.
 -k <boolean>       1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD,
                    random members and scores for ZADD.
  Using this option the benchmark will expand the string __rand_int__
  inside an argument with a 12 digits number in the specified range
  from 0 to keyspacelen-1. The substitution changes every time a command
  is executed. Default tests use this to hit random keys in the
  specified range.
 -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
 -e                 If server replies with errors, show them on stdout.
                    (no more than 1 error per second is displayed)
 -q                 Quiet. Just show query/sec values
 --precision        Number of decimal places to display in latency output (default 0)
 --csv              Output in CSV format
 -l                 Loop. Run the tests forever
 -t <tests>         Only run the comma separated list of tests. The test
                    names are the same as the ones produced as output.
 -I                 Idle mode. Just open N idle connections and wait.

Examples:

 Run the benchmark with the default configuration against 127.0.0.1:6379:
   $ redis-benchmark

 Use 20 parallel clients, for a total of 100k requests, against 192.168.1.1:
   $ redis-benchmark -h 192.168.1.1 -p 6379 -n 100000 -c 20

 Fill 127.0.0.1:6379 with about 1 million keys only using the SET test:
   $ redis-benchmark -t set -n 1000000 -r 100000000

 Benchmark 127.0.0.1:6379 for a few commands producing CSV output:
   $ redis-benchmark -t ping,set,get -n 100000 --csv

 Benchmark a specific command line:
   $ redis-benchmark -r 10000 -n 10000 eval 'return redis.call("ping")' 0

 Fill a list with 10000 random elements:
   $ redis-benchmark -r 10000 -n 10000 lpush mylist __rand_int__

 On user specified command lines __rand_int__ is replaced with a random integer
 with a range of values selected by the -r option.

二、使用

  1. 50个客户端，并发10000请求:redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 10000 -t get -a 123456
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
====== GET ======
  10000 requests completed in 0.14 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 900 1 300 10 60 10000
  host configuration "appendonly": no
  multi-thread: no

0.01% <= 0.2 milliseconds
4.93% <= 0.3 milliseconds
36.01% <= 0.4 milliseconds
61.16% <= 0.5 milliseconds
89.49% <= 0.6 milliseconds
94.82% <= 0.7 milliseconds
96.98% <= 0.8 milliseconds
97.92% <= 0.9 milliseconds
98.47% <= 1.0 milliseconds
99.04% <= 1.1 milliseconds
99.43% <= 1.2 milliseconds
99.69% <= 1.3 milliseconds
99.81% <= 1.4 milliseconds
99.91% <= 1.5 milliseconds
100.00% <= 1.5 milliseconds
72992.70 requests per second
  1. 50个客户端，并发10000请求，4线程处理(6.0.0):redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 10000 -t get -a 123456 --threads 4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
====== GET ======
  10000 requests completed in 0.12 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 900 1 300 10 60 10000
  host configuration "appendonly": no
  multi-thread: yes
  threads: 4

0.03% <= 0.1 milliseconds
0.80% <= 0.2 milliseconds
3.29% <= 0.3 milliseconds
10.22% <= 0.4 milliseconds
36.53% <= 0.5 milliseconds
82.34% <= 0.6 milliseconds
90.32% <= 0.7 milliseconds
94.44% <= 0.8 milliseconds
96.65% <= 0.9 milliseconds
98.01% <= 1.0 milliseconds
98.92% <= 1.1 milliseconds
99.32% <= 1.2 milliseconds
99.56% <= 1.3 milliseconds
99.75% <= 1.4 milliseconds
99.92% <= 1.5 milliseconds
99.97% <= 1.6 milliseconds
100.00% <= 1.7 milliseconds
85470.09 requests per second
  1. 只统计最终结果redis-benchmark -h 127.0.0.1 -p 6379 -a 123456 -q
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PING_INLINE: 69930.07 requests per second
PING_BULK: 56689.34 requests per second
SET: 60422.96 requests per second
GET: 60753.34 requests per second
INCR: 61087.36 requests per second
LPUSH: 53619.30 requests per second
RPUSH: 58754.41 requests per second
LPOP: 53590.57 requests per second
RPOP: 56625.14 requests per second
SADD: 56274.62 requests per second
HSET: 56497.18 requests per second
SPOP: 56882.82 requests per second
ZADD: 52994.17 requests per second
ZPOPMIN: 56882.82 requests per second
LPUSH (needed to benchmark LRANGE): 54975.26 requests per second
LRANGE_100 (first 100 elements): 13368.98 requests per second
LRANGE_300 (first 300 elements): 5611.36 requests per second
LRANGE_500 (first 450 elements): 3538.19 requests per second
LRANGE_600 (first 600 elements): 2504.45 requests per second
MSET (10 keys): 30293.85 requests per second

三、参考

  1. 参考一