[Redis开发规范解析(三)--一个Redis最好存多少key]一文中详细介绍了Rehash的基本原理和可能产生的问题,这里不做详细介绍。
简单回顾下超出key阈值后,需要额外的hash表内存大小:
键值个数 | 需要额外的hash表内存大小 |
---|---|
134,217,728 | 2GB |
67,108,864 | 1GB |
33,554,432 | 512.0MB |
16,777,216 | 256.0MB |
8,388,608 | 128.0MB |
4,194,304 | 64.0MB |
2,097,152 | 32.0MB |
1,048,576 | 16.0MB |
524,288 | 8.0MB |
随着key个数越多,rehash需要的额外内存也越大,所带来的可用性风险(大量逐出,Redis同步)和数据丢失(逐出)风险也越高。
Redis 6.2后对Rehash做了相关优化:
Limit the main db dictionaries expansion to prevent key eviction (#7954)
In the past big dictionary rehashing could result in massive data eviction.Now this rehashing is delayed (up to a limit), which can result in performance loss due to hash collisions.
简单翻译:
在大的hash表(键值数多)场景下,rehash会延迟执行防止大量数据丢失和可用性问题。
第一次灌入:67,100,000个key,观察内存:
maxmemory: 5.6GB
used_memory_human: 5.5GB
dbsize: 67,100,000
第二次灌入:2,000,000个key,观察可用性和逐出
客户端执行超时, 超时近30秒以上,redis-cli --latency-history
min: 0, max: 1, avg: 0.06 (1485 samples) -- 15.01 seconds range
min: 0, max: 36511, avg: 45.63 (801 samples) -- 44.61 seconds range
逐出大量key:500万以上
evicted_keys:5,909,376
峰值内存:rehash多了一个GB
used_memory_peak_human:6.50G
第一次灌入:67,100,000个key,观察内存:
maxmemory: 5.6GB
used_memory_human: 5.5GB
dbsize: 67,100,000
第二次灌入:2,000,000个key,观察可用性和逐出
客户端执行无超时,redis-cli --latency-history
min: 0, max: 2, avg: 0.05 (1484 samples) -- 15.01 seconds range
min: 0, max: 3, avg: 0.24 (1454 samples) -- 15.00 seconds range
evicted_keys:152485
evicted_keys:443253
evicted_keys:751165
evicted_keys:1058191
evicted_keys:1367445
evicted_keys:1662485
evicted_keys:1662485
峰值内存:没有进行rehash
used_memory_peak_human:5.50G
版本 | 是否发生rehash | 严重超时 |
---|---|---|
6.0.15 | 有 | 有,30秒以上不可用 |
7.0.11 | 无 | 无,正常逐出 |
新加的expandAllowed,决定dict当前是否做rehash
typedef struct dictType {
....
int (*expandAllowed)(size_t moreMem, double usedRatio);
....
} dictType;
dict在做扩容时候会加入新的判断dictTypeExpandAllowed
/ Expand the hash table if needed /
static int _dictExpandIfNeeded(dict *d)
{
......
if (!dictTypeExpandAllowed(d))
return DICT_OK;
......
}
若dict的expandAllowed为空则允许rehash,否则执行expandAllowed
static int dictTypeExpandAllowed(dict *d) {
if (d->type->expandAllowed == NULL) return 1;
return d->type->expandAllowed(
DICTHT_SIZE(_dictNextExp(d->ht_used[0] + 1)) sizeof(dictEntry),
(double)d->ht_used[0] / DICTHT_SIZE(d->ht_size_exp[0]));
}
redis的dbDictType和dbExpiresDictType
/ Db->dict, keys are sds strings, vals are Redis objects. /
dictType dbDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
dictSdsDestructor, /* key destructor */
dictObjectDestructor, /* val destructor */
dictExpandAllowed, /* allow to expand */
dictEntryMetadataSize /* size of entry metadata in bytes */
};
/ Db->expires /
dictType dbExpiresDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
NULL, /* key destructor */
NULL, /* val destructor */
dictExpandAllowed /* allow to expand */
};
可以看到:
dict使用率小于hash负载因子:
如果当前对象内存(使用内存 - 额外内存(各种缓冲区)) + rehash需要内存小于maxmemory,可以做rehash
int dictExpandAllowed(size_t moreMem, double usedRatio) {
if (usedRatio <= HASHTABLE_MAX_LOAD_FACTOR) {
return !overMaxmemoryAfterAlloc(moreMem);
} else {
return 1;
}
}
/* Return 1 if used memory is more than maxmemory after allocating more memory,
int overMaxmemoryAfterAlloc(size_t moremem) {
if (!server.maxmemory) return 0; /* No limit. */
/* Check quickly. */
size_t mem_used = zmalloc_used_memory();
if (mem_used + moremem <= server.maxmemory) return 0;
size_t overhead = freeMemoryGetNotCountedMemory();
mem_used = (mem_used > overhead) ? mem_used - overhead : 0;
return mem_used + moremem > server.maxmemory;
}
如果觉得我的文章对您有用,请点赞。您的支持将鼓励我继续创作!
赞4
添加新评论0 条评论