å¸éè¿æ»¤å¨çåºç¨
æ们å æ¥çä¸å¸éè¿æ»¤å¨çåºç¨åºæ¯ï¼è®©å¤§å®¶ç¥éç¥å¥çå¸éè¿æ»¤å¨å°åºè½åä»ä¹ã
ç¼åç©¿é
æ们ç»å¸¸ä¼æä¸é¨åæ°æ®æ¾å¨Redisçç¼åï¼æ¯å¦äº§å详æ ã
è¿æ ·ææ¥è¯¢è¯·æ±è¿æ¥ï¼æ们å¯ä»¥æ ¹æ®äº§åIdç´æ¥å»ç¼åä¸åæ°æ®ï¼èä¸ç¨è¯»åæ°æ®åºï¼è¿æ¯æåæ§è½æç®åï¼ææ®éï¼ä¹æ¯æææçåæ³ã
ä¸è¬çæ¥è¯¢è¯·æ±æµç¨æ¯è¿æ ·çï¼å æ¥ç¼åï¼æç¼åçè¯ç´æ¥è¿åï¼å¦æç¼åä¸æ²¡æï¼åå»æ°æ®åºæ¥è¯¢ï¼ç¶ååææ°æ®åºååºæ¥çæ°æ®æ¾å ¥ç¼åï¼ä¸åçèµ·æ¥å¾ç¾å¥½ã
ä½æ¯å¦æç°å¨æ大é请æ±è¿æ¥ï¼èä¸é½å¨è¯·æ±ä¸ä¸ªä¸åå¨ç产åIdï¼ä¼åçä»ä¹ï¼
æ¢ç¶äº§åIdé½ä¸åå¨ï¼é£ä¹è¯å®æ²¡æç¼åï¼æ²¡æç¼åï¼é£ä¹å¤§éç请æ±é½æ¼å°æ°æ®åºï¼æ°æ®åºçååä¸ä¸åå°±ä¸æ¥äºï¼è¿æå¯è½ææ°æ®åºææ»ã
è½ç¶æå¾å¤åæ³é½å¯ä»¥è§£å³è¿é®é¢ï¼ä½æ¯æ们ç主è§æ¯âå¸éè¿æ»¤å¨âï¼æ²¡éï¼âå¸éè¿æ»¤å¨âå°±å¯ä»¥è§£å³ï¼ç¼è§£ï¼ç¼åç©¿éé®é¢ãè³äºä¸ºä»ä¹è¯´æ¯âç¼è§£âï¼çä¸å»ä½ å°±æç½äºã
大éæ°æ®ï¼å¤æç»å®çæ¯å¦å¨å ¶ä¸
ç°å¨æ大éçæ°æ®ï¼èè¿äºæ°æ®ç大å°å·²ç»è¿è¿è¶ åºäºæå¡å¨çå åï¼ç°å¨åç»ä½ ä¸ä¸ªæ°æ®ï¼å¦ä½å¤æç»ä½ çæ°æ®å¨ä¸å¨å ¶ä¸ãå¦ææå¡å¨çå å足å¤å¤§ï¼é£ä¹ç¨HashMapæ¯ä¸ä¸ªä¸éç解å³æ¹æ¡ï¼ç论ä¸çæ¶é´å¤æ度å¯ä»¥è¾¾å°O(1)ï¼ä½æ¯ç°å¨æ°æ®ç大å°å·²ç»è¿è¿è¶ åºäºæå¡å¨çå åï¼æ以æ æ³ä½¿ç¨HashMapï¼è¿ä¸ªæ¶åå°±å¯ä»¥ä½¿ç¨âå¸éè¿æ»¤å¨âæ¥è§£å³è¿ä¸ªé®é¢ãä½æ¯è¿æ¯åæ ·çï¼ä¼æä¸å®çâ误å¤çâã
ä»ä¹æ¯å¸éè¿æ»¤å¨
å¸éè¿æ»¤å¨æ¯ä¸ä¸ªå«âå¸éâç人æåºçï¼å®æ¬èº«æ¯ä¸ä¸ªå¾é¿çäºè¿å¶åéï¼æ¢ç¶æ¯äºè¿å¶çåéï¼é£ä¹æ¾èæè§çï¼åæ¾çä¸æ¯0ï¼å°±æ¯1ã
ç°å¨æ们æ°å»ºä¸ä¸ªé¿åº¦ä¸º16çå¸éè¿æ»¤å¨ï¼é»è®¤å¼é½æ¯0
ç°å¨éè¦æ·»å ä¸ä¸ªæ°æ®ï¼
æ们éè¿æç§è®¡ç®æ¹å¼ï¼æ¯å¦Hash1ï¼è®¡ç®åºäºHash1(æ°æ®)=5ï¼æ们就æä¸æ 为5çæ ¼åæ¹æ1
æ们åéè¿æç§è®¡ç®æ¹å¼ï¼æ¯å¦Hash2ï¼è®¡ç®åºäºHash2(æ°æ®)=9
è¿æ¯éè¿æç§è®¡ç®æ¹å¼ï¼æ¯å¦Hash3ï¼è®¡ç®åºäºHash3(æ°æ®)=2
è¿æ ·ï¼åææ·»å çæ°æ®å°±å æ®äºå¸éè¿æ»¤å¨â5âï¼â9âï¼â2âä¸ä¸ªæ ¼åã
å¯ä»¥çåºï¼ä» ä» ä»å¸éè¿æ»¤å¨æ¬èº«èè¨ï¼æ ¹æ¬æ²¡æåæ¾å®æ´çæ°æ®ï¼åªæ¯è¿ç¨ä¸ç³»åéæºæ å°å½æ°è®¡ç®åºä½ç½®ï¼ç¶åå¡«å äºè¿å¶åéã
è¿æä»ä¹ç¨å¢ï¼
æ¯å¦ç°å¨åç»ä½ ä¸ä¸ªæ°æ®ï¼ä½ è¦å¤æè¿ä¸ªæ°æ®æ¯å¦éå¤ï¼ä½ æä¹åï¼
ä½ åªéå©ç¨ä¸é¢çä¸ç§åºå®ç计ç®æ¹å¼ï¼è®¡ç®åºè¿ä¸ªæ°æ®å æ®åªäºæ ¼åï¼ç¶åççè¿äºæ ¼åéé¢æ¾ç½®çæ¯å¦é½æ¯1ï¼å¦ææä¸ä¸ªæ ¼åä¸ä¸º1ï¼é£ä¹å°±ä»£è¡¨è¿ä¸ªæ°åä¸å¨å ¶ä¸ãè¿å¾å¥½ç解å§ï¼æ¯å¦ç°å¨åç»ä½ äºåæä½ æ·»å è¿å»çæ°æ®ï¼ä½ éè¿ä¸ç§åºå®ç计ç®æ¹å¼ï¼ç®åºçç»æè¯å®åä¸é¢çæ¯ä¸æ¨¡ä¸æ ·çï¼ä¹æ¯å æ®äºå¸éè¿æ»¤å¨â5âï¼â9âï¼â2âä¸ä¸ªæ ¼åã
ä½æ¯æä¸ä¸ªé®é¢éè¦æ³¨æï¼å¦æè¿äºæ ¼åéé¢æ¾ç½®çé½æ¯1ï¼ä¸ä¸å®ä»£è¡¨ç»å®çæ°æ®ä¸å®éå¤ï¼ä¹è®¸å ¶ä»æ°æ®ç»è¿ä¸ç§åºå®ç计ç®æ¹å¼ç®åºæ¥çç»æä¹æ¯ç¸åçãè¿ä¹å¾å¥½ç解å§ï¼æ¯å¦æ们éè¦å¤æ对象æ¯å¦ç¸çï¼æ¯ä¸å¯ä»¥ä» ä» å¤æä»ä»¬çåå¸å¼æ¯å¦ç¸ççã
ä¹å°±æ¯è¯´å¸éè¿æ»¤å¨åªè½å¤ææ°æ®æ¯å¦ä¸å®ä¸åå¨ï¼èæ æ³å¤ææ°æ®æ¯å¦ä¸å®åå¨ã
æçæ¥è¯´ï¼ä»ç»å®äºæ°å¢ãæ¥è¯¢çæµç¨ï¼å°±è¦ä»ç»å é¤çæµç¨äºï¼ä½æ¯å¾éæ¾çæ¯å¸éè¿æ»¤å¨æ¯å¾é¾åå°å é¤æ°æ®çï¼ä¸ºä»ä¹ï¼ä½ æ³æ³ï¼æ¯å¦ä½ è¦å é¤åæç»ä½ çæ°æ®ï¼ä½ æâ5âï¼â9âï¼â2âä¸ä¸ªæ ¼åé½æ¹æäº0ï¼ä½æ¯å¯è½å ¶ä»çæ°æ®ä¹æ å°å°äºâ5âï¼â9âï¼â2âä¸ä¸ªæ ¼ååï¼è¿ä¸å°±ä¹±å¥äºåï¼
ç¸ä¿¡ç»è¿æè¿ä¹ä¸ä»ç»ï¼å¤§å®¶å¯¹å¸éè¿æ»¤å¨åºè¯¥æä¸ä¸ªæµ æ¾ç认è¯äºï¼è³å°ä½ åºè¯¥æ¸ æ¥å¸éè¿æ»¤å¨çä¼ç¼ºç¹äºï¼
å¸éè¿æ»¤å¨çä¼ç¼ºç¹
ä¼ç¹ï¼ç±äºåæ¾çä¸æ¯å®æ´çæ°æ®ï¼æ以å ç¨çå åå¾å°ï¼èä¸æ°å¢ï¼æ¥è¯¢é度å¤å¿«ï¼
缺ç¹ï¼ éçæ°æ®çå¢å ï¼è¯¯å¤çéä¹å¢å ï¼æ æ³åå°å é¤æ°æ®ï¼åªè½å¤ææ°æ®æ¯å¦ä¸å®ä¸åå¨ï¼èæ æ³å¤ææ°æ®æ¯å¦ä¸å®åå¨ã
å¯ä»¥çå°ï¼å¸éè¿æ»¤å¨çä¼ç¹å缺ç¹ä¸æ ·ææ¾ã
å¨ä¸æä¸ï¼æ举çä¾åäºè¿å¶åéé¿åº¦ä¸º16ï¼ç±ä¸ä¸ªéæºæ å°å½æ°è®¡ç®ä½ç½®ï¼å¨å®é å¼åä¸ï¼å¦æä½ è¦æ·»å 大éçæ°æ®ï¼ä» ä» 16ä½æ¯è¿è¿ä¸å¤çï¼ä¸ºäºè®©è¯¯å¤çéä½ï¼æ们è¿å¯ä»¥ç¨æ´å¤çéæºæ å°å½æ°ãæ´é¿çäºè¿å¶åéå»è®¡ç®ä½ç½®ã
guavaå®ç°å¸éè¿æ»¤å¨
ç°å¨ç¸ä¿¡ä½ 对å¸éè¿æ»¤å¨åºè¯¥æä¸ä¸ªæ¯è¾ææ§ç认è¯äºï¼å¸éè¿æ»¤å¨æ ¸å¿ææ³å ¶å®å¹¶ä¸é¾ï¼é¾çå¨äºå¦ä½è®¾è®¡éæºæ å°å½æ°ï¼å°åºæ å°å 次ï¼äºè¿å¶åéçé¿åº¦è®¾ç½®ä¸ºå¤å°æ¯è¾å¥½ï¼è¿å¯è½å°±ä¸æ¯ä¸è¬çå¼åå¯ä»¥é©¾é©çäºï¼å¥½å¨Google大佬ç»æ们æä¾äºå¼ç®±å³ç¨çç»ä»¶ï¼æ¥å¸®å©æ们å®ç°å¸éè¿æ»¤å¨ï¼ç°å¨å°±è®©æ们ççæä¹Google大佬éç»æ们çâ礼ç©âå§ã
é¦å å¨pomå¼å ¥â礼ç©âï¼
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>19.0</version>
</dependency>
ç¶åå°±å¯ä»¥æµè¯å¦ï¼
private static int size = 1000000;//é¢è®¡è¦æå
¥å¤å°æ°æ®
private static double fpp = 0.01;//ææç误å¤ç
private static BloomFilter<Integer> bloomFilter = BloomFilter.create(Funnels.integerFunnel(), size, fpp);
public static void main(String[] args) {
//æå
¥æ°æ®
for (int i = 0; i < 1000000; i++) {
bloomFilter.put(i);
}
int count = 0;
for (int i = 1000000; i < 2000000; i++) {
if (bloomFilter.mightContain(i)) {
count++;
System.out.println(i + "误å¤äº");
}
}
System.out.println("æ»å
±ç误å¤æ°:" + count);
}
代ç ç®ååæï¼
æ们å®ä¹äºä¸ä¸ªå¸éè¿æ»¤å¨ï¼æ两个éè¦çåæ°ï¼åå«æ¯ æ们é¢è®¡è¦æå ¥å¤å°æ°æ®ï¼æ们æææç误å¤çï¼è¯¯å¤çä¸è½ä¸º0ã
æåå¸éè¿æ»¤å¨æå ¥äº0-1000000ï¼ç¶åç¨1000000-2000000æ¥æµè¯è¯¯å¤çã
è¿è¡ç»æï¼
1999501误å¤äº
1999567误å¤äº
1999640误å¤äº
1999697误å¤äº
1999827误å¤äº
1999942误å¤äº
æ»å
±ç误å¤æ°:10314
ç°å¨æ»å ±æ100ä¸æ°æ®æ¯ä¸åå¨çï¼è¯¯å¤äº10314次ï¼æ们计ç®ä¸è¯¯å¤ç
image.png
åæ们å®ä¹çææ误å¤ç0.01ç¸å·®æ å ã
rediså®ç°å¸éè¿æ»¤å¨
ä¸é¢ä½¿ç¨guavaå®ç°å¸éè¿æ»¤å¨æ¯ææ°æ®æ¾å¨æ¬å°å åä¸ï¼æ æ³å®ç°å¸éè¿æ»¤å¨çå ±äº«ï¼æ们è¿å¯ä»¥ææ°æ®æ¾å¨redisä¸ï¼ç¨ redisæ¥å®ç°å¸éè¿æ»¤å¨ï¼æ们è¦ä½¿ç¨çæ°æ®ç»ææ¯bitmapï¼ä½ å¯è½ä¼æçé®ï¼redisæ¯æäºç§æ°æ®ç»æï¼Stringï¼Listï¼Hashï¼Setï¼ZSetï¼æ²¡æbitmapåã没éï¼å®é ä¸bitmapçæ¬è´¨è¿æ¯Stringã
å¯è½æå°ä¼ä¼´ä¼è¯´ï¼çº³å°¼ï¼å¸éè¿æ»¤å¨è¿æ²¡ä»ç»å®ï¼æä¹ååºæ¥ä¸ä¸ªbitmapï¼æ²¡äºï¼ä½ å¯ä»¥æbitmapå°±ç解为ä¸ä¸ªäºè¿å¶åéã
è¦ç¨redisæ¥å®ç°å¸éè¿æ»¤å¨ï¼æ们éè¦èªå·±è®¾è®¡æ å°å½æ°ï¼èªå·±åº¦éäºè¿å¶åéçé¿åº¦ï¼è¿å¯¹ææ¥è¯´ï¼æ çæ¯ä¸ä¸ªä¸å¯è½å®æçä»»å¡ï¼åªè½åå©æç´¢å¼æï¼ä¸é¢ç´æ¥æ¾åºä»£ç æã
public class RedisMain {
static final int expectedInsertions = 100;//è¦æå
¥å¤å°æ°æ®
static final double fpp = 0.01;//ææç误å¤ç
//bitæ°ç»é¿åº¦
private static long numBits;
//hashå½æ°æ°é
private static int numHashFunctions;
static {
numBits = optimalNumOfBits(expectedInsertions, fpp);
numHashFunctions = optimalNumOfHashFunctions(expectedInsertions, numBits);
}
public static void main(String[] args) {
Jedis jedis = new Jedis("192.168.0.109", 6379);
for (int i = 0; i < 100; i++) {
long[] indexs = getIndexs(String.valueOf(i));
for (long index : indexs) {
jedis.setbit("codebear:bloom", index, true);
}
}
for (int i = 0; i < 100; i++) {
long[] indexs = getIndexs(String.valueOf(i));
for (long index : indexs) {
Boolean isContain = jedis.getbit("codebear:bloom", index);
if (!isContain) {
System.out.println(i + "è¯å®æ²¡æéå¤");
}
}
System.out.println(i + "å¯è½éå¤");
}
}
/**
* æ ¹æ®keyè·åbitmapä¸æ
*/
private static long[] getIndexs(String key) {
long hash1 = hash(key);
long hash2 = hash1 >>> 16;
long[] result = new long[numHashFunctions];
for (int i = 0; i < numHashFunctions; i++) {
long combinedHash = hash1 + i * hash2;
if (combinedHash < 0) {
combinedHash = ~combinedHash;
}
result[i] = combinedHash % numBits;
}
return result;
}
private static long hash(String key) {
Charset charset = Charset.forName("UTF-8");
return Hashing.murmur3_128().hashObject(key, Funnels.stringFunnel(charset)).asLong();
}
//计ç®hashå½æ°ä¸ªæ°
private static int optimalNumOfHashFunctions(long n, long m) {
return Math.max(1, (int) Math.round((double) m / n * Math.log(2)));
}
//计ç®bitæ°ç»é¿åº¦
private static long optimalNumOfBits(long n, double p) {
if (p == 0) {
p = Double.MIN_VALUE;
}
return (long) (-n * Math.log(p) / (Math.log(2) * Math.log(2)));
}
}
88å¯è½éå¤
89å¯è½éå¤
90å¯è½éå¤
91å¯è½éå¤
92å¯è½éå¤
93å¯è½éå¤
94å¯è½éå¤
95å¯è½éå¤
96å¯è½éå¤
97å¯è½éå¤
98å¯è½éå¤
99å¯è½éå¤