Shuwen.redis proxy by pengsven · Pull Request #102 · bsc-s2/lua-acid

pengsven · 2018-05-16T09:25:40Z

封装1下resty.readis，添加：

使用rpc_logging记录redis访问日志
重试执行redis命令机制
一个事务执行多条redis命令

添加客户端实现的一致性hash访问redis
添加代理实现的一致性hash访问redis

ps：测试后续补上

drmingdrmer · 2018-05-16T17:29:42Z

lib/acid/redis.lua

+local function get_redis_cli(self)
+    local redis_cli, err_msg = resty_redis:new()
+    if redis_cli == nil then
+        return nil, nil, 'NewRedisError', err_msg


4个返回...

drmingdrmer · 2018-05-16T17:31:56Z

lib/acid/redis.lua

+        local cmd, cmd_args = unpack(cmd_and_args)
+        local rst, err_code, err_msg = self[cmd](self, unpack(cmd_args or {}))
+        if err_code ~= nil then
+            self['discard'](self)


为啥不直接self:discard(), 为了某种风格一致吗?

还是直接用self:discard()吧

drmingdrmer · 2018-05-16T17:33:16Z

lib/acid/redis_chash.lua

@@ -0,0 +1,237 @@
+local strutil = require("acid.strutil")


感觉这个文件里大部分逻辑跟redis都没关系...

感觉这个文件实现了2个东西...一个是一致性hash,一个是nwr...感觉逻辑上是可以分开的, 并且独立于redis的.

1.acid.chash.lua已经是一个一致性hash的实现了，这里主要实现的是redis的一致性hash，
2.和nwr时有些耦合，我想想怎么分开

这个想完之后...有打算修改吗?还是就酱了?

nwr的区分开了，hash部分还是在redia_hash中

drmingdrmer · 2018-05-16T17:33:51Z

lib/acid/redis_chash.lua

+    return nil
+end
+
+local function optimize_choose_servers(addrs, least)


这个函数干嘛的...没懂...

一个提供按照某种规则(如s2按照service_stat的server响应时间排序优先访问哪些server)优化chash选择的server接口，这里是一个直接返回addr是的默认函数

drmingdrmer · 2018-05-16T17:34:41Z

lib/acid/redis_chash.lua

+
+        if (tonumber(multi_rst[1]) ~= 1 or tonumber(multi_rst[1]) ~= 0)
+            and tonumber(multi_rst[2]) ~= 1 then
+            return nil, 'RunRedisCMDError', 'transaction HSET error'


HSET这个名字是copy过来的吧...

drmingdrmer · 2018-05-16T17:39:37Z

lib/acid/redis_chash.lua

+end
+
+function _M.hset(self, args, n, w, expire)
+    return run_xset_cmd(self, 'hset', args, n, w, expire)


一般对于一个业务,nwr是确定的, 读/写的wr必须 w+r>n才有意义.所以这里直接传1个参数包括nwr3个值, 对业务使用可能更友好(一个业务1个nwr配置) 实现也简单点. 不需要区分读写操作配置不同的nw / nr. 直接都传nwr进来. 好像这样好点?

drmingdrmer · 2018-05-16T17:40:22Z

lib/acid/redis_proxy.lua

+    HGET = {'GET', 'hget', 2, false, {}},
+
+    -- hset(hashname, hashkey, val, expire=nil)
+    HSET = {'PUT', 'hset', 3, true, {'expire'}},


这种配置方式是跟保海串通好的吧?~ <(￣︶￣)>

我改了下，不需要get set hget hset，直接table的key lower()下就可以了

(⊙v⊙)嗯

drmingdrmer · 2018-05-16T17:40:48Z

lib/acid/redis_proxy.lua

+
+local redis_cmd_names = tableutil.keys(redis_cmd_model)
+
+local function get_secret_key(access_key, secret_key)


这个函数在干嘛...没懂啊...

aws_authenticator用到的，需要通过access_key得到secret_key

drmingdrmer · 2018-05-16T17:42:00Z

lib/acid/redis_proxy.lua

+end
+
+local function output(rst, err_code, err_msg)
+    local status, body, headers = 200, '', {}


output独立到一个公用函数里吧, 我记得acid里好多模块都有这么一坨定义

之前桐伟写过一个output()，没啥太大实用性 #78

为啥说没啥实用性?求详解

_M.output(status, headers, body)只是实现了将发送出去status, headers, body

drmingdrmer · 2018-05-16T17:43:09Z

lib/acid/redis_proxy.lua

+
+local function get_cmd_args()
+    --local uri_ptr = '^/redisproxy/v\\d+/(\\S+)/(\\S+)$'
+    local uri_ptr = '^/redisproxy/v\\d+/(\\S+?)/(\\S+)$'


印象里..ptr是pointer的缩写...这里要缩写的话, ptn好点...

上面的注释是打酱油的吗...

O(∩_∩)O哈哈~，一看你就是写C
ptn是pattern的缩写，比ptr些
想想用regex吧，ngx.re.match用的是regex
syntax: captures, err = ngx.re.match(subject, regex, options?, ctx?, res_table?)

么看懂....

liubaohai · 2018-05-17T07:01:19Z

lib/acid/redis.lua

+
+local to_str = strutil.to_str
+
+local _M = {}


统一风格，加下ver吧

liubaohai · 2018-05-17T07:09:59Z

lib/acid/redis.lua

+        end
+    end
+
+    local multi_rst, err_code, err_msg = self['exec'](self)


这种调用方式不如xp提的self:exec()

liubaohai · 2018-05-17T07:40:02Z

lib/acid/redis_proxy.lua

+    HGET = {'GET', 'hget', 2, false, {}},
+
+    -- hset(hashname, hashkey, val, expire=nil)
+    HSET = {'PUT', 'hset', 3, true, {'expire'}},


我改了下，不需要get set hget hset，直接table的key lower()下就可以了

liubaohai · 2018-05-17T09:08:09Z

lib/acid/redis_chash.lua

+
+    local redis_cli = acid_redis:new(ip, port, r_opts)
+
+    if pexpire ~= nil and cmd == 'hset' then


只有hset需要事务的方式吗，set呢

set不需要事务，set命令本身支持带expire参数

liubaohai · 2018-05-17T09:12:28Z

lib/acid/redis_proxy.lua

+
+    local rst, err_code, err_msg
+    if cmd == 'hget' then
+        rst, err_code, err_msg = self.redis_chash:hget(cmd_args, n, r)


nwr作为一个table传进去，expire也传进去，直接self.redis_chash[cmd]()

liubaohai · 2018-05-18T03:49:41Z

lib/acid/redis_proxy.lua

+
+    local cmd, cmd_args, nwr, expire = args.cmd, args.cmd_args, args.nwr, args.expire
+
+    local rst, err_code, err_msg = self.redis_cli[cmd](self, cmd_args, nwr, expire)


这里不太对吧，self有redis_cli吗，self.redis_chash?

liubaohai · 2018-05-18T06:13:16Z

lib/acid/redis_proxy.lua

+        return nil, 'InvalidCommand', to_str('just support: ', redis_cmd_names)
+    end
+
+    local http_method, cmd, nargs, needed_value, _ = unpack(cmd_model)


这里的cmd不需要配置，string.lower上面的cmd就可以了。少配置个参数。

drmingdrmer · 2018-05-23T06:06:35Z

lib/acid/redis.lua

+end
+
+setmetatable(_M, {__index = function(_, cmd)
+    local method = function (self, ...)


设置metatable有点不好懂, 直接列出所有的method逐个生成吧.

这里是为了自动支持redis所有命令，不过从可读性来说是有点不太好懂，或者加个注释如何

静态设定吧,没有注释的代码最好了...

drmingdrmer · 2018-05-23T06:06:53Z

lib/acid/redis_chash.lua

@@ -0,0 +1,237 @@
+local strutil = require("acid.strutil")


这个想完之后...有打算修改吗?还是就酱了?

drmingdrmer · 2018-05-23T06:09:46Z

lib/acid/redis_chash.lua

+
+local function get_chash(self)
+    local conf = self.conf
+    local now = ngx.time()


这个update的代码好像跟async_cache的逻辑(或cache)模块一样,应该可以复用吧?

async_cache好像可以

pengsven requested review from drmingdrmer and liubaohai May 16, 2018 09:25

drmingdrmer reviewed May 16, 2018

View reviewed changes

liubaohai reviewed May 17, 2018

View reviewed changes

liubaohai reviewed May 18, 2018

View reviewed changes

pengsven force-pushed the shuwen.redis_proxy branch from 30dd31b to d6ec951 Compare May 18, 2018 09:42

add redis proxy client

fa0f376

liubaohai approved these changes May 22, 2018

View reviewed changes

drmingdrmer reviewed May 23, 2018

View reviewed changes

renzhi and others added 10 commits May 23, 2018 17:26

add chash

191ae9a

add consistenthash test

6f95904

add redis.lua

7261426

add redis_chash.lua

1325fc4

add redis_proxy.lua

0a1d4e2

fix redis.lua

a61725f

fix redis_chash.lua

acf65e4

fix redis_proxy.lua

c76cddd

add nwr.lua

bfade98

simplify redis_cmd_model

9c82997

pengsven force-pushed the shuwen.redis_proxy branch from d6ec951 to 9c82997 Compare May 23, 2018 11:38

drmingdrmer approved these changes May 28, 2018

View reviewed changes

pengsven merged commit 267ef7a into renzhi_chash May 28, 2018

pengsven deleted the shuwen.redis_proxy branch October 24, 2018 06:08


		local redis_cmd_names = tableutil.keys(redis_cmd_model)

		local function get_secret_key(access_key, secret_key)


		local redis_cli = acid_redis:new(ip, port, r_opts)

		if pexpire ~= nil and cmd == 'hset' then


		local cmd, cmd_args, nwr, expire = args.cmd, args.cmd_args, args.nwr, args.expire

		local rst, err_code, err_msg = self.redis_cli[cmd](self, cmd_args, nwr, expire)

Comments

Conversation

pengsven commented May 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees