Going Distributed #19
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| main | ||
| .vscode/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,57 @@ | ||
| github.com/DataDog/datadog-go v2.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ= | ||
| github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878 h1:EFSB7Zo9Eg91v7MJPVsifUysc/wPdN+NOnVe6bWbdBM= | ||
| github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878/go.mod h1:3AMJUQhVx52RsWOnlkpikZr01T/yAVN2gn0861vByNg= | ||
| github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= | ||
| github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4= | ||
| github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps= | ||
| github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag= | ||
| github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I= | ||
| github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= | ||
| github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= | ||
| github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= | ||
| github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= | ||
| github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc= | ||
| github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= | ||
| github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= | ||
| github.com/hashicorp/go-hclog v0.9.1 h1:9PZfAcVEvez4yhLH2TBU64/h/z4xlFI80cWXRrxuKuM= | ||
| github.com/hashicorp/go-hclog v0.9.1/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= | ||
| github.com/hashicorp/go-immutable-radix v1.0.0 h1:AKDB1HM5PWEA7i4nhcpwOrO2byshxBjXVn/J/3+z5/0= | ||
| github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= | ||
| github.com/hashicorp/go-msgpack v0.5.5 h1:i9R9JSrqIz0QVLz3sz+i3YJdT7TTSLcfLLzJi9aZTuI= | ||
| github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= | ||
| github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs= | ||
| github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= | ||
| github.com/hashicorp/golang-lru v0.5.0 h1:CL2msUPvZTLb5O648aiLNJw3hnBxN2+1Jq8rCOH9wdo= | ||
| github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= | ||
| github.com/hashicorp/raft v1.1.0/go.mod h1:4Ak7FSPnuvmb0GV6vgIAJ4vYT4bek9bb6Q+7HVbyzqM= | ||
| github.com/hashicorp/raft v1.1.2 h1:oxEL5DDeurYxLd3UbcY/hccgSPhLLpiBZ1YxtWEq59c= | ||
| github.com/hashicorp/raft v1.1.2/go.mod h1:vPAJM8Asw6u8LxC3eJCUZmRP/E4QmUGE1R7g7k8sG/8= | ||
| github.com/hashicorp/raft-boltdb v0.0.0-20171010151810-6e5ba93211ea/go.mod h1:pNv7Wc3ycL6F5oOWn+tPGo2gWD4a5X+yp/ntwdKLjRk= | ||
| github.com/hashicorp/raft-boltdb v0.0.0-20191021154308-4207f1bf0617 h1:CJDRE/2tBNFOrcoexD2nvTRbQEox3FDxl4NxIezp1b8= | ||
| github.com/hashicorp/raft-boltdb v0.0.0-20191021154308-4207f1bf0617/go.mod h1:aUF6HQr8+t3FC/ZHAC+pZreUBhTaxumuu3L+d37uRxk= | ||
| github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= | ||
| github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= | ||
| github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= | ||
| github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= | ||
| github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM= | ||
| github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= | ||
| github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= | ||
| github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= | ||
| github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= | ||
| github.com/rs/zerolog v1.19.0 h1:hYz4ZVdUgjXTBUmrkrw55j1nHx68LfOKIQk5IYtyScg= | ||
| github.com/rs/zerolog v1.19.0/go.mod h1:IzD0RJ65iWH0w97OQQebJEvTZYvsCUm9WVLWBQrJRjo= | ||
| github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= | ||
| github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= | ||
| github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= | ||
| github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM= | ||
| golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= | ||
| golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= | ||
| golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= | ||
| golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= | ||
| golang.org/x/sys v0.0.0-20190523142557-0e01d883c5c5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20190602015325-4c4f7f33c9ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= | ||
| golang.org/x/tools v0.0.0-20190828213141-aed303cbaa74/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= | ||
| golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,109 @@ | ||||||
| package consensus | ||||||
|
|
||||||
| import ( | ||||||
| "encoding/json" | ||||||
| "fmt" | ||||||
| "io" | ||||||
|
|
||||||
| "github.com/SystemBuilders/LocKey/internal/lockservice" | ||||||
| "github.com/hashicorp/raft" | ||||||
| ) | ||||||
|
|
||||||
| type fsm RaftStore | ||||||
|
|
||||||
| type fsmSnapshot struct { | ||||||
| lockMap map[string]string | ||||||
| } | ||||||
|
|
||||||
| func (f *fsm) Apply(l *raft.Log) interface{} { | ||||||
| var c command | ||||||
| if err := json.Unmarshal(l.Data, &c); err != nil { | ||||||
| panic(fmt.Sprintf("failed to unmarshal command: %s", err.Error())) | ||||||
| } | ||||||
|
|
||||||
| switch c.Op { | ||||||
| case "acquire": | ||||||
| return f.applyAcquire(c.Key, c.Value) | ||||||
|
|
||||||
| case "release": | ||||||
| return f.applyRelease(c.Key, c.Value) | ||||||
|
|
||||||
| default: | ||||||
| panic(fmt.Sprintf("unrecognized command op: %s", c.Op)) | ||||||
|
|
||||||
|
|
||||||
| } | ||||||
| } | ||||||
|
|
||||||
| func (f *fsm) applyAcquire(lock, owner string) interface{} { | ||||||
| desc := lockservice.NewSimpleDescriptor(lock, owner) | ||||||
|
|
||||||
| err := f.ls.Acquire(desc) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| func (f *fsm) applyRelease(lock, owner string) interface{} { | ||||||
| desc := lockservice.NewSimpleDescriptor(lock, owner) | ||||||
|
|
||||||
| err := f.ls.Release(desc) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| // Snapshot returns a snapshot of the key-value store. We wrap | ||||||
| // the things we need in fsmSnapshot and then send that over to Persist. | ||||||
| // Persist encodes the needed data from fsmsnapshot and transport it to | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| // Restore where the necessary data is replicated into the finite state machine. | ||||||
| // This allows the consensus algorithm to truncate the replicated log. | ||||||
| func (f *fsm) Snapshot() (raft.FSMSnapshot, error) { | ||||||
| return &fsmSnapshot{lockMap: f.ls.GetLockMap()}, nil | ||||||
| } | ||||||
|
|
||||||
| // Restores the lockMap to a previous state | ||||||
| func (f *fsm) Restore(lockMap io.ReadCloser) error { | ||||||
| lockMapSnapshot := make(map[string]string) | ||||||
| if err := json.NewDecoder(lockMap).Decode(&lockMapSnapshot); err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| // Set the state from snapshot. No need to use mutex lock according | ||||||
| // to Hasicorp doc | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| f.ls.SetLockMap(lockMapSnapshot) | ||||||
|
|
||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| func (f *fsmSnapshot) Persist(sink raft.SnapshotSink) error { | ||||||
| err := func() error { | ||||||
| // Encode data | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| b, err := json.Marshal(f.lockMap) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| // Write data to sink | ||||||
|
Contributor
There was a problem hiding this comment. Remove the comment or write what exactly is happening or why it is happening instead. |
||||||
| if _, err := sink.Write(b); err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| // Close the sink | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| if err := sink.Close(); err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| return nil | ||||||
| }() | ||||||
|
|
||||||
| if err != nil { | ||||||
| sink.Cancel() | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| func (f *fsmSnapshot) Release() {} | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,43 @@ | ||||||
| package consensus | ||||||
|
|
||||||
| import ( | ||||||
| "log" | ||||||
| "net" | ||||||
| "net/http" | ||||||
| ) | ||||||
|
|
||||||
| // Start starts the http service using the listener within a RaftStore. | ||||||
| // | ||||||
| // The HTTP server is used to redirect commands like Set, Delete and Join | ||||||
| // to the leader RaftStore in a cluster. The HTTP address is always | ||||||
| // one away from the Raft address which the raft node uses for communication | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
Contributor
There was a problem hiding this comment. Because one away seems a bit vague. |
||||||
| // with other raft nodes. | ||||||
| // | ||||||
| // This policy is maybe a bit to trivial and in the future, a more dynamic | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| // mapping between a Raft node and its listener can be integrated | ||||||
| func (rs *RaftStore) Start() error { | ||||||
| server := http.Server{ | ||||||
| Handler: rs, | ||||||
| } | ||||||
|
|
||||||
| ln, err := net.Listen("tcp", rs.httpAddr) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
| rs.ln = ln | ||||||
|
|
||||||
| go func() { | ||||||
| err := server.Serve(rs.ln) | ||||||
| if err != nil { | ||||||
| log.Fatalf("HTTP serve: %s", err) | ||||||
| } | ||||||
| }() | ||||||
|
|
||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| // Close stops the listener corresponding to a Raft node. | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
| func (rs *RaftStore) Close() { | ||||||
| rs.ln.Close() | ||||||
| return | ||||||
| } | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you not panic and return the error?
Panics stop the system function, we shouldn't allow that.