From 2e33c34d6c9910cae7af197ffad665676d3a50bc Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Fri, 16 Mar 2018 18:02:29 -0400 Subject: [PATCH 1/6] Create obip-0009.md --- obip-0009.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 obip-0009.md diff --git a/obip-0009.md b/obip-0009.md new file mode 100644 index 0000000..8c798f6 --- /dev/null +++ b/obip-0009.md @@ -0,0 +1,21 @@ +
+OBIP: 9
+Title: Data sharing via Pubsub
+Author: Chris Pacia 
+Discussions-To: Github Issues
+Status: Draft
+Type: Standards Track
+Created: 02/28/2018
+Copyright: MIT
+
+ +## Abstract +This obip defines a new data sharing protocol using IPFS pubsub to ensure persistent of content when users are offline. + +## Motivation +In basic IPFS content only persists when users go offline if other users have viewed/downloaded the content and those users remain online. Thus IPFS provides no hard garanutees that OpenBazaar stores will remain available when a user goes offline. Currently the protocol tries to remedy this problem by defining a list of "data peers" in the config file and +by pushing content to them on each publish to ensure persistence. The downside to this approach is the sharing of data is limited only to those peers found in the config file. While users remain free to change the default list of data peers, in practice very few do which elevates the default peers to the status of critcal infrastructure. + +By creating a pubsub channel for user content anyone on the network is free to subscribe to the channel and receive updates to all new content published on the network. They are then free to sort through that content and decide what they want to re-seed. + +## Specification From 79849075435351ba06d8252945d4b2930d153e02 Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Fri, 16 Mar 2018 21:00:27 -0400 Subject: [PATCH 2/6] Add subscribe and publish specs --- obip-0009.md | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/obip-0009.md b/obip-0009.md index 8c798f6..737f380 100644 --- a/obip-0009.md +++ b/obip-0009.md @@ -19,3 +19,83 @@ by pushing content to them on each publish to ensure persistence. The downside t By creating a pubsub channel for user content anyone on the network is free to subscribe to the channel and receive updates to all new content published on the network. They are then free to sort through that content and decide what they want to re-seed. ## Specification + +#### Protocol IDs +| Network | ID | +| ------------- |:-------------:| +| Mainnet | /openbazaar/floodsub/1.0.0 | +| Testnet | /openbazaar/floodsub/testnet/1.0.0 | + +#### Channel ID + +The pubsub channel ID is `zdj7Wdn68j9nHZZgksKfPQJG8XDZhqRpNKPJSgb7hk2wj5oaP` which is the CIDv1 (type DagProtobuf) of the sha256 multihash of []byte("floodsub:usercontent"). + +### Message Format +```protobuf +syntax = "proto2"; + +message RPC { + repeated SubOpts subscriptions = 1; + repeated Message publish = 2; + + message SubOpts { + optional bool subscribe = 1; // subscribe or unsubcribe + optional string topicid = 2; + } +} + +message Message { + optional bytes from = 1; + optional bytes data = 2; + optional bytes seqno = 3; + repeated string topicIDs = 4; +} + +// topicID = hash(topicDescriptor); (not the topic.name) +message TopicDescriptor { + optional string name = 1; + optional AuthOpts auth = 2; + optional EncOpts enc = 3; + + message AuthOpts { + optional AuthMode mode = 1; + repeated bytes keys = 2; // root keys to trust + + enum AuthMode { + NONE = 0; // no authentication, anyone can publish + KEY = 1; // only messages signed by keys in the topic descriptor are accepted + WOT = 2; // web of trust, certificates can allow publisher set to grow + } + } + + message EncOpts { + optional EncMode mode = 1; + repeated bytes keyHashes = 2; // the hashes of the shared keys used (salted) + + enum EncMode { + NONE = 0; // no encryption, anyone can read + SHAREDKEY = 1; // messages are encrypted with shared key + WOT = 2; // web of trust, certificates can allow publisher set to grow + } + } +} +``` + +#### Subscribe + +To `subscribe` to the content sharing channel a node MUST: + +1) Make a `GET_PROVIDERS` query in the DHT using the channel ID as the key. +2) Pick at least eight of the returned providers and open a connection to them. +3) Send a `hello` packet to each peer upon connection. +4) Set self as a provider for the channel ID in the DHT using the `ADD_PROVIDERS` command. + +The `hello` packet contains only the list of topic subscriptions. Currently the only topic for this channel is `rawblocks`. + +#### Publish + +1) Prior to updating its root directory the node should fetch a list of CIDs in its graph. +2) After updating the root directory the node should calculate the diff between old graph and new graph. +3) Make a `GET_PROVIDERS` query in the DHT using the channel ID as the key. +4) Pick at least eight of the returned providers and open a connection to them. +5) For each CID in the diff publish a `Message` to each peer using the corresponding raw block data as `Message.data` with a topic of `rawblocks`. From 4b6c82c197ee4df5a9fb1fffc4968d78f76848ac Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Fri, 16 Mar 2018 21:57:06 -0400 Subject: [PATCH 3/6] Finish spec --- obip-0009.md | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/obip-0009.md b/obip-0009.md index 737f380..228905d 100644 --- a/obip-0009.md +++ b/obip-0009.md @@ -10,7 +10,7 @@ Copyright: MIT ## Abstract -This obip defines a new data sharing protocol using IPFS pubsub to ensure persistent of content when users are offline. +This obip defines a new data sharing protocol using IPFS pubsub to ensure persistence of content when users are offline. ## Motivation In basic IPFS content only persists when users go offline if other users have viewed/downloaded the content and those users remain online. Thus IPFS provides no hard garanutees that OpenBazaar stores will remain available when a user goes offline. Currently the protocol tries to remedy this problem by defining a list of "data peers" in the config file and @@ -28,7 +28,7 @@ By creating a pubsub channel for user content anyone on the network is free to s #### Channel ID -The pubsub channel ID is `zdj7Wdn68j9nHZZgksKfPQJG8XDZhqRpNKPJSgb7hk2wj5oaP` which is the CIDv1 (type DagProtobuf) of the sha256 multihash of []byte("floodsub:usercontent"). +The pubsub channel ID is `zdj7Wdn68j9nHZZgksKfPQJG8XDZhqRpNKPJSgb7hk2wj5oaP` which is the CIDv1 (type `DagProtobuf`) of the sha256 multihash of []byte("floodsub:usercontent"). ### Message Format ```protobuf @@ -87,7 +87,7 @@ To `subscribe` to the content sharing channel a node MUST: 1) Make a `GET_PROVIDERS` query in the DHT using the channel ID as the key. 2) Pick at least eight of the returned providers and open a connection to them. -3) Send a `hello` packet to each peer upon connection. +3) Both peers should exchange`hello` packets upon connection. 4) Set self as a provider for the channel ID in the DHT using the `ADD_PROVIDERS` command. The `hello` packet contains only the list of topic subscriptions. Currently the only topic for this channel is `rawblocks`. @@ -99,3 +99,31 @@ The `hello` packet contains only the list of topic subscriptions. Currently the 3) Make a `GET_PROVIDERS` query in the DHT using the channel ID as the key. 4) Pick at least eight of the returned providers and open a connection to them. 5) For each CID in the diff publish a `Message` to each peer using the corresponding raw block data as `Message.data` with a topic of `rawblocks`. +6) Disconnect from each peer. + +#### Message Relay + +Upon receving new `RPC` message the node should relay it to all other peers who are subscribed to the `rawblocks` topic. + +#### Unsubscribing + +Sinces it's not possible to remove yourself as a subscriber from the DHT prior to provider expiration, nodes should respond to any incoming `hello` packets with a `hello` packet with the `rawblocks` subscription set to false. + +## CID or Raw Block? + +This spec calls for publishers to share the full raw block data in the channel. A possibly more efficient alternative would be to only share the CIDs for each block and let the subscribing peers decide whether to download the full block via IPFS. However, if it turns out that most subscribers would prefer to download each block then this strategy could end up being less effient as it requires another round trip and DHT crawl per CID. + +Additionally, publishers may be behind restricted NATs which may make establishing a connection with them difficult and prevent subscribers from acquring the data. By pushing the raw blocks directly we can get around NAT traversal issues. + +## Future Extensions + +We may wish to create a new topic which serializes the published data differently. For example we might want to specify the data type or originating peer such that subscribers can better determine if they wish to re-seed the block. + +## References + +https://github.com/ipld/cid + +https://github.com/libp2p/go-floodsub + +https://github.com/multiformats/go-multihash + From 9688c140b5f6125d15a280db6435995d89c5c325 Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Fri, 16 Mar 2018 22:29:43 -0400 Subject: [PATCH 4/6] Update auth and enc modes --- obip-0009.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/obip-0009.md b/obip-0009.md index 228905d..36a6045 100644 --- a/obip-0009.md +++ b/obip-0009.md @@ -101,6 +101,8 @@ The `hello` packet contains only the list of topic subscriptions. Currently the 5) For each CID in the diff publish a `Message` to each peer using the corresponding raw block data as `Message.data` with a topic of `rawblocks`. 6) Disconnect from each peer. +`auth.mode` and `enc.mode` should be set to `NONE`. + #### Message Relay Upon receving new `RPC` message the node should relay it to all other peers who are subscribed to the `rawblocks` topic. From af0f7a5bd13681713b388cb1ad6c6999ebce5201 Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Fri, 16 Mar 2018 23:08:50 -0400 Subject: [PATCH 5/6] Update pubsub channel ID --- obip-0009.md | 33 +-------------------------------- 1 file changed, 1 insertion(+), 32 deletions(-) diff --git a/obip-0009.md b/obip-0009.md index 36a6045..ba85002 100644 --- a/obip-0009.md +++ b/obip-0009.md @@ -28,7 +28,7 @@ By creating a pubsub channel for user content anyone on the network is free to s #### Channel ID -The pubsub channel ID is `zdj7Wdn68j9nHZZgksKfPQJG8XDZhqRpNKPJSgb7hk2wj5oaP` which is the CIDv1 (type `DagProtobuf`) of the sha256 multihash of []byte("floodsub:usercontent"). +The pubsub channel ID is `zb2rhezUFMJAfLymEQwukCy5f57hmMUpKDzLu58kaEwtZV5cB` which is the CIDv1 (type `Raw`) of the sha256 multihash of []byte("floodsub:usercontent"). ### Message Format ```protobuf @@ -50,35 +50,6 @@ message Message { optional bytes seqno = 3; repeated string topicIDs = 4; } - -// topicID = hash(topicDescriptor); (not the topic.name) -message TopicDescriptor { - optional string name = 1; - optional AuthOpts auth = 2; - optional EncOpts enc = 3; - - message AuthOpts { - optional AuthMode mode = 1; - repeated bytes keys = 2; // root keys to trust - - enum AuthMode { - NONE = 0; // no authentication, anyone can publish - KEY = 1; // only messages signed by keys in the topic descriptor are accepted - WOT = 2; // web of trust, certificates can allow publisher set to grow - } - } - - message EncOpts { - optional EncMode mode = 1; - repeated bytes keyHashes = 2; // the hashes of the shared keys used (salted) - - enum EncMode { - NONE = 0; // no encryption, anyone can read - SHAREDKEY = 1; // messages are encrypted with shared key - WOT = 2; // web of trust, certificates can allow publisher set to grow - } - } -} ``` #### Subscribe @@ -101,8 +72,6 @@ The `hello` packet contains only the list of topic subscriptions. Currently the 5) For each CID in the diff publish a `Message` to each peer using the corresponding raw block data as `Message.data` with a topic of `rawblocks`. 6) Disconnect from each peer. -`auth.mode` and `enc.mode` should be set to `NONE`. - #### Message Relay Upon receving new `RPC` message the node should relay it to all other peers who are subscribed to the `rawblocks` topic. From cf771453a8e36d7499cffa4445c77f36009bd1fc Mon Sep 17 00:00:00 2001 From: Chris Pacia Date: Sat, 17 Mar 2018 10:40:29 -0400 Subject: [PATCH 6/6] Fix typos --- obip-0009.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/obip-0009.md b/obip-0009.md index ba85002..d6e85bf 100644 --- a/obip-0009.md +++ b/obip-0009.md @@ -68,7 +68,7 @@ The `hello` packet contains only the list of topic subscriptions. Currently the 1) Prior to updating its root directory the node should fetch a list of CIDs in its graph. 2) After updating the root directory the node should calculate the diff between old graph and new graph. 3) Make a `GET_PROVIDERS` query in the DHT using the channel ID as the key. -4) Pick at least eight of the returned providers and open a connection to them. +4) Pick at least eight of the returned providers and open a connection to them. Check the `hello` packet for each to make sure they are subscribed to the `rawblocks` topic. 5) For each CID in the diff publish a `Message` to each peer using the corresponding raw block data as `Message.data` with a topic of `rawblocks`. 6) Disconnect from each peer. @@ -78,7 +78,7 @@ Upon receving new `RPC` message the node should relay it to all other peers who #### Unsubscribing -Sinces it's not possible to remove yourself as a subscriber from the DHT prior to provider expiration, nodes should respond to any incoming `hello` packets with a `hello` packet with the `rawblocks` subscription set to false. +Since it's not possible to remove yourself as a subscriber from the DHT prior to provider expiration, nodes should respond to any incoming `hello` packets with a `hello` packet with the `rawblocks` subscription set to false. ## CID or Raw Block?