-
Notifications
You must be signed in to change notification settings - Fork 5
Enhance Authentication, Homeserver, and PKARR Architecture docs #49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -4,32 +4,55 @@ aliases: | |||||
| - "../Pubky-Core/Homeserver" | ||||||
| --- | ||||||
|
|
||||||
| The Pubky network allows multiple, independent data stores, known as "Homeservers." This improves [[Censorship|censorship-resistance]] and prevents any single entity from controlling the flow of information, or locking people & data in as a walled garden. | ||||||
| The Pubky network allows multiple, independent data stores, known as "Homeservers." This improves [[Censorship|censorship-resistance]] and prevents any single entity from controlling the flow of information, or locking people & data in as a walled garden. | ||||||
|
|
||||||
| Homeservers are meant to represent a primary place to retrieve data from a specific [[0.Introduction|PKARR]] public key, but the user can redefine the location of their Homeserver at will by updating their [[0.Introduction|PKARR]] in the [[MainlineDHT|Mainline DHT]]. | ||||||
| Homeservers are meant to represent a primary place to retrieve data from a specific [[0.Introduction|PKARR]] public key, but the user can redefine the location of their Homeserver at will by updating their [[0.Introduction|PKARR]] record in the [[MainlineDHT|Mainline DHT]]. | ||||||
|
|
||||||
| ## Architecture | ||||||
|
|
||||||
| The Homeserver implementation consists of several components: the main HTTP API server (supporting both ICANN HTTP and Pubky TLS), an admin server, a Prometheus metrics server, and republishers that keep user and server keys alive on the DHT. | ||||||
|
|
||||||
| See the [repository](https://github.com/pubky/pubky-core/tree/main/pubky-homeserver) for API details and configuration. | ||||||
|
|
||||||
| ## Public vs Private Data | ||||||
|
|
||||||
| Current implementations across Pubky currently only support public, unencrypted data use cases. This does not prevent applications from using encrypted data within Homeservers, we simply haven't implemented that set of features in any direct way yet. Feel free to contribute! | ||||||
| Current implementations only support public, unencrypted data. Encrypted data and guarded (access-controlled) data are planned — see [[SecurityModel|Security Model]] for the trust implications. | ||||||
|
|
||||||
| ## Event Stream | ||||||
|
|
||||||
| Homeservers expose event streams for clients to sync data changes: | ||||||
|
|
||||||
| ## Data Retrieval and Synchronization | ||||||
| - `GET /events/` — Paginated event feed (cursor-based, 1000 events per batch) | ||||||
| - `GET /events-stream` — Server-Sent Events (SSE) real-time stream | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The main difference between these is that
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. obviously SSE is also another difference worth mentioning |
||||||
|
|
||||||
| - **API**: Any service can access data through a RESTful API | ||||||
| Used by clients like [[PubkyBackup|Pubky Backup]] to stay in sync. Note: the SSE broadcast channel is per-instance, so horizontal scaling requires architectural changes (e.g., Redis pub/sub). | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. dont mention the horizontal scaling issue here, it will be fixed soon and just confuses things |
||||||
|
|
||||||
| ## Decentralized Storage Architecture | ||||||
| ## Transport Security | ||||||
|
|
||||||
| - Homeservers can be operated by individuals, cooperatives, or commercial entities, with the option for anonymous operation, promoting a decentralized and resilient storage ecosystem where users could tune their preferred redundancy while maintaining the benefits of a central Homeserver. | ||||||
| - We do not yet have any mirroring or replication tools available yet. Feel free to build with us! | ||||||
| Pubky hosts use **PubkyTLS** for transport security (TLS with Raw Public Keys, RFC 7250). ICANN hosts use standard X.509 TLS. See [[SecurityModel|Security Model]] for details. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think actually that all homeservers expose both, confirm though please |
||||||
|
|
||||||
| ## User Data Control and Credible Exit | ||||||
|
|
||||||
| - The current network is being bootstrapped by Synonym's first Homeserver, which supports key accounts as a service, as well as all services required to operate the [[Explore/PubkyApp/Introduction|pubky App]]. | ||||||
| - While Synonym is currently not charging fees for homeserving, it is totally possible for anyone to run their own homerserver and require service fees or peer-to-peer micropayments, or any other requirement, to improve data redundancy and competition. | ||||||
| - Homeserver operators can use [[Homegate]] for signup verification, implementing SMS or Lightning Network verification to prevent spam while preserving user privacy. | ||||||
| - Synonym does not control who can join or serve data on the network, but in order to truly have a [[CredibleExit|"credible exit"]] the network will need to mature to have more providers of Homeservers and Pubky applications. | ||||||
| - We will also need to make it as easy as possible for people to host their own Homeservers, and to host homerserver clouds. | ||||||
| - We also need to ensure that applications remain performant & reliable as the network fragments. | ||||||
| - The current network is being bootstrapped by Synonym's first Homeserver — over time, more independent Homeserver operators and Pubky applications are needed for the network to fully decentralize | ||||||
| - Anyone can run their own Homeserver and set their own terms | ||||||
| - Homeserver operators can use [[Homegate]] for signup verification, implementing SMS or Lightning Network verification to prevent spam while preserving user privacy | ||||||
| - For true [[CredibleExit|credible exit]], users should maintain local backups via [[PubkyBackup|Pubky Backup]]. Homeserver mirroring is planned but not yet implemented | ||||||
| - Users can migrate to a new Homeserver at any time by moving their data and updating their [[0.Introduction|PKARR]] record | ||||||
|
|
||||||
| ## Event Stream | ||||||
| See [[SecurityModel|Security Model]] for the full trust analysis and failure recovery scenarios. | ||||||
|
|
||||||
| ## Running a Homeserver | ||||||
|
|
||||||
| > **Note:** Production deployment guides are not yet available. Easy deployment packages (Umbrel, apt, docker, start9) are planned. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| For local development and testing: | ||||||
|
|
||||||
| ```bash | ||||||
| cargo run -p pubky-homeserver | ||||||
| ``` | ||||||
|
|
||||||
| To spin up an ephemeral testnet: | ||||||
|
|
||||||
| Homeservers expose `/events/` and `/events-stream` endpoints for indexers/aggregators to sync data changes using cursor-based pagination. | ||||||
| ```bash | ||||||
| cargo run -p pubky-testnet | ||||||
| ``` | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,20 +3,56 @@ aliases: | |
| - "../../Pubky-Core/Pkarr/4.Architecture" | ||
| --- | ||
|
|
||
| In-depth look at the architecture of [[0.Introduction|PKARR]], including its components and how they interact. | ||
| In-depth look at the architecture of [[0.Introduction|PKARR]], including its components, data formats, and how they interact. | ||
|
|
||
| ## Components | ||
|
|
||
| - **Client**: Applications or users that publish or query [[0.Introduction|PKARR]]. | ||
| - **Relay**: Optional intermediary that helps clients behind NAT or firewall to communicate with the [[MainlineDHT|Mainline DHT]]. | ||
| - **Client**: Applications or users that publish or query [[0.Introduction|PKARR]] records. The main Rust crate provides `Keypair`/`PublicKey` types, `SignedPacket` builder, and `Client` for publishing/resolving. | ||
| - **Relay**: HTTP relay for environments without UDP access (browsers, firewalled networks). Also serves as intermediary for services on major cloud providers (AWS, GCP, Azure), whose IP ranges are often blocked by DHT nodes. | ||
| - **[[MainlineDHT|Mainline DHT]]**: The peer-to-peer network used to announce and resolve [[0.Introduction|PKARR]] records. | ||
| - **Republisher**: Services that keep [[0.Introduction|PKARR]] alive on the [[MainlineDHT|Mainline DHT]] by [periodically republishing](https://github.com/pubky/pkarr-churn/blob/main/results-node_decay.md) them. | ||
| - **Republisher**: Keeps [[0.Introduction|PKARR]] records alive on the [[MainlineDHT|Mainline DHT]] by [periodically republishing](https://github.com/pubky/pkarr-churn/blob/main/results-node_decay.md) them (~hourly). The `pkarr-republisher` component handles this for Homeserver operators. | ||
|
|
||
| ## SignedPacket Format | ||
|
|
||
| ``` | ||
| SignedPacket = public-key(32) + signature(64) + timestamp(8) + dns-packet(≤1000) | ||
| ``` | ||
|
|
||
| Maximum total size: 1104 bytes. Public keys are encoded as 52-character z-base32 strings for DNS compatibility. All packets are Ed25519-signed, ensuring authenticity and integrity. Published to the DHT using BEP44 mutable item operations. The `dns-packet` is a standard compressed DNS packet (see [Supported DNS Record Types](#supported-dns-record-types) below). | ||
|
|
||
| See the [PKARR repository](https://github.com/pubky/pkarr) for the full format specification and API reference. | ||
|
|
||
| ## Interaction Flow | ||
|
|
||
| 1. **Publishing**: Clients publish [[0.Introduction|PKARR]] to the [[MainlineDHT|Mainline DHT]], either directly or through a relay (required for browsers since the DHT uses UDP). | ||
| 2. **Republishing**: Homeservers and relays republish records for their users to [keep them available](https://github.com/pubky/pkarr-churn/blob/main/results-node_decay.md) on the [[MainlineDHT|Mainline DHT]]. | ||
| 3. **Querying**: Clients query the [[MainlineDHT|Mainline DHT]] for [[0.Introduction|PKARR]], either directly or through a relay. | ||
| 2. **Republishing**: Homeservers and relays republish records for their users to [keep them available](https://github.com/pubky/pkarr-churn/blob/main/results-node_decay.md) on the [[MainlineDHT|Mainline DHT]]. Records degrade over hours to days without republishing, so hourly republishing is recommended. | ||
| 3. **Querying**: Clients query the [[MainlineDHT|Mainline DHT]] for [[0.Introduction|PKARR]] using the SHA1 hash of the public key, either directly or through a relay. | ||
|
|
||
| ## Supported DNS Record Types | ||
|
|
||
| A, AAAA, CNAME, TXT, HTTPS/SVCB (RFC 9460). | ||
|
|
||
| The `_pubky` SVCB record points to a user's [[Homeserver|Homeserver]], enabling discovery, migration, and failover: | ||
|
|
||
| ``` | ||
| _pubky.<public-key> SVCB 1 homeserver.example.com port=443 | ||
| ``` | ||
|
|
||
| ## Caching | ||
|
|
||
| - **InMemoryCache**: LRU cache for lightweight deployments | ||
| - **LmdbCache**: Persistent LMDB-backed cache for relays and servers | ||
|
|
||
| ## Failure Modes | ||
|
|
||
| **DHT unreachable**: Clients fall back to cached records from relays (cached for weeks to months). | ||
|
|
||
| **Records expired**: Identity discovery fails until republished, but the identity itself remains valid — only the Homeserver pointer becomes unavailable. | ||
|
|
||
| **Under attack**: Cached versions on relays remain accessible. Attacks can only prevent updates, not erase existing records. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This needs more detail, what is the attack scenario? |
||
|
|
||
| ## Key Technologies | ||
|
|
||
| - **[[MainlineDHT|Mainline DHT]]**: A global, censorship-resistant p2p network of 10+ million peers. [[0.Introduction|PKARR]] are announced here using BEP44. | ||
| - **[[MainlineDHT|Mainline DHT]]**: A global, censorship-resistant p2p network of 10+ million peers. [[0.Introduction|PKARR]] records are announced here using BEP44. | ||
|
|
||
| See [[SecurityModel|Security Model]] for the DHT threat model and the [pkarr repository](https://github.com/pubky/pkarr) for code examples and multi-platform support details. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a forever limitation rather than a current one i think?