From 7577962418e03904ac2a476449b515592c8b47b1 Mon Sep 17 00:00:00 2001 From: rch Date: Wed, 21 Jan 2026 13:50:39 -0800 Subject: [PATCH 1/3] Tentatively add connector dev docs --- baton/deploy.mdx | 151 +++++++++- baton/intro.mdx | 8 +- developer/baton-sdk.mdx | 513 +++++++++++++++++++++++++++++++++ developer/c1-api.mdx | 360 +++++++++++++++++++++++ developer/cel-expressions.mdx | 135 +++++++++ developer/community.mdx | 205 +++++++++++++ developer/concepts.mdx | 292 +++++++++++++++++++ developer/config-schema.mdx | 359 +++++++++++++++++++++++ developer/debugging.mdx | 363 +++++++++++++++++++++++ developer/error-codes.mdx | 294 +++++++++++++++++++ developer/glossary.mdx | 145 ++++++++++ developer/http-authoring.mdx | 247 ++++++++++++++++ developer/intro.mdx | 269 +++++++++++++++-- developer/pagination.mdx | 379 ++++++++++++++++++++++++ developer/provisioning.mdx | 414 ++++++++++++++++++++++++++ developer/recipes-auth.mdx | 144 +++++++++ developer/recipes-caching.mdx | 308 ++++++++++++++++++++ developer/recipes-id.mdx | 106 +++++++ developer/recipes-modeling.mdx | 292 +++++++++++++++++++ developer/recipes-testing.mdx | 242 ++++++++++++++++ developer/sql-authoring.mdx | 306 ++++++++++++++++++++ developer/submit.mdx | 197 +++++++++++++ developer/syncing.mdx | 455 +++++++++++++++++++++++++++++ docs.json | 68 ++++- 24 files changed, 6223 insertions(+), 29 deletions(-) create mode 100644 developer/baton-sdk.mdx create mode 100644 developer/c1-api.mdx create mode 100644 developer/cel-expressions.mdx create mode 100644 developer/community.mdx create mode 100644 developer/concepts.mdx create mode 100644 developer/config-schema.mdx create mode 100644 developer/debugging.mdx create mode 100644 developer/error-codes.mdx create mode 100644 developer/glossary.mdx create mode 100644 developer/http-authoring.mdx create mode 100644 developer/pagination.mdx create mode 100644 developer/provisioning.mdx create mode 100644 developer/recipes-auth.mdx create mode 100644 developer/recipes-caching.mdx create mode 100644 developer/recipes-id.mdx create mode 100644 developer/recipes-modeling.mdx create mode 100644 developer/recipes-testing.mdx create mode 100644 developer/sql-authoring.mdx create mode 100644 developer/submit.mdx create mode 100644 developer/syncing.mdx diff --git a/baton/deploy.mdx b/baton/deploy.mdx index 5c1685f..397570d 100644 --- a/baton/deploy.mdx +++ b/baton/deploy.mdx @@ -234,5 +234,154 @@ Complete Steps 1-3 in [Set up an external data source](/product/admin/external-d You can run the script on demand, or set up a scheduler to run it periodically. The S3 bucket syncs with ConductorOne once an hour. - +## Deployment examples + +### Docker + +Build a container for your self-hosted connector: + +```dockerfile +FROM golang:1.23-alpine AS builder +WORKDIR /app +COPY . . +RUN make build + +FROM alpine:latest +COPY --from=builder /app/dist/baton-yourservice /usr/local/bin/ +ENTRYPOINT ["baton-yourservice"] +``` + +Run in service mode: + +```bash +docker run -d \ + -e BATON_CLIENT_ID=$CLIENT_ID \ + -e BATON_CLIENT_SECRET=$CLIENT_SECRET \ + -e BATON_API_KEY=$API_KEY \ + baton-yourservice +``` + +### Kubernetes + +Create secrets for connector credentials: + +```yaml +# baton-secrets.yaml +apiVersion: v1 +kind: Secret +metadata: + name: baton-yourservice-secrets +type: Opaque +stringData: + BATON_CLIENT_ID: "" + BATON_CLIENT_SECRET: "" + BATON_API_KEY: "" +``` + +Deploy the connector: + +```yaml +# baton-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: baton-yourservice + labels: + app: baton-yourservice +spec: + replicas: 1 + selector: + matchLabels: + app: baton-yourservice + template: + metadata: + labels: + app: baton-yourservice + spec: + containers: + - name: baton-yourservice + image: ghcr.io/conductorone/baton-yourservice:latest + imagePullPolicy: IfNotPresent + envFrom: + - secretRef: + name: baton-yourservice-secrets +``` + +For high availability, increase replicas. Multiple instances can share the same client ID and secret; ConductorOne distributes tasks round-robin. + +### Systemd (Linux) + +Create a service unit: + +```ini +# /etc/systemd/system/baton-yourservice.service +[Unit] +Description=Baton Connector for YourService +After=network.target + +[Service] +Type=simple +User=baton +ExecStart=/usr/local/bin/baton-yourservice +Restart=always +RestartSec=10 +EnvironmentFile=/etc/baton/yourservice.env + +[Install] +WantedBy=multi-user.target +``` + +Environment file: + +```bash +# /etc/baton/yourservice.env +BATON_CLIENT_ID=your-client-id +BATON_CLIENT_SECRET=your-client-secret +BATON_API_KEY=your-api-key +``` + +Enable and start: + +```bash +sudo systemctl enable baton-yourservice +sudo systemctl start baton-yourservice +sudo systemctl status baton-yourservice +``` + +## Production considerations + +### Credential management + +Never commit credentials to source control. Options: + +| Method | Use case | +|--------|----------| +| Environment variables | Works everywhere | +| Secrets manager | AWS Secrets Manager, HashiCorp Vault | +| Kubernetes secrets | For K8s deployments | + +### Network requirements + +Service mode requires outbound HTTPS only: +- Connector initiates connections to ConductorOne +- No inbound ports required +- Works behind NAT and most firewalls + +If you have egress filtering, allow HTTPS to `*.conductorone.com`. + +### Resource requirements + +Connectors are lightweight: + +| Resource | Recommendation | +|----------|----------------| +| CPU | 1 CPU (or fractional) is sufficient | +| Memory | Typically under 100MB; allocate 250-500MB for large syncs | + +### Monitoring + +ConductorOne sends email alerts after three consecutive sync failures. Monitor: +- Sync status in the ConductorOne UI +- Connector logs for errors +- Memory usage (should stay stable) diff --git a/baton/intro.mdx b/baton/intro.mdx index c8e6793..4048aa1 100644 --- a/baton/intro.mdx +++ b/baton/intro.mdx @@ -7,15 +7,19 @@ og:description: "Here you'll find our library of pre-built connectors, plus the --- -**If we don’t have the connector you’re looking for today, never fear.** +**If we don't have the connector you're looking for today, never fear.** ConductorOne supports custom connectors via our open-source Baton SDK, so you can integrate any (and we do mean ANY) app, even niche or legacy systems. -You can build it yourself, or we can build it for you. Whether it’s API-based, file-based, or something else, you won’t ever be limited to the connectors in our library. We give you multiple ways to connect and govern any app. +You can build it yourself, or we can build it for you. Whether it's API-based, file-based, or something else, you won't ever be limited to the connectors in our library. We give you multiple ways to connect and govern any app. **Let us know what you need. We can help!** + +**Building a connector?** The [Developer tab](/developer/intro) has comprehensive guides for connector development: SDK patterns, pagination, provisioning, testing, and troubleshooting. For configuration-driven integration without code, see [baton-http](/developer/http-authoring) and [baton-sql](/developer/sql-authoring). + + ## Connectors for every need and use case diff --git a/developer/baton-sdk.mdx b/developer/baton-sdk.mdx new file mode 100644 index 0000000..2075715 --- /dev/null +++ b/developer/baton-sdk.mdx @@ -0,0 +1,513 @@ +--- +title: "Baton SDK API reference" +sidebarTitle: "Baton SDK" +description: "Baton SDK interfaces, methods, and helper functions for connector development." +--- + +This reference documents the public APIs in `baton-sdk` that connector developers use. The SDK follows an inversion-of-control pattern: you implement interfaces, the SDK calls your methods. + +## Connector builder interfaces + +### Connectorbuilder (entry point) + +The main interface your connector must implement: + +```go +type ConnectorBuilder interface { + MetadataProvider + ValidateProvider + ResourceSyncers(ctx context.Context) []ResourceSyncer +} + +// V2 preferred - uses enhanced sync options +type ConnectorBuilderV2 interface { + MetadataProvider + ValidateProvider + ResourceSyncers(ctx context.Context) []ResourceSyncerV2 +} +``` + +**Required sub-interfaces:** + +```go +type MetadataProvider interface { + Metadata(ctx context.Context) (*v2.ConnectorMetadata, error) +} + +type ValidateProvider interface { + Validate(ctx context.Context) (annotations.Annotations, error) +} +``` + +**Usage:** + +```go +func NewConnector(ctx context.Context, in interface{}, opts ...Opt) (types.ConnectorServer, error) + +// Options +func WithTicketingEnabled() Opt +func WithMetricsHandler(h metrics.Handler) Opt +func WithSessionStore(ss sessions.SessionStore) Opt +``` + +### ResourceSyncer (core sync) + +The primary interface for syncing data: + +```go +// V1 Interface (legacy) +type ResourceSyncer interface { + ResourceType(ctx context.Context) *v2.ResourceType + List(ctx context.Context, parentResourceID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) + Entitlements(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Entitlement, string, annotations.Annotations, error) + Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) +} + +// V2 Interface (preferred) +type ResourceSyncerV2 interface { + ResourceType(ctx context.Context) *v2.ResourceType + List(ctx context.Context, parentResourceID *v2.ResourceId, + opts resource.SyncOpAttrs) ([]*v2.Resource, *resource.SyncOpResults, error) + Entitlements(ctx context.Context, resource *v2.Resource, + opts resource.SyncOpAttrs) ([]*v2.Entitlement, *resource.SyncOpResults, error) + Grants(ctx context.Context, resource *v2.Resource, + opts resource.SyncOpAttrs) ([]*v2.Grant, *resource.SyncOpResults, error) +} +``` + +**V2 Sync Types:** + +```go +type SyncOpAttrs struct { + SyncID string + PageToken string + Session sessions.SessionStore +} + +type SyncOpResults struct { + NextPageToken string + Annotations annotations.Annotations +} +``` + +**Key differences V1 vs V2:** +- V2 receives `SyncOpAttrs` with session store for caching +- V2 returns structured `SyncOpResults` instead of raw values +- V2 is preferred for new connectors + +### ResourceTargetedSyncer (single resource fetch) + +Extension for fetching a single resource by ID: + +```go +type ResourceTargetedSyncer interface { + ResourceSyncer + Get(ctx context.Context, resourceId *v2.ResourceId, + parentResourceId *v2.ResourceId) (*v2.Resource, annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_TARGETED_SYNC` for faster incremental updates. + +### ResourceProvisioner (grant/revoke) + +For connectors that can modify access: + +```go +// V1 (legacy) +type ResourceProvisioner interface { + ResourceSyncer + Grant(ctx context.Context, resource *v2.Resource, + entitlement *v2.Entitlement) (annotations.Annotations, error) + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} + +// V2 (preferred) - Grant returns created grants +type ResourceProvisionerV2 interface { + ResourceSyncer + Grant(ctx context.Context, resource *v2.Resource, + entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} +``` + +**V2 advantage:** `Grant()` returns list of created grants, useful when one grant creates multiple assignments. + +Enables `CAPABILITY_PROVISION`. + +### AccountManager (user provisioning) + +For creating user accounts (JIT provisioning): + +```go +type AccountManager interface { + ResourceSyncer + CreateAccount(ctx context.Context, + accountInfo *v2.AccountInfo, + credentialOptions *v2.LocalCredentialOptions, + ) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) + + CreateAccountCapabilityDetails(ctx context.Context, + ) (*v2.CredentialDetailsAccountProvisioning, annotations.Annotations, error) +} + +type CreateAccountResponse interface { + proto.Message + GetIsCreateAccountResult() bool +} +``` + +**Response types:** +- `*v2.CreateAccountResponse_SuccessResult` - Account created successfully +- `*v2.CreateAccountResponse_ActionRequiredResult` - User action needed + +Enables `CAPABILITY_ACCOUNT_PROVISIONING`. + +### ResourceManager (create/delete resources) + +For creating and deleting resources: + +```go +// V1 (legacy) +type ResourceManager interface { + ResourceSyncer + Create(ctx context.Context, resource *v2.Resource) (*v2.Resource, annotations.Annotations, error) + Delete(ctx context.Context, resourceId *v2.ResourceId) (annotations.Annotations, error) +} + +// V2 (preferred) - Delete receives parent ID +type ResourceManagerV2 interface { + ResourceSyncer + Create(ctx context.Context, resource *v2.Resource) (*v2.Resource, annotations.Annotations, error) + Delete(ctx context.Context, resourceId *v2.ResourceId, + parentResourceID *v2.ResourceId) (annotations.Annotations, error) +} + +// Separate interfaces if you only need one operation +type ResourceDeleter interface { + ResourceSyncer + Delete(ctx context.Context, resourceId *v2.ResourceId) (annotations.Annotations, error) +} + +type ResourceDeleterV2 interface { + ResourceSyncer + Delete(ctx context.Context, resourceId *v2.ResourceId, + parentResourceID *v2.ResourceId) (annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_RESOURCE_CREATE` and `CAPABILITY_RESOURCE_DELETE`. + +### CredentialManager (credential rotation) + +For rotating credentials: + +```go +type CredentialManager interface { + ResourceSyncer + Rotate(ctx context.Context, + resourceId *v2.ResourceId, + credentialOptions *v2.LocalCredentialOptions, + ) ([]*v2.PlaintextData, annotations.Annotations, error) + + RotateCapabilityDetails(ctx context.Context, + ) (*v2.CredentialDetailsCredentialRotation, annotations.Annotations, error) +} +``` + +Returns plaintext credentials; the SDK encrypts them before sending to ConductorOne. + +Enables `CAPABILITY_CREDENTIAL_ROTATION`. + +### CustomActionManager (custom actions) + +For connectors with custom operations: + +```go +type CustomActionManager interface { + ListActionSchemas(ctx context.Context) ([]*v2.BatonActionSchema, annotations.Annotations, error) + GetActionSchema(ctx context.Context, name string) (*v2.BatonActionSchema, annotations.Annotations, error) + InvokeAction(ctx context.Context, name string, args *structpb.Struct, + ) (string, v2.BatonActionStatus, *structpb.Struct, annotations.Annotations, error) + GetActionStatus(ctx context.Context, id string, + ) (v2.BatonActionStatus, string, *structpb.Struct, annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_ACTIONS`. + +### EventFeed (real-time events) + +For streaming events from the target system: + +```go +// V2 (preferred) - Multiple feeds +type EventProviderV2 interface { + ConnectorBuilder + EventFeeds(ctx context.Context) []EventFeed +} + +type EventFeed interface { + ListEvents(ctx context.Context, earliestEvent *timestamppb.Timestamp, + pToken *pagination.StreamToken, + ) ([]*v2.Event, *pagination.StreamState, annotations.Annotations, error) + EventFeedMetadata(ctx context.Context) *v2.EventFeedMetadata +} +``` + +Enables `CAPABILITY_EVENT_FEED_V2`. + +## Type builders + +### Resource builder + +```go +// Create resource type definition +func NewResourceType(name string, requiredTraits []v2.ResourceType_Trait, + msgs ...proto.Message) *v2.ResourceType + +// Create resource ID +func NewResourceID(resourceType *v2.ResourceType, objectID interface{}) (*v2.ResourceId, error) + +// Create resource instance +func NewResource(name string, resourceType *v2.ResourceType, + objectID interface{}, resourceOptions ...ResourceOption) (*v2.Resource, error) +``` + +**Resource Options:** + +```go +func WithAnnotation(msgs ...proto.Message) ResourceOption +func WithExternalID(externalID *v2.ExternalId) ResourceOption +func WithParentResourceID(parentResourceID *v2.ResourceId) ResourceOption +func WithDescription(description string) ResourceOption +func WithUserTrait(opts ...UserTraitOption) ResourceOption +func WithGroupTrait(opts ...GroupTraitOption) ResourceOption +func WithRoleTrait(opts ...RoleTraitOption) ResourceOption +func WithAppTrait(opts ...AppTraitOption) ResourceOption +func WithSecretTrait(opts ...SecretTraitOption) ResourceOption +``` + +### User trait options + +```go +func WithStatus(status v2.UserTrait_Status_Status) UserTraitOption +func WithDetailedStatus(status v2.UserTrait_Status_Status, details string) UserTraitOption +func WithEmail(email string, primary bool) UserTraitOption +func WithUserLogin(login string, aliases ...string) UserTraitOption +func WithEmployeeID(employeeIDs ...string) UserTraitOption +func WithUserIcon(assetRef *v2.AssetRef) UserTraitOption +func WithUserProfile(profile map[string]interface{}) UserTraitOption +func WithAccountType(accountType v2.UserTrait_AccountType) UserTraitOption +func WithCreatedAt(createdAt time.Time) UserTraitOption +``` + +**Status values:** + +| Status | Meaning | +|--------|---------| +| `STATUS_ENABLED` | Active user | +| `STATUS_DISABLED` | Suspended user | +| `STATUS_DELETED` | Soft-deleted user | +| `STATUS_UNSPECIFIED` | Unknown status | + +**Account types:** + +| Type | Meaning | +|------|---------| +| `ACCOUNT_TYPE_HUMAN` | Human user | +| `ACCOUNT_TYPE_SERVICE` | Service account | +| `ACCOUNT_TYPE_SYSTEM` | System account | + +### Entitlement builder + +```go +// Permission entitlement (e.g., "admin", "editor") +func NewPermissionEntitlement(resource *v2.Resource, name string, + entitlementOptions ...EntitlementOption) *v2.Entitlement + +// Assignment entitlement (e.g., group membership) +func NewAssignmentEntitlement(resource *v2.Resource, name string, + entitlementOptions ...EntitlementOption) *v2.Entitlement + +// Generic entitlement +func NewEntitlement(resource *v2.Resource, name, purposeStr string, + entitlementOptions ...EntitlementOption) *v2.Entitlement + +// Generate entitlement ID +func NewEntitlementID(resource *v2.Resource, permission string) string +``` + +**Entitlement Options:** + +```go +func WithAnnotation(msgs ...proto.Message) EntitlementOption +func WithGrantableTo(grantableTo ...*v2.ResourceType) EntitlementOption +func WithDisplayName(displayName string) EntitlementOption +func WithDescription(description string) EntitlementOption +``` + +### Grant builder + +```go +// Create grant linking principal to entitlement +func NewGrant(resource *v2.Resource, entitlementName string, + principal GrantPrincipal, grantOptions ...GrantOption) *v2.Grant + +// Generate grant ID +func NewGrantID(principal GrantPrincipal, entitlement *v2.Entitlement) string +``` + +**Grant Options:** + +```go +func WithGrantMetadata(metadata map[string]interface{}) GrantOption +func WithExternalPrincipalID(externalID *v2.ExternalId) GrantOption +func WithAnnotation(msgs ...proto.Message) GrantOption +``` + +## Pagination + +### Token types + +```go +type Token struct { + Size int // Page size + Token string // Opaque token from API +} + +type StreamToken struct { + Size int // Page size + Cursor string // Stream cursor +} + +type StreamState struct { + Cursor string // Next cursor + HasMore bool // More events available +} +``` + +### Pagination bag + +For nested or multi-resource pagination: + +```go +type Bag struct { + // Internal state stack +} + +func (pb *Bag) Push(state PageState) +func (pb *Bag) Pop() *PageState +func (pb *Bag) Next(pageToken string) error +func (pb *Bag) NextToken(pageToken string) (string, error) +func (pb *Bag) Current() *PageState +func (pb *Bag) Marshal() (string, error) +func (pb *Bag) Unmarshal(data string) error +``` + +## HTTP client utilities + +### Creating clients + +```go +func NewClient(ctx context.Context, options ...Option) (*http.Client, error) + +// Options +func WithTLSClientConfig(tlsConfig *tls.Config) Option +func WithLogger(log bool, logger *zap.Logger) Option +func WithUserAgent(userAgent string) Option +``` + +### Caching + +```go +func CreateCacheKey(req *http.Request) (string, error) +func ClearCaches(ctx context.Context) error +``` + +Features: built-in response caching, rate limit extraction, automatic retry with exponential backoff on 429. + +## Session store + +For caching data across pagination calls: + +```go +type SessionStore interface { + Get(ctx context.Context, key string) (interface{}, error) + Set(ctx context.Context, key string, value interface{}) error + Clear(ctx context.Context, filters ...SessionFilter) error + Stats(ctx context.Context) StoreStats +} +``` + +**Accessing in sync methods (V2):** + +```go +func (u *userBuilder) List(ctx context.Context, parentID *v2.ResourceId, + opts resource.SyncOpAttrs) ([]*v2.Resource, *resource.SyncOpResults, error) { + + // Use session store for caching + if cached, err := opts.Session.Get(ctx, "user_cache"); err == nil { + // Use cached data + } + + // Store for next call + opts.Session.Set(ctx, "user_cache", userData) +} +``` + +## Capabilities matrix + +Capabilities are automatically detected based on which interfaces you implement: + +| Capability | Interface required | +|------------|-------------------| +| `CAPABILITY_SYNC` | ResourceSyncer | +| `CAPABILITY_TARGETED_SYNC` | ResourceTargetedSyncer | +| `CAPABILITY_PROVISION` | ResourceProvisioner or ResourceProvisionerV2 | +| `CAPABILITY_ACCOUNT_PROVISIONING` | AccountManager | +| `CAPABILITY_RESOURCE_CREATE` | ResourceManager | +| `CAPABILITY_RESOURCE_DELETE` | ResourceManager, ResourceDeleter, or ResourceDeleterV2 | +| `CAPABILITY_CREDENTIAL_ROTATION` | CredentialManager | +| `CAPABILITY_EVENT_FEED_V2` | EventFeed | +| `CAPABILITY_TICKETING` | TicketManager (with WithTicketingEnabled option) | +| `CAPABILITY_ACTIONS` | CustomActionManager | + +## Quick reference + +### Import paths + +```go +import ( + "github.com/conductorone/baton-sdk/pkg/connectorbuilder" + "github.com/conductorone/baton-sdk/pkg/pagination" + "github.com/conductorone/baton-sdk/pkg/types/entitlement" + "github.com/conductorone/baton-sdk/pkg/types/grant" + "github.com/conductorone/baton-sdk/pkg/types/resource" + "github.com/conductorone/baton-sdk/pkg/uhttp" + "github.com/conductorone/baton-sdk/pb/c1/connector/v2" +) +``` + +### Common patterns + +| Task | Functions | +|------|-----------| +| Create resource type | `resource.NewResourceType(name, traits)` | +| Create resource | `resource.NewResource(name, type, id, opts...)` | +| Add user trait | `resource.WithUserTrait(resource.WithEmail(...))` | +| Create entitlement | `entitlement.NewAssignmentEntitlement(resource, name)` | +| Create grant | `grant.NewGrant(resource, entitlement, principal)` | +| Handle pagination | `pagination.Bag{}` with Marshal/Unmarshal | + + + + How to use these interfaces in practice + + + CLI flags and environment variables + + diff --git a/developer/c1-api.mdx b/developer/c1-api.mdx new file mode 100644 index 0000000..9a302b4 --- /dev/null +++ b/developer/c1-api.mdx @@ -0,0 +1,360 @@ +--- +title: "C1 API integration reference" +sidebarTitle: "C1 API" +description: "How connectors communicate with the ConductorOne platform in daemon mode." +--- + +When running in daemon mode, connectors communicate with ConductorOne via gRPC. The SDK handles all API communication - connector developers implement interfaces, the SDK calls your code when tasks arrive. + +This document describes the C1 APIs that connectors use internally. You don't call these APIs directly; the SDK manages them. + +## Communication architecture + +``` +ConductorOne Platform Connector (Daemon Mode) ++---------------------+ +------------------------+ +| | | | +| Task Queue |<--- gRPC ---------| BatonServiceClient | +| (Sync, Grant, etc) | | | +| |--- Task ---------->| Task Handler | +| | | - SyncTaskHandler | +| Upload Service |<--- c1z upload ---| ConnectorBuilderV2 | +| | | | +| Heartbeat |<--- keepalive ----| HeartbeatLoop | +| | | | ++---------------------+ +------------------------+ +``` + +**Key points:** +- Connectors poll for tasks (pull model, not push) +- Tasks include: Sync, Grant, Revoke, CreateAccount, DeleteResource, RotateCredential +- Results upload via streaming (c1z files in 512KB chunks) +- Heartbeats keep tasks alive during long operations + +## BatonServiceClient + +The client interface used by connectors: + +```go +type BatonServiceClient interface { + // Initial connection handshake + Hello(ctx context.Context, req *v1.BatonServiceHelloRequest, + ) (*v1.BatonServiceHelloResponse, error) + + // Poll for next task + GetTask(ctx context.Context, req *v1.BatonServiceGetTaskRequest, + ) (*v1.BatonServiceGetTaskResponse, error) + + // Keep task alive during processing + Heartbeat(ctx context.Context, req *v1.BatonServiceHeartbeatRequest, + ) (*v1.BatonServiceHeartbeatResponse, error) + + // Report task completion + FinishTask(ctx context.Context, req *v1.BatonServiceFinishTaskRequest, + ) (*v1.BatonServiceFinishTaskResponse, error) + + // Upload sync output (c1z file) + Upload(ctx context.Context, task *v1.Task, r io.ReadSeeker) error +} +``` + +## Task types + +### Sync tasks + +Full or targeted data sync from the external system: + +```go +type SyncTask struct { + SyncId string + ResourceTypes []string // Types to sync (empty = all) + ResourceIds []string // Specific resources (targeted sync) + SkipEntitlements bool + SkipGrants bool +} +``` + +**Connector produces:** c1z file uploaded to ConductorOne + +### Grant task + +Provision access to a user: + +```go +type GrantTask struct { + Entitlement *v2.Entitlement // What access to grant + Principal *v2.Resource // Who receives access +} +``` + +### Revoke task + +Remove access from a user: + +```go +type RevokeTask struct { + Grant *v2.Grant // The grant to revoke +} +``` + +### CreateAccount task + +Provision a new user account (JIT): + +```go +type CreateAccountTask struct { + AccountInfo *v2.AccountInfo + CredentialOptions *v2.LocalCredentialOptions +} +``` + +**Returns to ConductorOne:** Created account, credentials (encrypted by SDK) + +### DeleteResource task + +```go +type DeleteResourceTask struct { + ResourceId *v2.ResourceId + ParentResourceId *v2.ResourceId // V2 only +} +``` + +### RotateCredential task + +```go +type RotateCredentialTask struct { + ResourceId *v2.ResourceId + CredentialOptions *v2.LocalCredentialOptions +} +``` + +**Returns to ConductorOne:** New credentials (encrypted by SDK) + +## Task lifecycle + +### Polling loop + +``` +Connector ConductorOne + | | + |--- Hello (identify connector) -------->| + |<-- HelloResponse (config) -------------| + | | + |--- GetTask (poll for work) ----------->| + |<-- Task or NoTask --------------------| + | | + [If task received] | + | | + |--- Heartbeat (keep alive) ------------>| (every 30s) + |<-- HeartbeatResponse ------------------| + | | + |--- FinishTask (report result) -------->| + |<-- FinishTaskResponse -----------------| + | | + |--- Upload (c1z file, if sync) -------->| + |<-- UploadComplete --------------------| +``` + +### Heartbeat intervals + +```go +var ( + maxHeartbeatInterval = 5 * time.Minute + minHeartbeatInterval = 1 * time.Second + defaultHeartbeatInterval = 30 * time.Second +) +``` + +ConductorOne can adjust heartbeat interval per-task. If heartbeats stop, the task may be reassigned to another connector instance. + +### Task completion + +```go +// Successful completion +FinishTask(ctx, &v1.BatonServiceFinishTaskRequest{ + TaskId: task.Id, + Status: v1.TaskStatus_TASK_STATUS_SUCCESS, + Result: resultProto, +}) + +// Failed completion +FinishTask(ctx, &v1.BatonServiceFinishTaskRequest{ + TaskId: task.Id, + Status: v1.TaskStatus_TASK_STATUS_FAILED, + Error: errorMessage, +}) +``` + +## Upload mechanism + +Sync results (c1z files) upload via streaming: + +```go +const fileChunkSize = 512 * 1024 // 512KB chunks + +func (c *client) Upload(ctx context.Context, task *v1.Task, r io.ReadSeeker) error { + stream, err := c.client.Upload(ctx) + + for { + chunk := make([]byte, fileChunkSize) + n, err := r.Read(chunk) + if err == io.EOF { + break + } + stream.Send(&v1.BatonServiceUploadAssetRequest{ + Data: chunk[:n], + }) + } + + return stream.CloseAndRecv() +} +``` + +**Upload characteristics:** +- Streaming (not buffered in memory) +- 512KB chunks +- Resumable on network errors +- Compressed c1z format + +## Authentication + +### Client credentials + +Connectors authenticate using OAuth2 client credentials: + +```bash +./baton-myservice \ + --client-id "your-client-id" \ + --client-secret "your-client-secret" +``` + +The SDK handles: +- Token acquisition from ConductorOne's OAuth endpoint +- Token refresh before expiration +- Token injection into request metadata + +### Host identification + +Connectors identify themselves to ConductorOne: + +```go +hostId := os.Getenv("BATON_HOST_ID") +if hostId == "" { + hostId, _ = os.Hostname() +} +``` + +This helps ConductorOne track which host is running which connector instance. + +## Error handling + +### Retryable vs non-retryable + +```go +var ( + ErrTaskCancelled = errors.New("task was cancelled") + ErrTaskHeartbeatFailed = errors.New("task failed heartbeat") + ErrTaskNonRetryable = errors.New("task failed and is non-retryable") +) + +taskMaximumHeartbeatFailures = 10 +``` + +### Error flow + +| Connector Error | SDK Handling | ConductorOne Action | +|-----------------|--------------|---------------------| +| Temporary failure | Retry with backoff | Task stays queued | +| Permanent failure | FinishTask(FAILED) | Task marked failed | +| Heartbeat timeout | Task abandoned | Reassign to other instance | +| Cancelled by ConductorOne | Stop processing | Task cancelled | + +### Annotations for error context + +Return annotations to provide context: + +```go +func (g *groupBuilder) Grant(ctx context.Context, principal *v2.Resource, + entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) { + + err := g.client.AddMember(ctx, groupID, userID) + if err != nil { + annos := annotations.New() + annos.Append(&v2.ErrorAnnotation{ + Message: fmt.Sprintf("API error: %v", err), + Code: "API_ERROR", + }) + return nil, annos, err + } + + return grants, nil, nil +} +``` + +## Debugging API communication + +### Enable debug logging + +```bash +./baton-myservice \ + --client-id ID \ + --client-secret SECRET \ + --log-level debug +``` + +Debug output includes: +- Task received notifications +- Heartbeat timing +- Upload progress +- API response codes + +### Common issues + +| Symptom | Likely Cause | Solution | +|---------|--------------|----------| +| `authentication error` | Invalid client credentials | Verify client-id/secret | +| `task heartbeat failed` | Processing too slow | Optimize or add heartbeats | +| `connection refused` | Network/firewall issue | Check connectivity to ConductorOne | +| `task cancelled` | Task timeout or user cancel | Check task duration | +| `upload failed` | Large c1z or network issue | Check file size, retry | + +## Quick reference + +### Daemon mode checklist + +```bash +# Required for daemon mode +--client-id "your-oauth-client-id" +--client-secret "your-oauth-client-secret" + +# Optional +--skip-full-sync # Don't process sync tasks +--log-level debug # Verbose logging +``` + +### Task processing flow + +1. **Hello** - Connector identifies itself +2. **GetTask** - Poll for work (returns task or empty) +3. **Process** - Execute sync/grant/revoke/etc +4. **Heartbeat** - Keep task alive during processing +5. **FinishTask** - Report success or failure +6. **Upload** - Send c1z file (sync tasks only) + +### SDK handles + +- OAuth2 token management +- Connection pooling +- Heartbeat scheduling +- Chunk-based uploads +- Retry with backoff +- Error classification + + + + CLI flags and environment variables + + + Debug connector issues + + diff --git a/developer/cel-expressions.mdx b/developer/cel-expressions.mdx new file mode 100644 index 0000000..4494757 --- /dev/null +++ b/developer/cel-expressions.mdx @@ -0,0 +1,135 @@ +--- +title: "CEL expressions" +sidebarTitle: "CEL reference" +description: "Common Expression Language reference for baton-http and baton-sql data transformation." +--- + +Both meta-connectors use the Common Expression Language (CEL) for data transformation. CEL is a simple, fast, and secure expression language - a safer alternative to embedding arbitrary code in your config. + +## Basic field mapping + +```yaml +map: + id: ".id" # Direct field access + display_name: ".first_name + ' ' + .last_name" # String concatenation + status: ".is_active ? 'enabled' : 'disabled'" # Ternary +``` + +## Available functions + +| Function | Purpose | Example | +|----------|---------|---------| +| `lowercase()` | Convert to lowercase | `lowercase(.email)` | +| `uppercase()` | Convert to uppercase | `uppercase(.code)` | +| `titlecase()` | Title case | `titlecase(.name)` | +| `trim()` | Remove whitespace | `trim(.value)` | +| `match()` | Regex matching | `match(.email, ".*@corp\\.com")` | +| `extract()` | Extract regex group | `extract(.urn, "user-([0-9]+)")` | +| `replace()` | String replacement | `replace(.name, {"old": "_", "new": "-"})` | +| `get()` | Get with default | `get(.optional, "default")` | +| `has()` | Check field exists | `has(input.employee_id)` | +| `parse_json()` | Parse JSON string | `parse_json(.metadata).type` | +| `json_path()` | Extract from JSON | `json_path(.data, "user.name")` | + +## Context variables + +Available variables depend on context: + +| Variable | Available in | Purpose | +|----------|--------------|---------| +| `item` or `.column` | List/Grants | Current row/item | +| `resource` | Grants, Provisioning | Current resource | +| `principal` | Provisioning | User being granted/revoked | +| `entitlement` | Provisioning | Entitlement being modified | +| `input` | Account creation | User-provided form values | +| `password` | Account creation | Generated password | + +## Common patterns + +### Conditional field mapping + +```yaml +account_type: ".type == 'employee' ? 'human' : 'service'" +``` + +### Null handling + +```yaml +last_login: ".last_login != null ? string(.last_login) : ''" +``` + +### String operations + +```yaml +# Lowercase email +email: "lowercase(.email)" + +# Build full name +display_name: ".first_name + ' ' + .last_name" + +# Extract username from email +username: "extract(.email, '([^@]+)@.*')" +``` + +### Type conversion + +```yaml +# Convert numeric ID to string +id: "string(.numeric_id)" +``` + +### Field existence check + +```yaml +# Use optional field with fallback +department: "has(input.department) ? input.department : 'Unknown'" +``` + +### Skip conditions in grants + +```yaml +grants: + - query: "SELECT * FROM role_members WHERE role_id = ?" + map: + - skip_if: ".user_type == 'system'" # Skip system accounts + principal_id: ".user_id" + principal_type: "user" + entitlement_id: "member" +``` + +## Quick reference + +```yaml +# Direct field access +".field_name" + +# String concatenation +".first + ' ' + .last" + +# Ternary +".active ? 'enabled' : 'disabled'" + +# Null check +".value != null ? .value : 'default'" + +# Type conversion +"string(.numeric_id)" + +# Field existence check +"has(input.optional_field)" + +# Lowercase/uppercase +"lowercase(.email)" +"uppercase(.code)" +``` + +## Related + + + + baton-http for REST API integration + + + baton-sql for database integration + + diff --git a/developer/community.mdx b/developer/community.mdx new file mode 100644 index 0000000..b812c2a --- /dev/null +++ b/developer/community.mdx @@ -0,0 +1,205 @@ +--- +title: "Community and support" +sidebarTitle: "Community" +description: "Get help, report issues, and contribute to the connector ecosystem." +--- + +You're not alone in this. The Baton community is active and welcoming. + +## Getting help + +### GitHub discussions + +The primary place for questions and discussions is GitHub: + +| Repository | Use for | +|------------|---------| +| [baton-sdk](https://github.com/ConductorOne/baton-sdk/discussions) | SDK questions, general connector development | +| Specific connector repos | Issues with that connector | + +Before asking: +1. Search existing issues and discussions +2. Check this documentation +3. Review the [recipes](/developer/recipes-auth) for common patterns + +### What makes a good question + +Include: +- What you're trying to accomplish +- What you've tried +- Error messages (full text, not screenshots) +- Relevant code snippets +- Connector version and target system + +**Bad:** "My connector doesn't work" + +**Good:** "Sync fails with 'unauthorized' error when listing users. Using baton-okta v0.5.2 with API token auth. Token has read_users scope. Full error: [paste error]" + +### Support channels + +| Channel | Response time | Use for | +|---------|--------------|---------| +| GitHub Issues | Days | Bugs, feature requests | +| GitHub Discussions | Days | Questions, how-to help | +| ConductorOne Support | Hours | Production issues (customers) | + +## Reporting issues + +### Bug reports + +When reporting a bug, include: + +| Information | Why it helps | +|------------|--------------| +| Connector name and version | Reproducibility | +| Steps to reproduce | Root cause analysis | +| Expected vs actual behavior | Understanding the issue | +| Error messages | Debugging | +| Environment (OS, Go version) | Platform-specific issues | + +**Template:** + +```markdown +## Bug Report + +**Connector:** baton-yourservice v1.2.3 +**Environment:** macOS 14.0, Go 1.23 + +**Steps to reproduce:** +1. Configure connector with API key +2. Run `baton-yourservice --api-key $KEY` +3. Observe error + +**Expected:** Sync completes successfully +**Actual:** Sync fails with error: [paste error] + +**Logs:** (if helpful) +``` + +### Feature requests + +For new features: +1. Check if the feature already exists (search issues) +2. Describe the use case, not just the solution +3. Explain why existing approaches don't work + +Good feature requests describe the problem: +- "I need to sync custom attributes from Okta user profiles so that access reviews show department information" + +Not just the solution: +- "Add custom attribute support" + +### Security issues + + +**DO NOT** file security issues publicly. + + +Report security vulnerabilities to: [security@conductorone.com](mailto:security@conductorone.com) + +Security researchers are publicly acknowledged for responsible disclosure. + +## Contributing + +### Quick contribution checklist + +Before submitting a PR: + +- Code follows SDK patterns +- Lint passes: `make lint` +- Tests pass: `make test` +- README documents required permissions +- No credentials in code or logs + +### Contribution flow + +1. Fork repository +2. Create feature branch +3. Make changes +4. Run local validation (build, lint, test) +5. Submit pull request +6. Address review feedback +7. Merge + +See [Publishing](/developer/submit) for the complete contribution workflow. + +## Code of conduct + +The Baton ecosystem follows the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). + +### Our pledge + +We pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. + +### Standards + +**Positive behaviors:** +- Demonstrating empathy and kindness +- Being respectful of differing opinions +- Giving and gracefully accepting constructive feedback +- Accepting responsibility for mistakes and learning from them +- Focusing on what is best for the community + +**Unacceptable behaviors:** +- Sexualized language or imagery +- Trolling, insulting, or derogatory comments +- Public or private harassment +- Publishing others' private information without permission +- Other conduct inappropriate in a professional setting + +### Enforcement + +Report violations to: [open-source@conductorone.com](mailto:open-source@conductorone.com) + +All complaints will be reviewed and investigated promptly and fairly. + +## Governance + +### Decision making + +The Baton ecosystem is maintained by ConductorOne with community input. + +| Decision type | Who decides | +|--------------|-------------| +| SDK changes | ConductorOne team | +| New connectors (ConductorOne org) | ConductorOne team | +| New connectors (your org) | You | +| Feature requests | Maintainers of affected repo | + +### Roadmap + +The project roadmap is tracked through GitHub issues in the [baton-sdk repository](https://github.com/ConductorOne/baton-sdk/issues). + +Community members can: +- View open issues and planned features +- Comment on proposed changes +- Suggest priorities through issue labels + +## Quick reference + +### Key links + +| Resource | URL | +|----------|-----| +| baton-sdk | https://github.com/ConductorOne/baton-sdk | +| Security Reports | security@conductorone.com | +| Code of Conduct Reports | open-source@conductorone.com | + +### Issue labels + +| Label | Meaning | +|-------|---------| +| `bug` | Something isn't working | +| `enhancement` | New feature or improvement | +| `documentation` | Documentation improvements | +| `good first issue` | Good for newcomers | +| `help wanted` | Maintainers want community help | + + + + Share your connector + + + Implementation guide + + diff --git a/developer/concepts.mdx b/developer/concepts.mdx new file mode 100644 index 0000000..9db8bf1 --- /dev/null +++ b/developer/concepts.mdx @@ -0,0 +1,292 @@ +--- +title: "Core concepts" +sidebarTitle: "Core concepts" +description: "Build a correct mental model of the resource/entitlement/grant graph and how the SDK walks it." +--- + +Every target system has its own vocabulary (teams vs groups, roles vs permission sets, projects vs workspaces). This diversity is a strength - each system evolved for its specific purpose. But it creates a challenge: how do you get unified visibility across all of them? + +Baton solves this by normalizing every system into a consistent shape so ConductorOne can ask one question across all systems: who has access to what? Without this normalization, each system would be an island - auditors would see chaos, and access reviews would require expert knowledge of every platform. + +The minimal "connector surface area" is expressed through the SDK's `ResourceSyncer` interface: list resources, list entitlements, list grants. + +## The access graph + +Your connector produces an access graph that powers access reviews, certification campaigns, provisioning workflows, and compliance reporting. This single data structure drives everything ConductorOne does. The graph has three main node/edge types: + +``` ++------------------+ +------------------+ +------------------+ +| RESOURCES | | ENTITLEMENTS | | GRANTS | +|------------------| |------------------| |------------------| +| Things that exist| | Permissions that | | Who has what | +| | | can be assigned | | | +| - Users | --> | - Admin access | --> | Alice has Admin | +| - Groups | | - Read access | | on Database X | +| - Roles | | - Member of team | | | +| - Applications | | | | Bob is Member | ++------------------+ +------------------+ | of Team Y | + +------------------+ +``` + +- **Resources**: things that exist (users, groups, apps, roles, projects, etc.) +- **Entitlements**: permissions you can assign on a resource (member, admin, read, etc.) +- **Grants**: facts connecting principals to entitlements (Alice is a member of Engineering) + +This is not a theoretical model: these are concrete protobuf types and services in the SDK. + +### Hierarchical resources + +Most access management systems flatten everything into a single list. Baton takes a different approach: resources can have parent-child relationships. This preserves the natural structure of your target systems. + +Consider GitHub: organizations contain repositories, repositories have branches. Or AWS: accounts contain services, services have resources. When your connector models these hierarchies, ConductorOne can: + +- Show access in context (this role applies to *this* project, not globally) +- Enable scoped access reviews (review all access within a single org unit) +- Support inheritance patterns where they exist in the target system + +You express hierarchy through the `parentResourceID` parameter in your `List()` method. The SDK calls your `List()` first with no parent (top-level resources), then again for each parent that might have children. This inversion of control means you describe the structure; the SDK walks it. + +## Resource types and traits + +Every resource has a **resource type** (string id) and can declare **traits** that tell ConductorOne how to interpret it. Traits let ConductorOne understand *what kind of thing* a resource is, even when different systems call it different names. + +```go +var userResourceType = &v2.ResourceType{ + Id: "user", + DisplayName: "User", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_USER}, +} +``` + +The trait enum includes: +- `TRAIT_USER` +- `TRAIT_GROUP` +- `TRAIT_ROLE` +- `TRAIT_APP` +- `TRAIT_SECRET` + +Traits are optional for custom resource types, but they unlock powerful features. When you mark a resource with `TRAIT_USER`, ConductorOne knows it can correlate that resource with users from other systems, display it in user-centric views, and apply user-specific policies. + +| Trait | Use for | +|-------|---------| +| `TRAIT_USER` | Individual accounts | +| `TRAIT_GROUP` | Collections of users | +| `TRAIT_ROLE` | Permission sets | +| `TRAIT_APP` | Applications or services | +| `TRAIT_SECRET` | Credentials or tokens | + +### Entitlements + +Entitlements define *what can be granted*. They're attached to resources - permissions don't float free in the system, they belong to something specific. + +```go +// A group offers membership as an entitlement +entitlement := &v2.Entitlement{ + Id: "member", + DisplayName: "Member", + Resource: groupResource, +} +``` + +One resource can offer multiple entitlements. A GitHub repository might offer: read, write, maintain, admin. + +### Grants + +Grants record *who has what*: + +```go +// Alice is a member of the engineering team +grant := &v2.Grant{ + Principal: aliceResource, + Entitlement: memberEntitlement, +} +``` + +Grants connect a **principal** (usually a user) to an **entitlement** (a permission on a resource). + +## The sync lifecycle + +Understanding the sync lifecycle helps you write cleaner code. The SDK orchestrates everything; you implement the callbacks. The "shape" of work per resource type is: +- Define type (`ResourceType`) +- List instances (`List`) +- For each resource instance, list entitlements (`Entitlements`) +- For each resource instance, list grants (`Grants`) + +Each method is paginated: you return a list of results and a `nextPageToken` string (empty when done). + + +The SDK detects a common pagination bug: returning the same next-page token you were given. This prevents infinite loops from subtle bugs. + + +**Key insight:** The SDK processes ALL resource types together for each stage, not one resource type completely before the next. This "inversion of control" pattern keeps your connector code focused on data transformation rather than orchestration logic. + +``` +Stage 1: ResourceType() + - SDK learns what resource types exist (user, group, role, etc.) + +Stage 2: List() + - SDK fetches all instances of each resource type + - Returns: 127 users, 23 groups, 15 roles + +Stage 3: Entitlements() + - SDK asks each resource what entitlements it offers + - Returns: group-A offers "member", role-X offers "assigned" + +Stage 4: Grants() + - SDK discovers who has each entitlement + - Returns: alice has "member" on group-A, bob has "assigned" on role-X +``` + +### The sync pipeline + +When a connector runs, data flows through several stages. The clean separation between what you control and what ConductorOne controls makes the system reliable and testable: + +``` ++-----------+ +-----------+ +-------+ +-------+ +--------+ +| External | | Connector | | .c1z | | Sync | | Domain | +| System | -> | (yours) | -> | File | -> |Service| -> | Objects| ++-----------+ +-----------+ +-------+ +-------+ +--------+ + | + "Uplift" +``` + +1. **Fetch** - Your connector calls the external API +2. **Transform** - Your connector creates Resource/Entitlement/Grant objects +3. **Output** - SDK writes objects to a .c1z file (gzip-compressed SQLite) +4. **Ingest** - ConductorOne's sync service reads the .c1z file +5. **Uplift** - Raw connector records become domain objects (Apps, Resources, Grants) + +**What you control:** Steps 1-3. Your connector fetches, transforms, and outputs. + +**What ConductorOne controls:** Steps 4-5. The sync service and uplift process. + +### ID correlation + +ConductorOne needs to know whether a resource in this sync is the same resource from a previous sync. This is where the `RawId` annotation matters. + +When you build a resource, include its external system ID: + +```go +resource, _ := resourceBuilder.NewGroupResource( + group.Name, + groupResourceType, + group.Id, // Used internally by SDK + []resource.GroupTraitOption{}, +) +// Add the external system's ID for correlation +resource.WithAnnotation(&v2.RawId{Id: group.Id}) +``` + +The `RawId` annotation carries the external system's identifier through the pipeline: +- **Connector**: Sets `RawId` annotation on the resource +- **Sync storage**: Stored as `external_id` on the connector record +- **Domain objects**: Tracked in `source_connector_ids` map + +This enables ConductorOne to: +- Correlate resources across syncs (same ID = same resource) +- Track which connector discovered which resource +- Support pre-sync reservation patterns + +**What value to use:** The external system's native, stable identifier. For Okta, that's `app.Id`. For AWS, that's the ARN. For GCP, that's the project ID. + +### ID vocabulary + +These terms appear throughout ConductorOne when discussing identity correlation: + +| Term | Where it appears | Purpose | +|------|-----------------|---------| +| **RawId** | Connector output annotation | External system's stable identifier, set by connector | +| **external_id** | Sync layer storage | Same value stored on ConnectorResource records | +| **source_connector_ids** | Domain objects | Map of connector_id to external_id for multi-connector scenarios | +| **raw_baton_id** | Domain objects | Set after merge; the canonical external ID | +| **match_baton_id** | Terraform/API | Pre-sync reservation (allows creating objects before connector discovers them) | + +The flow is: `RawId` (connector) -> `external_id` (sync) -> `source_connector_ids` or `raw_baton_id` (domain). + +## Modeling decisions + +Your modeling choices expand or constrain what your organization can do with access control. Two connectors can both be "correct" and still produce very different experiences in ConductorOne: + +- **Entitlement granularity** + - Fine-grained (read/write/admin): more precision in reviews and provisioning, more total grants and API calls. + - Coarse-grained (access/no-access): simpler, but you can't request/revoke specific privilege levels. + +- **Capability surface** + - Sync-only vs sync + provision (Grant/Revoke/Create/Delete) + - Check per connector/version rather than assuming. + +| Decision | Impact | +|----------|--------| +| **Granular entitlements** (read, write, admin separately) | More control, more complexity | +| **Coarse entitlements** (access vs no access) | Simpler, less visibility | +| **Group membership as grants** | Users can request to join groups | +| **Roles as resources** | Roles can be granted/revoked | + +**Guidance:** Model what matters for access decisions. If you need to revoke admin access separately from read access, make them separate entitlements. + +## Constraints and guardrails + +The SDK includes guardrails that catch common mistakes early: + +- **Pagination invariants**: your next token must progress; the SDK will detect and error on "same token" loops. +- **Optional advanced behaviors exist but are not universal**: targeted sync (`Get`), account management, resource deletion, credential rotation. Start with the basics; add these when you need them. + +### Error handling + +If an API call fails, return an error. The SDK handles retries for transient failures. For permanent failures (bad credentials, missing permissions), the sync stops with a clear error message. + +### Rate limiting + +Most APIs have rate limits. The SDK's HTTP client handles backoff automatically, but be mindful of page sizes - smaller pages mean more requests. + +## Execution modes + +Connectors can run in different modes: + +| Mode | Trigger | Behavior | +|------|---------|----------| +| **One-shot** | No `--client-id` | Run, sync to file, exit | +| **Daemon** | `--client-id` provided | Connect to C1, process tasks continuously | + +See [Deployment](/baton/deploy) for operational details on each mode. + +## Next steps + + + + Project structure and implementation patterns + + + Grant, revoke, and account lifecycle operations + + + Configure baton-http or baton-sql without writing Go + + + Run modes, credentials, monitoring + + + +## Quick reference + +### Resource types + +| Type | Description | Example | +|------|-------------|---------| +| User | Individual accounts | alice@company.com | +| Group | Collections of users | engineering-team | +| Role | Permission bundles | admin-role | +| App | Applications | billing-service | +| Team | GitHub/Slack style teams | platform-team | +| Project | Scoped containers | production-project | + +### The four methods + +Every resource syncer implements: + +| Method | Purpose | Called | +|--------|---------|--------| +| `ResourceType()` | Define the resource type | Once per sync | +| `List()` | Fetch all instances | May be called multiple times (pagination) | +| `Entitlements()` | What permissions does this offer? | Once per resource instance | +| `Grants()` | Who has those permissions? | Once per entitlement | diff --git a/developer/config-schema.mdx b/developer/config-schema.mdx new file mode 100644 index 0000000..280eb6b --- /dev/null +++ b/developer/config-schema.mdx @@ -0,0 +1,359 @@ +--- +title: "Configuration schema reference" +sidebarTitle: "Configuration" +description: "All configuration options for connectors: CLI flags, environment variables, and config files." +--- + +Every baton connector uses a type-safe configuration system. Configuration flows through three layers: +1. **CLI flags** (`--domain example.com`) +2. **Environment variables** (`BATON_DOMAIN=example.com`) +3. **Config files** (YAML) + +CLI flags take precedence over environment variables, which take precedence over config files. + +## Standard flags (all connectors) + +Every connector automatically has these flags via the SDK. + +### Output and logging + +| Flag | Short | Type | Default | Description | +|------|-------|------|---------|-------------| +| `--file` | `-f` | string | `sync.c1z` | Output file path | +| `--log-level` | | string | `info` | `debug`, `info`, `warn`, `error` | +| `--log-format` | | string | auto | `json` or `console` (auto-detects TTY) | + +### Daemon mode (service) + +| Flag | Type | Description | +|------|------|-------------| +| `--client-id` | string | ConductorOne OAuth client ID (enables daemon mode) | +| `--client-secret` | string | ConductorOne OAuth client secret | +| `--skip-full-sync` | bool | Disable full sync in daemon mode | + +### Provisioning operations + +| Flag | Type | Description | +|------|------|-------------| +| `--provisioning` | bool | Enable provisioning mode | +| `--grant-entitlement` | string | Entitlement ID to grant | +| `--grant-principal` | string | Resource ID receiving grant | +| `--grant-principal-type` | string | Resource type of principal | +| `--revoke-grant` | string | Grant ID to revoke | + +### Account management + +| Flag | Type | Description | +|------|------|-------------| +| `--create-account-login` | string | Login for new account | +| `--create-account-email` | string | Email for new account | +| `--create-account-profile` | string | JSON profile data | +| `--delete-resource` | string | Resource ID to delete | +| `--delete-resource-type` | string | Resource type to delete | +| `--rotate-credentials` | string | Resource ID for rotation | +| `--rotate-credentials-type` | string | Resource type for rotation | + +### Targeted sync + +| Flag | Type | Description | +|------|------|-------------| +| `--sync-resources` | []string | Specific resource IDs to sync | +| `--sync-resource-types` | []string | Resource types to sync | +| `--skip-entitlements-and-grants` | bool | Skip E&G during sync | +| `--skip-grants` | bool | Skip only grants | + +### Opentelemetry (operators) + +| Flag | Type | Description | +|------|------|-------------| +| `--otel-collector-endpoint` | string | OTEL collector URL | +| `--otel-tracing-disabled` | bool | Disable tracing | +| `--otel-logging-disabled` | bool | Disable OTEL logging | + +## Environment variables + +All flags map to environment variables with the `BATON_` prefix: + +```bash +--domain -> BATON_DOMAIN +--api-key -> BATON_API_KEY +--skip-ous -> BATON_SKIP_OUS +--log-level -> BATON_LOG_LEVEL +--client-id -> BATON_CLIENT_ID +--client-secret -> BATON_CLIENT_SECRET +``` + +**Rules:** +- Prefix: `BATON_` +- Dashes become underscores: `--base-dn` -> `BATON_BASE_DN` +- Case insensitive: `BATON_DOMAIN` = `baton_domain` + +**Example:** + +```bash +export BATON_DOMAIN=example.okta.com +export BATON_API_TOKEN=00abc123... +export BATON_LOG_LEVEL=debug + +./baton-okta # Uses env vars +``` + +## Config file format + +Connectors can read YAML config files: + +```yaml +# ~/.baton/config.yaml or specified via --config + +domain: example.okta.com +api-token: "00abc123..." +log-level: debug + +# Arrays +skip-groups: + - "Test Group" + - "Temp Users" + +# Maps +custom-attributes: + department: "Engineering" + cost_center: "CC-1234" +``` + +**Config file locations (checked in order):** +1. Path specified via `--config` flag +2. `./baton.yaml` +3. `~/.baton/config.yaml` + +## Field types + +When building a connector, you define fields using these types: + +### StringField + +```go +field.StringField("domain", + field.WithRequired(true), + field.WithDescription("Your Okta domain"), + field.WithPlaceholder("example.okta.com"), +) +``` + +CLI: `--domain example.okta.com` +Env: `BATON_DOMAIN=example.okta.com` + +### BoolField + +```go +field.BoolField("ldaps", + field.WithDescription("Use LDAPS encryption"), + field.WithDefaultValue(false), +) +``` + +CLI: `--ldaps` or `--ldaps=true` +Env: `BATON_LDAPS=true` + +### IntField + +```go +field.IntField("port", + field.WithDescription("LDAP port"), + field.WithDefaultValue(389), +) +``` + +CLI: `--port 636` +Env: `BATON_PORT=636` + +### StringSliceField + +```go +field.StringSliceField("skip-groups", + field.WithDescription("Groups to exclude from sync"), +) +``` + +CLI: `--skip-groups "Group1" --skip-groups "Group2"` +Env: `BATON_SKIP_GROUPS="Group1,Group2"` + +### SelectField (enum) + +```go +field.SelectField("auth-type", []string{"token", "oauth", "basic"}, + field.WithDescription("Authentication method"), + field.WithDefaultValue("token"), +) +``` + +CLI: `--auth-type oauth` + +## Field options + +### Required fields + +```go +field.StringField("api-key", + field.WithRequired(true), +) +``` + +Error if missing: `required flag "api-key" not set` + +### Secret fields + +```go +field.StringField("api-token", + field.WithIsSecret(true), +) +``` + +Secrets are: +- Not printed in debug logs +- Masked as `***` in GUI +- Stored securely in config + +### Default values + +```go +field.IntField("timeout", + field.WithDefaultValue(30), +) +``` + +### Hidden fields + +```go +field.StringField("internal-id", + field.WithHidden(true), // Not shown in --help +) +``` + +### Shorthand flags + +```go +field.StringField("file", + field.WithShortHand("f"), // Enables -f shortcut +) +``` + +## Validation rules + +### String validation + +```go +field.StringField("domain", + field.WithString(func(r *field.StringRuler) { + r.MinLen(3) + r.MaxLen(253) + r.Pattern(`^[a-z0-9.-]+$`) + r.Prefix("https://") + r.Suffix(".com") + r.Contains("okta") + }), +) +``` + +### Integer validation + +```go +field.IntField("port", + field.WithInt(func(r *field.IntRuler) { + r.Gt(0) + r.Lt(65536) + r.Gte(1024) + r.Lte(49151) + r.In([]int64{80, 443, 8080}) + r.NotIn([]int64{22, 23}) + }), +) +``` + +## Field relationships + +### Required together + +Both must be provided or neither: + +```go +field.FieldsRequiredTogether( + bindUserField, + bindPasswordField, +) +``` + +### Mutually exclusive + +Only one can be provided: + +```go +field.FieldsMutuallyExclusive( + skipGroupsField, + onlyGroupsField, +) +``` + +### At least one required + +```go +field.FieldsAtLeastOneUsed( + apiKeyField, + oauthTokenField, + usernameField, +) +``` + +### Dependent fields + +```go +field.FieldsDependentOn( + []field.SchemaField{proxyField}, + []field.SchemaField{proxyUserField, proxyPasswordField}, +) +``` + +## Configuration precedence + +When the same setting is specified multiple ways: + +``` +CLI flag (highest priority) + | + v +Environment variable + | + v +Config file + | + v +Default value (lowest priority) +``` + +## Quick reference + +### Common patterns + +| Use Case | Flags | +|----------|-------| +| Debug sync | `--log-level debug` | +| Custom output | `-f /path/to/output.c1z` | +| Daemon mode | `--client-id ID --client-secret SECRET` | +| Targeted sync | `--sync-resource-types user,group` | +| Test provisioning | `-p --grant-entitlement ENT --grant-principal PRIN` | + +### Check connector-specific flags + +```bash +./baton-okta --help +./baton-github --help +./baton-aws --help +``` + + + + SDK interface reference + + + Platform communication details + + diff --git a/developer/debugging.mdx b/developer/debugging.mdx new file mode 100644 index 0000000..8131d0e --- /dev/null +++ b/developer/debugging.mdx @@ -0,0 +1,363 @@ +--- +title: "Troubleshooting" +sidebarTitle: "Debugging" +description: "Debug common connector issues: auth failures, pagination loops, empty syncs, performance problems." +--- + +Something broke? Start here. Most connector issues fall into recognizable patterns. + +## Debugging workflow + +When a connector fails, follow this diagnostic sequence: + +1. **Check logs** - What error message? +2. **Inspect output** - Did sync produce a .c1z file? +3. **Verify config** - Are credentials valid? +4. **Test API access** - Can you hit the target API directly? +5. **Isolate the issue** - Which resource type fails? + +### Enable debug logging + +```bash +./baton-yourservice --log-level debug --log-format json +``` + +Log levels: `debug`, `info`, `warn`, `error` + +### Inspect sync output + +Use the `baton` CLI to inspect what was synced: + +```bash +baton resources -f sync.c1z # List all resources +baton grants -f sync.c1z # List grants +baton entitlements -f sync.c1z # List entitlements +baton stats -f sync.c1z # Get stats +``` + +## Common errors + +### Pagination loop detected + +**Error:** `next page token is the same as the current page token` + +**Cause:** Your `List()`, `Entitlements()`, or `Grants()` method returned the same pagination token it received. + +```go +// BAD - Returns same token, causes infinite loop +func (u *userBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.GetUsers(ctx) + return resources, pToken.Token, nil, nil // WRONG - returning input token +} + +// GOOD - Returns next page token from API, or empty when done +func (u *userBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, nextPage, err := u.client.GetUsers(ctx, pToken.Token) + return resources, nextPage, nil, nil // CORRECT +} +``` + +The SDK detects this and returns an error to prevent infinite loops. + +### Authentication failed + +**Error:** `401 Unauthorized` or `403 Forbidden` + +**Causes:** +- Invalid or expired credentials +- Missing required scopes/permissions +- Wrong API endpoint + +**Diagnostic steps:** + +```bash +# Test credentials directly +curl -H "Authorization: Bearer $TOKEN" https://api.example.com/v1/users + +# Check token expiration (for JWT tokens, decode and check 'exp' claim) + +# Verify required scopes in target API documentation +``` + +| Issue | Fix | +|-------|-----| +| Expired token | Refresh or regenerate credentials | +| Missing scope | Request additional API permissions | +| Wrong endpoint | Check `--base-url` or environment config | + +### Rate limited + +**Error:** `429 Too Many Requests` or slow sync performance + +**Solutions:** + +1. **Reduce page size** (more pages, smaller requests): + ```go + users, next, err := client.ListUsers(ctx, pageSize: 100) + ``` + +2. **The SDK handles backoff automatically** via `uhttp`: + ```go + httpClient, _ := uhttp.NewBaseHttpClient(ctx) + // Automatic exponential backoff on 429 + ``` + +3. **Add explicit delays** if needed: + ```go + time.Sleep(time.Second) // Between API calls + ``` + +### Resource type mismatch + +**Error:** Grants reference principals that don't exist + +**Cause:** The `ResourceId` in a grant doesn't match any synced resource. + +```go +// BAD - Type ID doesn't match +g := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "User", Resource: member.ID}) // "User" vs "user" + +// GOOD - Type ID matches exactly +g := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "user", Resource: member.ID}) // Matches userBuilder +``` + +**Debug:** +```bash +baton resources -f sync.c1z | grep -i "resource_type" +baton grants -f sync.c1z | grep "principal" +``` + +### Empty sync results + +**Error:** Sync completes but produces no resources + +**Causes:** +- API returns empty results (permissions issue) +- Pagination starts past end of data +- Filter too restrictive + +```bash +baton stats -f sync.c1z +./baton-yourservice --log-level debug 2>&1 | grep -i "response\|count" +``` + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| 0 users | Missing read permission | Check API credentials scope | +| 0 grants | No memberships exist | Verify data exists in target system | +| Partial data | Pagination bug | Check token handling | + +### Connection refused + +**Error:** `connection refused` or `no such host` + +```bash +curl -v https://api.example.com/health # Test connectivity +nslookup api.example.com # Check DNS +./baton-yourservice --base-url https://internal.example.com +``` + +### TLS certificate errors + +**Error:** `x509: certificate signed by unknown authority` + +**For testing only** - skip TLS verification: +```bash +./baton-yourservice --insecure-skip-verify +``` + + +Never use `--insecure-skip-verify` in production. + + +**For production** - add the CA certificate: +```bash +export SSL_CERT_FILE=/path/to/ca-bundle.crt +``` + +## Performance issues + +### Slow sync + +| Cause | Diagnosis | Fix | +|-------|-----------|-----| +| Too many API calls | Debug logs show thousands of requests | Batch where possible | +| No pagination | Single huge request | Implement proper paging | +| N+1 queries | Fetching details one-by-one | Use bulk endpoints | +| Rate limiting | 429 errors in logs | Reduce page size | + +```go +// BAD - N+1 pattern (one API call per user) +for _, user := range users { + details, _ := client.GetUserDetails(ctx, user.ID) +} + +// GOOD - Bulk fetch +details, _ := client.GetUsersWithDetails(ctx, userIDs) +``` + +### Memory issues + +| Cause | Fix | +|-------|-----| +| Loading all data in memory | Use pagination, stream processing | +| Large responses not garbage collected | Process in batches | +| Caching too much | Limit cache size, use LRU | + +## Daemon mode issues + +### Task polling failures + +**Error:** `failed to poll for tasks` or `authentication error` + +```bash +./baton-yourservice \ + --client-id $CLIENT_ID \ + --client-secret $CLIENT_SECRET \ + --log-level debug +``` + +### Connector not receiving tasks + +**Symptoms:** Daemon runs but never processes anything + +Check: +- Connector registered in ConductorOne admin UI +- Sync scheduled +- Correct connector ID + +## Provisioning issues + +### Grant/revoke failures + +**Error:** `error: provisioner not found for resource type` + +**Cause:** Declared provisioning capability but didn't implement the interface. + +```go +// Implement ResourceProvisionerV2 interface +func (g *groupBuilder) Grant(ctx context.Context, + principal *v2.Resource, entitlement *v2.Entitlement) ( + []*v2.Grant, annotations.Annotations, error) { + // Implementation +} + +func (g *groupBuilder) Revoke(ctx context.Context, + grant *v2.Grant) (annotations.Annotations, error) { + // Implementation +} +``` + +Verify capabilities: +```bash +./baton-yourservice capabilities | jq '.resourceTypeCapabilities[] | select(.capabilities | contains(["CAPABILITY_PROVISION"]))' +``` + +## Duplicate objects after sync + +### Symptom + +After running a sync, you see duplicate resources in ConductorOne: one created via Terraform/API and a separate one discovered by the connector. + +### Causes + +1. **RawId not set** - Connector doesn't include the `RawId` annotation +2. **ID format mismatch** - Connector uses a different ID format than expected +3. **match_baton_id value incorrect** - Terraform resource has wrong ID value +4. **Case sensitivity** - IDs may be case-sensitive + +### Resolution + +**Step 1:** Verify the connector outputs RawId annotation: + +```bash +./baton-yourservice --log-level debug 2>&1 | grep -i rawid +``` + +**Step 2:** Check what ID the connector uses: + +| Connector | ID Field | Example | +|-----------|----------|---------| +| Okta | `app.Id` | `0oa1xyz789abcdef0h7` | +| Azure AD | Object ID | `12345678-1234-...` | +| GCP | Resource path | `projects/my-project` | +| AWS | Full ARN | `arn:aws:iam::123...` | + +**Step 3:** Update Terraform to use the exact match: + +```hcl +resource "conductorone_app" "example" { + display_name = "My App" + match_baton_id = "0oa1xyz789abcdef0h7" # Exact value from connector +} +``` + +**Step 4:** Delete duplicate and re-sync. + +## Diagnostic commands + +```bash +# Quick health check +make build && ./dist/*/baton-yourservice \ + --log-level debug \ + --log-format json \ + 2>&1 | head -100 + +# Inspect .c1z file +baton stats -f sync.c1z +baton resources -f sync.c1z --resource-type user +baton resources -f sync.c1z --output-format json > resources.json + +# Compare syncs +./baton-yourservice && mv sync.c1z sync1.c1z +./baton-yourservice && mv sync.c1z sync2.c1z +baton stats -f sync1.c1z +baton stats -f sync2.c1z +``` + +## Getting help + +If you've tried these steps and still have issues: + +1. **Search existing issues** on the connector's GitHub repo +2. **Check SDK issues** at [baton-sdk issues](https://github.com/ConductorOne/baton-sdk/issues) +3. **Open a new issue** with: + - Connector name and version + - Error message (full text) + - Debug logs (sanitized of credentials) + - Steps to reproduce + +## Quick reference + +### Debug flags + +| Flag | Purpose | +|------|---------| +| `--log-level debug` | Verbose logging | +| `--log-format json` | Structured logs for parsing | +| `--insecure-skip-verify` | Skip TLS verification (testing only) | +| `--base-url` | Override API endpoint | + +### Diagnostic tools + +| Tool | Command | Purpose | +|------|---------|---------| +| baton | `baton resources -f sync.c1z` | Inspect sync output | +| baton | `baton stats -f sync.c1z` | Summary statistics | +| curl | `curl -v $API_URL` | Test API connectivity | +| jq | `cat sync.json \| jq '.'` | Parse JSON output | + + + + Error code reference + + + Full SDK API reference + + diff --git a/developer/error-codes.mdx b/developer/error-codes.mdx new file mode 100644 index 0000000..3389fc8 --- /dev/null +++ b/developer/error-codes.mdx @@ -0,0 +1,294 @@ +--- +title: "Error codes reference" +sidebarTitle: "Error codes" +description: "Identify and resolve errors from connector output. Each error includes the cause and how to fix it." +--- + +This reference documents common errors from the baton-sdk and connector runtime. + +## Configuration errors + +### Connector does not have account manager configured + +**Cause:** Attempted to create an account but the connector doesn't implement `AccountManager`. + +**Solution:** Either: +- Implement the `AccountManager` interface for account creation +- Don't call account creation operations on this connector + +```go +// To support account creation, implement: +type AccountManager interface { + CreateAccount(ctx context.Context, accountInfo *v2.AccountInfo, + credentialOptions *v2.CredentialOptions) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) +} +``` + +### Resource type does not have credential manager configured + +**Cause:** Attempted credential rotation but the resource type doesn't support it. + +**Solution:** Implement `CredentialManager` for that resource type, or don't attempt rotation. + +### Error: old account manager interface implemented + +**Cause:** Using deprecated v1 interface instead of v2. + +**Solution:** Update to implement `AccountManagerV2`: + +```go +// Old (deprecated): +type AccountManager interface { ... } + +// New (correct): +type AccountManagerV2 interface { + CreateAccount(ctx context.Context, accountInfo *v2.AccountInfo, + credentialOptions *v2.LocalCredentialOptions) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) +} +``` + +### Error: duplicate resource type found for account manager + +**Cause:** Registered the same resource type twice with the builder. + +**Solution:** Only register each resource type once: + +```go +// BAD +b.WithAccountManager(userBuilder) +b.WithAccountManager(userBuilder) // Duplicate! + +// GOOD +b.WithAccountManager(userBuilder) +b.WithAccountManager(groupBuilder) +``` + +## Provisioning errors + +### Error: provisioner not found for resource type + +**Cause:** Grant/Revoke called on a resource type that doesn't implement provisioning. + +**Solution:** Implement `ResourceProvisionerV2`: + +```go +type ResourceProvisionerV2 interface { + Grant(ctx context.Context, principal *v2.Resource, entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} +``` + +### Error: preferred credential creation option is not set + +**Cause:** Account creation request missing required credential option. + +**Solution:** Include credential options in the request, or check connector capabilities first. + +### Error: preferred credential rotation option is not part of the supported options + +**Cause:** Requested credential type not supported by this connector. + +**Solution:** Check `GetCapabilities()` for supported credential options before requesting rotation. + +## Pagination errors + +### Next page token is the same as the current page token + +**Cause:** Your List/Entitlements/Grants method returned the same token it received, creating an infinite loop. + +**Solution:** + +```go +// BAD - Returns input token +return resources, pToken.Token, nil, nil + +// GOOD - Returns next token from API or empty string +return resources, nextPageToken, nil, nil +``` + +See [Troubleshooting - Pagination Loop](/developer/debugging#pagination-loop-detected). + +## C1z file errors + +### C1z: invalid file + +**Cause:** The .c1z file is corrupted or not a valid c1z format. + +**Solution:** +- Re-run the sync to generate a fresh file +- Check disk space and write permissions +- Verify the file wasn't truncated + +### C1z: max decoded size exceeded + +**Cause:** The decompressed c1z file exceeds the default size limit. + +**Solution:** Set environment variable to increase limit: + +```bash +export C1Z_DECODER_MAX_DECODED_SIZE=10737418240 # 10GB +``` + +### C1z: window size exceeded + +**Cause:** Decompression requires more memory than allowed. + +**Solution:** Increase decoder memory: + +```bash +export C1Z_DECODER_MAX_MEMORY=4294967296 # 4GB +``` + +### C1file: output file path is empty + +**Cause:** Connector invoked without specifying output path. + +**Solution:** + +```bash +./baton-yourservice --file output.c1z +# or +./baton-yourservice -f output.c1z +``` + +### C1file: sync is not active + +**Cause:** Attempted to write data outside of an active sync operation. + +**Solution:** This is usually an SDK internal error. Ensure you're not calling SDK methods outside the sync lifecycle. + +## Action errors + +### Error: action manager not implemented + +**Cause:** Connector doesn't support custom actions. + +**Solution:** Implement `CustomActionManager` if you need actions, or don't call action endpoints. + +### Action %s not found + +**Cause:** Requested an action that doesn't exist. + +**Solution:** List available actions first: + +```bash +./baton-yourservice actions list +``` + +### Action handler timed out + +**Cause:** Custom action took too long to complete. + +**Solution:** Optimize the action implementation or increase timeout (if configurable). + +## Validation errors + +### Error: validate provider not implemented + +**Cause:** Connector doesn't implement configuration validation. + +**Solution:** Implement `ValidateProvider` for config validation: + +```go +type ValidateProvider interface { + Validate(ctx context.Context) (annotations.Annotations, error) +} +``` + +### Validate failed: %w + +**Cause:** Connector validation failed (wrapped error contains details). + +**Solution:** Check the inner error message for specifics (usually credential or connectivity issues). + +## Input errors + +### Input cannot be nil + +**Cause:** Passed nil to a builder function. + +**Solution:** Ensure all required parameters are provided. + +### Input is not a connectorserver, connectorbuilder, or connectorbuilderv2 + +**Cause:** Passed wrong type to connector initialization. + +**Solution:** Use the correct builder type: + +```go +builder, err := connectorbuilder.NewConnector(ctx, yourConnector) +``` + +## S3/storage errors + +### Attempting to save to s3 without a valid client + +**Cause:** S3 upload configured but client not initialized. + +**Solution:** Ensure AWS credentials and S3 configuration are provided. + +### Attempting to save to s3 without a valid file path specified + +**Cause:** S3 destination path not configured. + +**Solution:** Set the S3 bucket and key path in configuration. + +### Accesskeyid and secretaccesskey must be specified + +**Cause:** AWS credentials incomplete. + +**Solution:** + +```bash +export AWS_ACCESS_KEY_ID=your-key +export AWS_SECRET_ACCESS_KEY=your-secret +``` + +## Encryption errors + +### Age: failed to encrypt: %w + +**Cause:** Encryption failed (age library error). + +**Solution:** Check the wrapped error for details. Common causes: +- Invalid recipient public key +- Memory allocation failure + +### Age: failed to decrypt: %w + +**Cause:** Decryption failed. + +**Solution:** Verify you're using the correct private key that matches the encryption. + +## Quick reference + +### Environment variables + +| Variable | Purpose | +|----------|---------| +| `C1Z_DECODER_MAX_DECODED_SIZE` | Max decompressed c1z size | +| `C1Z_DECODER_MAX_MEMORY` | Max memory for decompression | +| `AWS_ACCESS_KEY_ID` | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | + +### Diagnostic commands + +```bash +# Enable debug logging to see full error context +./baton-yourservice --log-level debug + +# Check connector capabilities +./baton-yourservice capabilities + +# Validate configuration +./baton-yourservice --validate-only +``` + + + + Debugging workflow and common issues + + + Full SDK API reference + + diff --git a/developer/glossary.mdx b/developer/glossary.mdx new file mode 100644 index 0000000..3981ca3 --- /dev/null +++ b/developer/glossary.mdx @@ -0,0 +1,145 @@ +--- +title: "Glossary" +sidebarTitle: "Glossary" +description: "Quick reference for terms used throughout the connector documentation." +--- + +When in doubt about terminology, check here first. + +## Core terms + +| Term | Definition | +|------|------------| +| **Baton** | The connector framework: Go SDK + individual connectors | +| **baton-sdk** | Go library that handles sync orchestration, pagination, and connector runtime | +| **Connector** | A Go binary that syncs access control data from a third-party service into ConductorOne | +| **c1z** | Compressed sync output file format (gzip SQLite) | +| **c1in** | C1 Integration Network - the overall connector ecosystem | +| **cone** | ConductorOne CLI for access management | +| **conductorone-sdk-go** | Go SDK for ConductorOne API integration | +| **Connector Hub** | User-facing name for the connector marketplace | +| **Meta-connector** | Configuration-driven connector that maps external systems via YAML instead of Go code | +| **baton-http** | Meta-connector for REST APIs using YAML configuration and CEL expressions | +| **baton-sql** | Meta-connector for SQL databases using YAML configuration and CEL expressions | + +## Access control model + +| Term | Definition | +|------|------------| +| **Resource** | An entity in the target system: User, Group, Role, App, or custom type | +| **Resource Type** | Classification of resources (e.g., "user", "group", "role") with associated traits | +| **Entitlement** | A permission that can be granted (e.g., "Admin on Database X") | +| **Entitlement Slug** | Stable identifier for an entitlement (e.g., "member", "admin", "read") | +| **Grant** | An assignment of an entitlement to a principal (e.g., "Alice has Admin on Database X") | +| **Principal** | An entity that receives grants (typically users or service accounts) | +| **Trait** | Resource type classification: TRAIT_USER, TRAIT_GROUP, TRAIT_ROLE, TRAIT_APP, TRAIT_SECRET | +| **Parent Resource** | Resource that contains child resources in a hierarchy (e.g., Organization containing Repositories) | +| **Child Resource** | Resource that exists within a parent resource context | +| **ChildResourceType Annotation** | Marker on parent declaring what child types it contains | + +## SDK concepts + +| Term | Definition | +|------|------------| +| **ResourceSyncer** | Interface that defines how to sync a resource type (ResourceType, List, Entitlements, Grants methods) | +| **Sync lifecycle** | The ordered stages: ResourceType -> List -> Entitlements -> Grants | +| **Sync stage** | One stage of the sync lifecycle (Stage 1-4: ResourceType, List, Entitlements, Grants) | +| **Inversion of control** | Pattern where SDK orchestrates when your code is called; you define builders, SDK calls them | +| **Resource Builder** | Implementation of ResourceSyncer for a specific resource type | +| **pagination.Token** | SDK type for managing page cursors across List/Entitlements/Grants calls | +| **pagination.Bag** | SDK type for managing nested pagination state (stack-based) | +| **PageState** | Single pagination state within a Bag (contains Token, ResourceTypeID, ResourceID) | +| **uhttp** | SDK package providing HTTP client with automatic retries and rate limiting | +| **Annotations** | Metadata attached to resources, entitlements, or grants (e.g., ChildResourceType, RawId) | +| **RawId** | Annotation carrying the external system's identifier; used for ID correlation during uplift | + +## Execution modes + +| Term | Definition | +|------|------------| +| **One-shot mode** | CLI mode: runs once, produces c1z file, exits (no --client-id) | +| **Daemon mode** | Long-running mode that polls ConductorOne for tasks (requires --client-id and --client-secret) | +| **Hosted mode** | Connector run by ConductorOne infrastructure on behalf of customers | +| **Service mode** | Synonym for daemon mode | +| **Client credentials** | OAuth2 client ID and secret for authenticating connector to ConductorOne | +| **Task polling** | Daemon mode behavior of periodically checking ConductorOne for work | + +## Provisioning operations + +| Term | Definition | +|------|------------| +| **Grant** (operation) | Operation to add an entitlement to a principal | +| **Revoke** | Operation to remove an entitlement from a principal | +| **CreateAccount** | JIT provisioning - create user account in target system | +| **DeleteResource** | Remove a resource from the target system | +| **ResourceProvisionerV2** | SDK interface for Grant/Revoke operations (recommended) | +| **AccountManager** | SDK interface for CreateAccount operations | +| **ResourceDeleterV2** | SDK interface for Delete operations | +| **Capability Manifest** | JSON file (baton_capabilities.json) declaring supported operations | + +## Integration concepts + +| Term | Definition | +|------|------------| +| **Sync** | Reading access data from a system into ConductorOne; produces .c1z file | +| **Uplift** | ConductorOne process that transforms raw connector records into domain objects (Apps, Resources, Grants) | +| **Provision** | Writing access changes back to a system (grant, revoke, create, delete) | +| **Reconciliation** | Comparing actual access (from sync) to desired access (from policy) and correcting drift | +| **external_id** | The identifier from an external system, stored with connector records during sync | +| **source_connector_ids** | Map on domain objects tracking which connector provided which external ID | +| **ID Correlation** | Matching connector output to existing ConductorOne objects using RawId and external_id | +| **JIT Provisioning** | Just-In-Time provisioning - creating user account when first needed, not before | +| **IdP** | Identity Provider - authoritative source of user identities (Okta, Azure AD, Google Workspace) | +| **Source of Truth** | The system that authoritatively defines an entity (IdPs are typically source of truth for users) | +| **Access Review** | Process of verifying that existing access grants are still appropriate | +| **Drift** | Difference between actual access state and desired access state | + +## Pagination + +| Term | Definition | +|------|------------| +| **Cursor-based pagination** | Pagination using opaque tokens returned by the API | +| **Offset-based pagination** | Pagination using numeric offset and limit parameters | +| **Page token** | String passed between calls to continue pagination | +| **LDAP paging** | Server-side pagination for LDAP using cookies | +| **Nested pagination** | Paginating children within each paginated parent (managed with Bag) | + +## Authentication + +| Term | Definition | +|------|------------| +| **API Key** | Simple token-based authentication passed in headers | +| **Bearer Token** | Token passed in Authorization header as "Bearer {token}" | +| **OAuth2 Client Credentials** | Flow exchanging client ID/secret for access token | +| **JWT Service Account** | Authentication using signed JSON Web Tokens (common for Google APIs) | +| **LDAP Bind** | Authentication to LDAP server using credentials | +| **Domain-wide Delegation** | Google pattern allowing service account to impersonate domain users | + +## Publishing + +| Term | Definition | +|------|------------| +| **Release Manifest** | Metadata describing a connector version (org, name, version, assets) | +| **Version State** | Lifecycle state: PENDING, UPLOADING, VALIDATING, PUBLISHED, YANKED, FAILED | +| **Asset** | Platform-specific binary (e.g., darwin-amd64, linux-arm64) | +| **Signing Key** | GPG or Cosign key used to sign connector releases | +| **Yank** | Withdraw a published version (remains visible but marked deprecated) | + +## Configuration + +| Term | Definition | +|------|------------| +| **CEL** | Common Expression Language - used for data transformation in meta-connectors | +| **Environment variable** | Configuration via BATON_* prefixed env vars | +| **Base URL** | Configurable API endpoint for testing against mocks | +| **Insecure flag** | Option to skip TLS verification for local testing | + +## Development + +| Term | Definition | +|------|------------| +| **golangci-lint** | Standard Go linter used for connector code quality | +| **Makefile targets** | Standard: build, lint, test, update-deps | +| **sync.Map** | Go's thread-safe map type used for connector caching | +| **Hot reload** | Automatic rebuild on code changes during development | +| **Mock server** | Local server mimicking target API for testing | diff --git a/developer/http-authoring.mdx b/developer/http-authoring.mdx new file mode 100644 index 0000000..2222858 --- /dev/null +++ b/developer/http-authoring.mdx @@ -0,0 +1,247 @@ +--- +title: "baton-http: REST API integration" +sidebarTitle: "HTTP authoring" +description: "Integrate any REST API without writing Go code. Configuration only." +--- + +Meta-connectors let you write YAML instead of Go code. Instead of implementing the `ResourceSyncer` interface, you describe how to map an API to ConductorOne's resource model. Your ops team can own integrations directly - no engineering queue. + +## When to use baton-http + +**Use baton-http when:** +- The target system has a REST API +- You need a quick integration without writing Go +- The access model maps to user/group/resource patterns +- You want non-developers to maintain the integration + +**Use a custom connector when:** +- The API requires complex authentication (OAuth2 flows, Kerberos) +- You need heavy data transformation or business logic +- You need maximum performance optimization + +## Connection configuration + +```yaml +version: "1" +app_name: "My Integration" +app_description: "Syncs users and groups from SaaS API" + +connect: + base_url: "https://api.example.com/v1" + auth: + type: "bearer" + token: "${API_KEY}" # Environment variable + request_defaults: + headers: + Accept: "application/json" + query_params: + limit: "100" +``` + +**Supported authentication types:** +- `bearer` - Bearer token in Authorization header +- `api_key` - API key in custom header +- `basic` - Basic authentication +- `oauth2_client_credentials` - OAuth2 client credentials flow + +## Listing resources + +```yaml +resource_types: + user: + name: "User" + list: + request: + url: "/users" + method: "GET" + response: + items_path: "data.users" # JSONPath to array in response + pagination: + strategy: "offset" + limit_param: "limit" + offset_param: "offset" + page_size: 100 + map: + id: ".id" + display_name: ".name" + traits: + user: + emails: + - ".email" + status: ".status" +``` + +## URL templates + +Use Go template syntax for dynamic URLs: + +```yaml +grants: + - request: + url: "tmpl:/groups/{{.resource.id}}/members" +``` + +Available template variables: +- `.resource` - Current resource being processed +- `.principal` - User/entity for provisioning +- `.item` - Current item in iteration + +## Pagination + +### Offset-based + +```yaml +pagination: + strategy: "offset" + limit_param: "limit" + offset_param: "offset" + page_size: 100 + total_path: "meta.total" # Optional: path to total count +``` + +### Cursor-based + +```yaml +pagination: + strategy: "cursor" + primary_key: "id" +``` + +## Entitlements and grants + +### Static entitlements + +Define fixed entitlements that don't require discovery: + +```yaml +resource_types: + group: + static_entitlements: + - id: "member" + display_name: "'Member'" + purpose: "assignment" + grantable_to: + - "user" +``` + +### Grants discovery + +```yaml +grants: + - request: + url: "tmpl:/roles/{{.resource.id}}/members" + method: "GET" + response: + items_path: "members" + map: + - principal_id: ".user_id" + principal_type: "user" + entitlement_id: "assigned" +``` + +## Complete example + +```yaml +version: "1" +app_name: "SaaS Application" +app_description: "Syncs users from SaaS API" + +connect: + base_url: "https://api.example.com/v2" + auth: + type: "bearer" + token: "${API_KEY}" + request_defaults: + headers: + Accept: "application/json" + +resource_types: + user: + name: "User" + + list: + request: + url: "/users" + method: "GET" + response: + items_path: "data" + pagination: + strategy: "offset" + limit_param: "per_page" + offset_param: "page" + page_size: 100 + map: + id: ".id" + display_name: ".attributes.name" + traits: + user: + emails: + - ".attributes.email" + status: ".attributes.active ? 'enabled' : 'disabled'" + + role: + name: "Role" + + list: + request: + url: "/roles" + method: "GET" + response: + items_path: "roles" + map: + id: ".id" + display_name: ".name" + + static_entitlements: + - id: "assigned" + display_name: "'Assigned'" + purpose: "assignment" + grantable_to: + - "user" + + grants: + - request: + url: "tmpl:/roles/{{.resource.id}}/members" + method: "GET" + response: + items_path: "members" + map: + - principal_id: ".user_id" + principal_type: "user" + entitlement_id: "assigned" +``` + +## Running baton-http + +### Validate configuration + +```bash +baton-http --config-path ./config.yaml --validate-config-only +``` + +### One-shot mode (local testing) + +```bash +baton-http --config-path ./config.yaml -f sync.c1z +baton resources -f sync.c1z +baton grants -f sync.c1z +``` + +### Service mode (production) + +```bash +baton-http --config-path ./config.yaml \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" +``` + +## Related + + + + baton-sql for database integration + + + Data transformation with CEL + + diff --git a/developer/intro.mdx b/developer/intro.mdx index ca3d108..4f87bf7 100644 --- a/developer/intro.mdx +++ b/developer/intro.mdx @@ -1,36 +1,261 @@ --- -title: "Welcome to the developer docs" -sidebarTitle: "Welcome" -description: "Welcome to the ConductorOne developer docs, where you'll learn how to use our API and SDKs to integrate with, manage, and extend the ConductorOne platform." -og:title: Welcome to the developer docs - ConductorOne -og:description: Welcome to the ConductorOne developer docs, where you'll learn how to use our API and SDKs to integrate with, manage, and extend the ConductorOne platform. +title: "Getting started with Baton connectors" +sidebarTitle: "Getting started" +description: "Understand what connectors are, when you need one, and what working looks like end-to-end." --- -Welcome to ConductorOne’s developer documentation! Here you’ll find instructions for working with ConductorOne’s SDKs, our Terraform and Postman providers, and more. Use the links below to jump to the tools and tutorials that are right for you. +What happens when an employee leaves your company but still has access to your production database? Or your billing system? Or your customer data? -## Software development kits (SDKs) +A **connector** answers the question: *who has access to what?* -ConductorOne offers SDKs for working with our API in Go and Typescript. SDKs for additional languages are in the works! +ConductorOne needs to know about users, groups, roles, and permissions across all your systems. But every system stores this information differently. Okta has users and groups. AWS has IAM roles and policies. Salesforce has profiles and permission sets. The Baton connector framework solves this: you write one integration, and ConductorOne handles the rest. - - - - +A connector bridges this gap. It translates access data from any system into a common format that ConductorOne understands. Once connected, you get unified visibility across your entire infrastructure. -## ConductorOne API in Postman +In Baton terms, a connector is a program that can: +- **List resources** (users, groups, roles, apps, projects, etc.) +- **Define entitlements** (the permissions you can grant on those resources) +- **Emit grants** (the facts of who currently has which entitlements) -Work with the ConductorOne API in Postman, where you can test and save API requests with ease. +## Where connectors fit - - - +``` +Your Systems ConductorOne ++------------------+ +------------------+ +| Okta (IdP) | <------> | | +| AWS | <------> | Unified Access | +| Salesforce | Connector | Management | +| GitHub | <------> | | +| Your Custom App | | | ++------------------+ +------------------+ + ^ | + | Reconciliation Loop | + +----------------------------------+ +``` -## ConductorOne Terraform provider +Most people interact with connectors in two ways: -Manage and configure ConductorOne using Terraform to practice “access management as code”. +- **Deploy an existing connector** (you configure it; you do not write Go) +- **Build or extend a connector** (you write Go against `baton-sdk`) - - - +### Sync and provision +Connectors do two things: +**Sync** (read): Pull access data from your systems into ConductorOne +- Who exists? What groups? What roles? +- What permissions are available? +- Who has what access right now? + +**Provision** (write): Push access changes back to your systems +- **Grant**: Give someone access they've been approved for +- **Revoke**: Remove access that's been terminated +- **Create Account**: JIT (Just-In-Time) provisioning +- **Delete Resource**: Remove accounts entirely + +Together, sync and provision create a **reconciliation loop**: ConductorOne sees what access exists (sync), compares it to what access *should* exist (policy), and corrects any drift (provision). Your access controls become self-healing. + +### The special role of identity providers + +Identity Providers (IdPs) like Okta, Azure AD, or Google Workspace have a unique position: they're often the **source of truth** for who your users are. + +Most connectors sync users *from* the target system. But IdP connectors do more: +- They define the canonical user identities +- Other systems' users are correlated back to IdP users +- User lifecycle (join, move, leave) often originates in the IdP + +When you connect an IdP, you're establishing the identity foundation that other connectors build upon. + +## Understanding the tools + +Before diving into implementation, it helps to understand which tools do what. The ConductorOne ecosystem has several SDKs and CLIs with different purposes. + +### For connector developers + +| Tool | Purpose | When to use | +|------|---------|-------------| +| **baton-sdk** | Go SDK for building connectors | Building or extending a connector | +| **baton** | CLI for inspecting sync output | Debugging `.c1z` files locally | + +**baton-sdk** is the Go library you import when building a connector. It provides: +- The `ResourceSyncer` interface you implement +- Pagination helpers for API calls +- Resource and grant builders +- HTTP client utilities with retry logic + +```go +import "github.com/conductorone/baton-sdk/pkg/connectorbuilder" +``` + +**baton** is a standalone CLI for inspecting `.c1z` files. After your connector runs, use it to verify the output: + +```bash +baton resources -f sync.c1z # List synced resources +baton grants -f sync.c1z # List grants +baton entitlements -f sync.c1z # List entitlements +``` + +### For ConductorOne users (not connector development) + +| Tool | Purpose | When to use | +|------|---------|-------------| +| **cone** | CLI for ConductorOne platform | Access requests, approvals, searches | +| **conductorone-sdk-go** | Go SDK for ConductorOne API | Integrating with C1 platform | + +**cone** is the ConductorOne CLI for end-users and administrators. It handles access management workflows: + +```bash +cone login # Authenticate to ConductorOne +cone search users "alice" # Search for users +cone task approve # Approve access requests +``` + +### Which tool do I need? + +| I want to... | Use this | +|--------------|----------| +| Build a new connector | `baton-sdk` | +| Debug my connector's output | `baton` CLI | +| Request or approve access | `cone` CLI | +| Build an app that uses ConductorOne | `conductorone-sdk-go` | +| Deploy a pre-built connector | The connector binary | + + +`cone` and `baton` are separate tools for separate purposes. You don't use `cone` to build connectors, and you don't use `baton` to manage access requests. + + +### The connector binary + +When you build a connector, you produce a standalone binary (e.g., `baton-okta`, `baton-github`). This binary: + +- **Embeds the SDK** - It's compiled with `baton-sdk`, not dependent on it at runtime +- **Is self-contained** - No runtime dependencies except the target system's API +- **Runs independently** - You don't need any other ConductorOne tools installed +- **Produces standard output** - A `.c1z` file that any ConductorOne environment can consume + +```bash +# The connector IS the binary +./baton-okta --domain example.okta.com --api-token $TOKEN + +# It produces a .c1z file +ls -la sync.c1z +``` + +The "connector" is NOT the SDK. It's the compiled program that uses the SDK. + +## The minimum contract + +The beauty of the Baton SDK is how little you need to implement. At the SDK level, the primary interface for connector developers is `ResourceSyncer`: + +- `ResourceType(ctx)`: define the type (and traits) of a resource (for example "user" with trait `TRAIT_USER`) +- `List(ctx, parentResourceID, token)`: list instances of that resource type (paged) +- `Entitlements(ctx, resource, token)`: list entitlements offered by that resource (paged) +- `Grants(ctx, resource, token)`: list who has those entitlements (paged) + +Four methods give you a working connector. The SDK handles pagination orchestration, error handling, and output formatting. You focus on translating your target system's API into the common model. Optional extensions add provisioning and lifecycle operations when you need them. + +## When to build vs reuse + +- **Use an existing connector** when it exists and your needs are met. The operational cost is configuration and deployment. +- **Contribute upstream** when the connector exists but is missing a capability you need (for example, adding a resource type or provisioning path). This reduces long-term fork burden and helps the community. +- **Build a new connector** when the target system is unsupported or proprietary. With the SDK handling the heavy lifting, most connectors can be built in a few days. + +One reality to keep in mind: connector capabilities vary across the ecosystem. If you need provisioning, check for it per connector/version. + +**Consider alternatives when:** +- A pre-built connector exists and meets your needs +- You need quick integration - use [baton-http](/developer/http-authoring) for REST APIs +- You need database integration - use [baton-sql](/developer/sql-authoring) for SQL databases + +## What "working" looks like + +`baton-demo` is an example connector with hardcoded data that demonstrates a fully runnable sync. No API credentials needed - you can try it right now. + +```bash +baton-demo +baton resources +``` + +Or via docker: + +```bash +docker run --rm -v $(pwd):/out ghcr.io/conductorone/baton-demo:latest -f "/out/sync.c1z" +docker run --rm -v $(pwd):/out ghcr.io/conductorone/baton:latest -f "/out/sync.c1z" resources +``` + +Example output: + +```bash +$ baton resources -f sync.c1z +Resource Type Count +user 127 +team 23 +repository 89 + +$ baton grants -f sync.c1z | head -5 +Principal Entitlement Resource +alice@corp.com member engineering-team +alice@corp.com admin api-repo +bob@corp.com member platform-team +bob@corp.com read docs-repo +``` + +## What "working" does not guarantee + +- **"Runs locally" does not mean "safe in production."** You still need to handle pagination, retries, and rate limits (and prove you do). +- **Provisioning is not implied by syncing.** Many connectors are read-only or partially provisionable; you must check per connector. +- **Docs-site capability tables are not the API contract.** They are a directory for humans; use the connector's own manifests and the SDK interfaces as ground truth. + + +Connectors sync access data, not business data. Users, roles, permissions, API keys - yes. Customer records, issues, messages, logs - no. + + +## Next steps + + + + Resources, entitlements, grants, traits, and the sync lifecycle + + + Project structure and implementation patterns + + + Configure baton-http or baton-sql without writing Go + + + Getting help, contributing, support channels + + + +## Prerequisites for building + +If you're going to build a connector, you'll need: + +| Tool | Version | Purpose | +|------|---------|---------| +| Go | See go.mod | Connector runtime | +| Git | Any | Version control | +| make | Any | Build automation | + + +Go version requirements vary by connector and SDK version. Check the `go.mod` file in baton-sdk or your target connector for the current minimum version. + + +Plus: +- Access to the system you want to connect to +- API credentials with read permissions +- An IDE with Go support (VS Code, GoLand) + +## Quick reference + +| Term | Meaning | +|------|---------| +| Connector | Go binary that syncs and provisions access data | +| c1z | Compressed sync output file | +| Resource | User, group, role, or custom entity | +| Entitlement | Permission that can be granted | +| Grant | Assignment of entitlement to principal | +| Baton SDK | Go library that handles sync orchestration | +| Sync | Reading access data from a system | +| Provision | Writing access changes back (grant, revoke, create, delete) | +| Reconciliation | Comparing actual vs desired access and correcting drift | diff --git a/developer/pagination.mdx b/developer/pagination.mdx new file mode 100644 index 0000000..83f919c --- /dev/null +++ b/developer/pagination.mdx @@ -0,0 +1,379 @@ +--- +title: "Pagination patterns" +sidebarTitle: "Pagination" +description: "Handle cursor-based, offset-based, LDAP, and nested pagination in your connector." +--- + +Always paginate. Even if your test environment has 10 users, production might have 10,000. The SDK handles pagination orchestration - use it from day one. + +## Basic pattern + +Every `List()`, `Entitlements()`, and `Grants()` method returns a next page token: + +```go +// Return next page token, or "" when done +return resources, nextPageToken, nil, nil +``` + +The SDK calls your method repeatedly until you return an empty token. Your job: fetch and convert one page at a time. + +## Pagination strategies + +APIs paginate results differently. Your connector adapts the token to whatever the API expects: + +| Strategy | How it works | Example APIs | +|----------|--------------|--------------| +| **Cursor-based** | Opaque token in response | Most modern APIs (Slack, Stripe) | +| **Offset-based** | `?offset=100&limit=50` | Traditional REST APIs | +| **Page number** | `?page=3&per_page=50` | Some REST APIs | +| **Link header** | RFC 5988 Link relations | GitHub API | +| **LDAP paging** | Server-side cookie | Active Directory | +| **GraphQL cursors** | `after: "cursor"` | GitHub GraphQL, Shopify | + +### Cursor-based (most modern APIs) + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + resp, err := u.client.ListUsers(ctx, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range resp.Users { + r, _ := resource.NewUserResource(user.Name, userType, user.ID) + resources = append(resources, r) + } + + // API returns opaque cursor for next page + return resources, resp.NextCursor, nil, nil +} +``` + +### Offset-based (traditional REST) + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + const limit = 100 + offset := 0 + if pToken.Token != "" { + offset, _ = strconv.Atoi(pToken.Token) + } + + resp, err := u.client.ListUsers(ctx, offset, limit) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range resp.Users { + r, _ := resource.NewUserResource(user.Name, userType, user.ID) + resources = append(resources, r) + } + + // Calculate next offset + nextToken := "" + if len(resp.Users) == limit { + nextToken = strconv.Itoa(offset + limit) + } + + return resources, nextToken, nil, nil +} +``` + +### LDAP paging (Active Directory) + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + const pageSize = 1000 + pagingControl := ldap.NewControlPaging(pageSize) + + // Restore cookie from previous page + if pToken.Token != "" { + cookie, _ := base64.StdEncoding.DecodeString(pToken.Token) + pagingControl.SetCookie(cookie) + } + + searchRequest := ldap.NewSearchRequest( + "dc=example,dc=com", + ldap.ScopeWholeSubtree, ldap.NeverDerefAliases, 0, 0, false, + "(objectClass=user)", + []string{"distinguishedName", "sAMAccountName", "mail"}, + []ldap.Control{pagingControl}, + ) + + result, err := u.conn.Search(searchRequest) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, entry := range result.Entries { + r, _ := resource.NewUserResource( + entry.GetAttributeValue("sAMAccountName"), + userType, + entry.GetAttributeValue("distinguishedName")) + resources = append(resources, r) + } + + // Extract cookie for next page + nextToken := "" + if ctrl := ldap.FindControl(result.Controls, ldap.ControlTypePaging); ctrl != nil { + if pc, ok := ctrl.(*ldap.ControlPaging); ok && len(pc.Cookie) > 0 { + nextToken = base64.StdEncoding.EncodeToString(pc.Cookie) + } + } + + return resources, nextToken, nil, nil +} +``` + +## Nested pagination with Bag + +When you have hierarchies, you often need to paginate at multiple levels: "page through orgs, and for each org, page through repos." The SDK's `pagination.Bag` handles this. + +### How Bag works + +The `Bag` acts as a stack for managing complex pagination state: + +```go +type Bag struct { + states []PageState + currentState *PageState +} + +type PageState struct { + Token string `json:"token,omitempty"` + ResourceTypeID string `json:"type,omitempty"` + ResourceID string `json:"id,omitempty"` +} +``` + +Key methods: +- `Push(state PageState)` - Push new pagination state onto stack +- `Pop() *PageState` - Pop current state, make top of stack current +- `Next(pageToken string)` - Update current state with new page token +- `Current() *PageState` - Get current state (may be nil) +- `Marshal() (string, error)` - Serialize state to opaque token +- `Unmarshal(token string) error` - Restore state from token + +### Example: paginating repos within orgs + +```go +func (r *repoBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + bag := &pagination.Bag{} + if pToken != nil && pToken.Token != "" { + // Restore state from previous call + if err := bag.Unmarshal(pToken.Token); err != nil { + return nil, "", nil, err + } + } + + // Get current pagination state + // IMPORTANT: bag.Current() returns nil on first call before any Push() + state := bag.Current() + pageToken := "" + if state != nil { + pageToken = state.Token + } + + // Fetch page of repos for this org + repos, nextPage, err := r.client.ListRepos(ctx, parentID.Resource, pageToken) + if err != nil { + return nil, "", nil, err + } + + // Build resources + var resources []*v2.Resource + for _, repo := range repos { + res, _ := resource.NewResource(repo.Name, repoType, repo.ID, + resource.WithParentResourceID(parentID)) + resources = append(resources, res) + } + + // Update state with next page token + if nextPage != "" { + bag.Next(nextPage) + } + + // Marshal state for next call + nextToken, err := bag.Marshal() + if err != nil { + return nil, "", nil, err + } + + return resources, nextToken, nil, nil +} +``` + + +**Bag.Current() nil safety:** The `Bag` starts empty. `Current()` returns nil until you `Push()` a state. Always check for nil before accessing fields. + + +### Multi-level nested pagination + +For deeper hierarchies (org -> team -> members), push and pop states: + +```go +func (m *memberBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + bag := &pagination.Bag{} + if pToken.Token != "" { + bag.Unmarshal(pToken.Token) + } + + // First call: initialize state + if bag.Current() == nil { + bag.Push(pagination.PageState{ResourceID: resource.Id.Resource}) + } + + state := bag.Current() + members, nextPage, err := m.client.ListTeamMembers(ctx, state.ResourceID, state.Token) + if err != nil { + return nil, "", nil, err + } + + var grants []*v2.Grant + for _, member := range members { + g := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "user", Resource: member.ID}) + grants = append(grants, g) + } + + if nextPage != "" { + bag.Next(nextPage) + } else { + bag.Pop() // Done with this resource, pop state + } + + nextToken, _ := bag.Marshal() + return grants, nextToken, nil, nil +} +``` + +## Modeling hierarchies + +Real systems often have deep hierarchies. The Baton model handles these naturally: declare parent-child relationships and the SDK walks the tree for you. + +### Two mechanisms connect parent and child + +**1. Parent declares child types** via `ChildResourceType` annotation: + +```go +orgResource, _ := resource.NewResource("Acme Corp", orgType, "org-123", + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "repository"}), + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "team"})) +``` + +**2. Child references parent** via `ParentResourceID`: + +```go +repoResource, _ := resource.NewResource("api-service", repoType, "repo-456", + resource.WithParentResourceID(&v2.ResourceId{ + ResourceType: "organization", + Resource: "org-123", + })) +``` + +When the SDK calls `List()` for child resources, it passes the parent's `ResourceId` so you can scope your API calls. + +### Example: GitHub hierarchy + +``` +Organization + | + +-- Team (membership entitlements) + | | + | +-- Team Members (grants) + | + +-- Repository + | + +-- Repository Permissions (admin, write, read) + | + +-- Collaborators (grants) +``` + +**Organization builder:** + +```go +func (o *orgBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + orgs, next, err := o.client.ListOrganizations(ctx, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, org := range orgs { + r, _ := resource.NewResource(org.Name, orgType, org.ID, + // Declare child types + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "team"}), + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "repository"})) + resources = append(resources, r) + } + return resources, next, nil, nil +} +``` + +**Repository builder - receives parent org in List():** + +```go +func (r *repoBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + // parentID.Resource contains the org ID + repos, next, err := r.client.ListOrgRepos(ctx, parentID.Resource, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, repo := range repos { + res, _ := resource.NewResource(repo.Name, repoType, repo.ID, + resource.WithParentResourceID(parentID)) // Link to parent org + resources = append(resources, res) + } + return resources, next, nil, nil +} +``` + +### When to use hierarchies vs flat models + +| Use hierarchies when... | Use flat models when... | +|------------------------|------------------------| +| Resources only exist within a parent (repos in orgs) | Resources are independent (standalone users) | +| API requires parent context to list children | API lists all resources globally | +| Access is scoped by parent (org-level vs repo-level) | Access is global | +| You need to express "admin of org X" vs "admin of repo Y" | All instances have the same entitlement shape | + +## Common hierarchy mistakes + +| Mistake | Consequence | Fix | +|---------|-------------|-----| +| Missing `ChildResourceType` annotation | SDK doesn't know to call List() for children | Add annotation to parent | +| Wrong `ParentResourceID` type | Child appears orphaned | Match exact type ID string | +| Not using `parentID` in List() | Returns all resources, not just children | Scope API call to parent | +| Flat model for hierarchical data | Loses context (which org owns this repo?) | Add parent-child links | +| Over-deep hierarchies | Sync takes too long | Flatten where parent context isn't needed | + +## Pagination invariants + +The SDK enforces invariants to catch common bugs: + +- **Token must progress**: Returning the same token you received causes an error. This prevents infinite loops. +- **Empty token means done**: Return `""` when there are no more pages. +- **Consistent page sizes**: While not enforced, use consistent page sizes for predictable behavior. + + +Test with small page sizes (10-20 items) during development to verify pagination works correctly before testing with production-sized datasets. + diff --git a/developer/provisioning.mdx b/developer/provisioning.mdx new file mode 100644 index 0000000..30b08f1 --- /dev/null +++ b/developer/provisioning.mdx @@ -0,0 +1,414 @@ +--- +title: "Implementing provisioning" +sidebarTitle: "Provisioning" +description: "Grant and revoke access programmatically. Create accounts. Delete resources. This is where your connector becomes actionable." +--- + +Sync tells ConductorOne what access exists. Provisioning lets ConductorOne *change* access. + +| Operation | What it does | +|-----------|--------------| +| **Grant** | Add an entitlement to a principal (e.g., add user to group) | +| **Revoke** | Remove an entitlement from a principal | +| **CreateAccount** | Create a new user account in the target system | +| **DeleteResource** | Remove a resource (user, group, etc.) from the target system | + +Provisioning is optional - many connectors sync only. But this is where your connector goes from "showing access" to "managing access." It's the difference between a dashboard and a control plane. + +## The provisioning interfaces + +The SDK provides several interfaces you can implement: + +### Grant and revoke + +```go +// V2 interface (recommended for new connectors) +type GrantProvisionerV2 interface { + Grant(ctx context.Context, resource *v2.Resource, entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) +} + +type RevokeProvisioner interface { + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} +``` + + +**V2 vs V1:** The V2 interface returns a list of grants from Grant(). This handles cases where one logical grant creates multiple underlying grants. New connectors should use V2. + + +### CreateAccount + +```go +type AccountManagerLimited interface { + CreateAccount(ctx context.Context, + accountInfo *v2.AccountInfo, + credentialOptions *v2.LocalCredentialOptions) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) + CreateAccountCapabilityDetails(ctx context.Context) (*v2.CredentialDetailsAccountProvisioning, annotations.Annotations, error) +} +``` + +### DeleteResource + +```go +// V2 interface (recommended) +type ResourceDeleterV2Limited interface { + Delete(ctx context.Context, resourceId *v2.ResourceId, parentResourceID *v2.ResourceId) (annotations.Annotations, error) +} +``` + +## Implementing grant and revoke + +### When to implement + +If your target system supports adding/removing users from groups, roles, or permissions programmatically, implement Grant and Revoke. This covers most access control systems. + +### Grant implementation + +```go +func (g *groupBuilder) Grant(ctx context.Context, principal *v2.Resource, entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) { + // 1. Validate principal type + if principal.Id.ResourceType != userResourceType.Id { + return nil, nil, fmt.Errorf("only users can have group membership granted") + } + + // 2. Extract IDs + groupID := entitlement.Resource.Id.Resource + userID := principal.Id.Resource + + // 3. Call your API + err := g.client.AddUserToGroup(ctx, groupID, userID) + if err != nil { + // 4. Handle "already exists" gracefully + if isAlreadyExistsError(err) { + return nil, nil, nil // Success - idempotent + } + return nil, nil, fmt.Errorf("failed to add user to group: %w", err) + } + + // 5. Return the created grant (V2) + grant := sdkGrant.NewGrant(entitlement.Resource, entitlement.Slug, principal.Id) + return []*v2.Grant{grant}, nil, nil +} +``` + +### Revoke implementation + +```go +func (g *groupBuilder) Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) { + // 1. Validate principal type + if grant.Principal.Id.ResourceType != userResourceType.Id { + return nil, fmt.Errorf("only users can have group membership revoked") + } + + // 2. Extract IDs from grant + groupID := grant.Entitlement.Resource.Id.Resource + userID := grant.Principal.Id.Resource + + // 3. Call your API + err := g.client.RemoveUserFromGroup(ctx, groupID, userID) + if err != nil { + // 4. Handle "not found" gracefully + if isNotFoundError(err) { + return nil, nil // Success - already revoked + } + return nil, fmt.Errorf("failed to remove user from group: %w", err) + } + + return nil, nil +} +``` + +## Idempotency + +Provisioning operations should be idempotent - calling Grant twice for the same user+entitlement should succeed both times. This makes retries safe and simplifies the whole system. + +**Grant idempotency:** +- If user already has the entitlement, return success (not an error) +- HTTP 409 Conflict typically means "already exists" + +**Revoke idempotency:** +- If user doesn't have the entitlement, return success +- HTTP 404 Not Found typically means "already revoked" + +### Using annotations + +The SDK provides annotations to signal idempotent states: + +```go +// For already-exists during Grant +annos := annotations.Annotations{} +annos.Update(&v2.GrantAlreadyExists{}) +return nil, annos, nil + +// For already-revoked during Revoke +annos := annotations.Annotations{} +annos.Update(&v2.GrantAlreadyRevoked{}) +return annos, nil +``` + +## Real examples + +### Active Directory group membership (LDAP) + +```go +func (g *groupResourceType) Grant(ctx context.Context, principal *v2.Resource, entitlement *v2.Entitlement) (annotations.Annotations, error) { + if principal.Id.ResourceType != resourceTypeUser.Id { + return nil, fmt.Errorf("only users can have group membership granted") + } + + // Get LDAP entries + group, err := getEntryByObjectGUID(ctx, g.client, entitlement.Resource.Id.Resource) + if err != nil { + return nil, fmt.Errorf("failed to find group: %w", err) + } + user, err := getEntryByObjectGUID(ctx, g.client, principal.Id.Resource) + if err != nil { + return nil, fmt.Errorf("failed to find user: %w", err) + } + + // LDAP modify to add member + modifyRequest := ldap.NewModifyRequest(group.DN, nil) + modifyRequest.Add(attrGroupMember, []string{user.DN}) + + err = g.client.LdapModify(ctx, modifyRequest) + if err != nil { + if strings.Contains(err.Error(), "Already Exists") { + annos := annotations.Annotations{} + annos.Update(&v2.GrantAlreadyExists{}) + return annos, nil + } + return nil, fmt.Errorf("failed to grant group membership: %w", err) + } + + return nil, nil +} +``` + +### Google Workspace role revocation (REST API) + +```go +func (o *roleResourceType) Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) { + if grant.Principal.Id.ResourceType != resourceTypeUser.Id { + return nil, errors.New("user principal is required") + } + l := ctxzap.Extract(ctx) + + // Use grant.Id to delete the specific assignment + r := o.roleProvisioningService.RoleAssignments.Delete(o.customerId, grant.Id) + err := r.Context(ctx).Do() + if err != nil { + gerr := &googleapi.Error{} + if errors.As(err, &gerr) { + if gerr.Code == http.StatusNotFound { + // Already deleted - log and return success + l.Info("role assignment not found (already revoked)") + return nil, nil + } + } + return nil, fmt.Errorf("failed to remove role: %w", err) + } + + return nil, nil +} +``` + +## Edge cases + +### Validate principal type + +Always check that the principal is the expected type: + +```go +if principal.Id.ResourceType != userResourceType.Id { + return nil, nil, fmt.Errorf("only users can receive this entitlement") +} +``` + +### Store grant IDs + +If the target system returns an assignment ID, store it in the grant: + +```go +grant := sdkGrant.NewGrant(entitlement.Resource, slug, principal.Id) +grant.Id = assignmentIDFromAPI // This enables targeted revoke +return []*v2.Grant{grant}, nil, nil +``` + +This allows Revoke to use `grant.Id` directly instead of searching. + +### Multiple entitlement types + +If a resource offers multiple entitlement types, dispatch appropriately: + +```go +switch entitlement.Slug { +case "member": + err = g.client.AddMember(ctx, groupID, userID) +case "admin": + err = g.client.AddAdmin(ctx, groupID, userID) +default: + return nil, nil, fmt.Errorf("unknown entitlement: %s", entitlement.Slug) +} +``` + +## CreateAccount (JIT provisioning) + +CreateAccount enables just-in-time (JIT) user provisioning - accounts are created in target systems only when access is needed. + +### When to implement + +- User accounts can be created via API +- You want to enable JIT provisioning workflows +- The target system supports account creation without interactive signup + +### Basic pattern + +```go +func (u *userBuilder) CreateAccount(ctx context.Context, + accountInfo *v2.AccountInfo, + credentialOptions *v2.LocalCredentialOptions) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) { + + // 1. Create user in target system + user, err := u.client.CreateUser(ctx, &CreateUserRequest{ + Email: accountInfo.GetEmails()[0].GetAddress(), + FirstName: accountInfo.GetProfile().GetFirstName(), + LastName: accountInfo.GetProfile().GetLastName(), + }) + if err != nil { + return nil, nil, nil, fmt.Errorf("failed to create user: %w", err) + } + + // 2. Build success response + result := &v2.CreateAccountResponse_SuccessResult{ + Resource: user.ToResource(), + } + + return result, nil, nil, nil +} + +func (u *userBuilder) CreateAccountCapabilityDetails(ctx context.Context) (*v2.CredentialDetailsAccountProvisioning, annotations.Annotations, error) { + return &v2.CredentialDetailsAccountProvisioning{ + SupportedCredentialTypes: []v2.CredentialType{ + v2.CredentialType_CREDENTIAL_TYPE_PASSWORD, + }, + }, nil, nil +} +``` + +## DeleteResource + +DeleteResource removes resources (users, groups, etc.) from the target system. + +### When to implement + +- You want to deprovision users when they leave +- Resources can be deleted via API +- You want cleanup automation + +### Basic pattern + +```go +func (u *userBuilder) Delete(ctx context.Context, resourceId *v2.ResourceId, parentResourceID *v2.ResourceId) (annotations.Annotations, error) { + if resourceId.ResourceType != userResourceType.Id { + return nil, fmt.Errorf("can only delete users") + } + + err := u.client.DeleteUser(ctx, resourceId.Resource) + if err != nil { + if isNotFoundError(err) { + return nil, nil // Already deleted + } + return nil, fmt.Errorf("failed to delete user: %w", err) + } + + return nil, nil +} +``` + +## Declaring capabilities + +Capabilities are **auto-generated**, not manually written. When you implement provisioning interfaces, the connector binary automatically advertises those capabilities. + +```bash +# Generate the capability manifest +./baton-yourservice capabilities > baton_capabilities.json +``` + +This produces: + +```json +{ + "@type": "type.googleapis.com/c1.connector.v2.ConnectorCapabilities", + "resourceTypeCapabilities": [ + { + "resourceType": { "id": "group" }, + "capabilities": [ + { "@type": "...GrantCapability" }, + { "@type": "...RevokeCapability" } + ] + } + ], + "connectorCapabilities": [ + "CAPABILITY_SYNC", + "CAPABILITY_PROVISION" + ] +} +``` + +### Interface-to-capability mapping + +| Interface implemented | Capability declared | +|----------------------|---------------------| +| `ResourceSyncer` | `CAPABILITY_SYNC` | +| `ResourceProvisionerV2` | `CAPABILITY_PROVISION` + Grant/Revoke per resource type | +| `AccountManager` | `CAPABILITY_ACCOUNT_PROVISIONING` | +| `ResourceDeleterV2` | Resource deletion capabilities | + +## Next steps + + + + Run your provisioning-enabled connector in production + + + More patterns and techniques + + + Configuration-driven provisioning without Go code + + + +## Quick reference + +### Interfaces + +| Interface | Methods | Use for | +|-----------|---------|---------| +| `ResourceProvisionerV2` | `Grant()`, `Revoke()` | Adding/removing entitlements | +| `AccountManager` | `CreateAccount()` | JIT user provisioning | +| `ResourceDeleterV2` | `Delete()` | Removing users/resources | + +### Method signatures + +```go +// Grant (V2 - recommended) +Grant(ctx, principal *v2.Resource, entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) + +// Revoke +Revoke(ctx, grant *v2.Grant) (annotations.Annotations, error) + +// CreateAccount +CreateAccount(ctx, accountInfo *v2.AccountInfo, credentialOptions *v2.LocalCredentialOptions) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) + +// Delete (V2) +Delete(ctx, resourceId *v2.ResourceId, parentResourceID *v2.ResourceId) (annotations.Annotations, error) +``` + +### Idempotency checklist + +| Scenario | Expected behavior | +|----------|-------------------| +| Grant already exists | Return success (not error) | +| Revoke already revoked | Return success (not error) | +| Delete already deleted | Return success (not error) | +| Invalid principal type | Return error immediately | diff --git a/developer/recipes-auth.mdx b/developer/recipes-auth.mdx new file mode 100644 index 0000000..7987d12 --- /dev/null +++ b/developer/recipes-auth.mdx @@ -0,0 +1,144 @@ +--- +title: "Authentication recipes" +sidebarTitle: "Auth recipes" +description: "Battle-tested patterns for authenticating with target system APIs." +--- + +Each recipe includes the problem, solution code, and rationale. + +## API key authentication + +**Problem:** Connect to an API that uses API key in the Authorization header. + +**Solution:** + +```go +// pkg/client/client.go +import "github.com/conductorone/baton-sdk/pkg/uhttp" + +func NewClient(ctx context.Context, apiKey string) (*Client, error) { + httpClient, err := uhttp.NewBaseHttpClient(ctx, + uhttp.WithBearerToken(apiKey)) + if err != nil { + return nil, err + } + return &Client{http: httpClient, baseURL: "https://api.example.com"}, nil +} +``` + +**Why:** The SDK's `uhttp` package handles retries and rate limiting automatically. `WithBearerToken` sets `Authorization: Bearer `. + +## OAuth2 client credentials + +**Problem:** Exchange client ID and secret for an access token. + +**Solution:** + +```go +import ( + "golang.org/x/oauth2/clientcredentials" +) + +func NewClient(ctx context.Context, clientID, clientSecret, tokenURL string) (*Client, error) { + config := &clientcredentials.Config{ + ClientID: clientID, + ClientSecret: clientSecret, + TokenURL: tokenURL, + Scopes: []string{"read", "write"}, + } + + // This client automatically refreshes tokens + httpClient := config.Client(ctx) + + return &Client{http: httpClient}, nil +} +``` + +**Why:** The `clientcredentials` package handles token refresh automatically. You don't need to manage token expiry. + +## JWT service account (Google-style) + +**Problem:** Authenticate with a service account JSON key file. + +**Solution:** + +```go +import ( + "google.golang.org/api/option" + admin "google.golang.org/api/admin/directory/v1" +) + +func NewGoogleClient(ctx context.Context, credentialsJSON []byte, adminEmail string) (*admin.Service, error) { + // Parse the service account key + config, err := google.JWTConfigFromJSON(credentialsJSON, + admin.AdminDirectoryUserReadonlyScope, + admin.AdminDirectoryGroupReadonlyScope, + ) + if err != nil { + return nil, fmt.Errorf("failed to parse credentials: %w", err) + } + + // Impersonate a domain admin for domain-wide access + config.Subject = adminEmail + + // Create the service + return admin.NewService(ctx, option.WithHTTPClient(config.Client(ctx))) +} +``` + +**Why:** Domain-wide delegation requires impersonating a domain admin. The `Subject` field specifies which user to impersonate. + +## LDAP bind + +**Problem:** Connect to Active Directory or LDAP server. + +**Solution:** + +```go +import "github.com/go-ldap/ldap/v3" + +func NewLDAPClient(ctx context.Context, serverURL, bindDN, bindPassword string) (*ldap.Conn, error) { + conn, err := ldap.DialURL(serverURL) // ldaps://dc.example.com:636 + if err != nil { + return nil, fmt.Errorf("failed to connect to LDAP: %w", err) + } + + // Simple bind with username/password + err = conn.Bind(bindDN, bindPassword) + if err != nil { + conn.Close() + return nil, fmt.Errorf("failed to bind: %w", err) + } + + return conn, nil +} + +// For Kerberos/GSSAPI (domain-joined machines) +func NewLDAPClientKerberos(ctx context.Context, serverURL string) (*ldap.Conn, error) { + conn, err := ldap.DialURL(serverURL) + if err != nil { + return nil, err + } + + err = conn.ExternalBind() + if err != nil { + conn.Close() + return nil, err + } + + return conn, nil +} +``` + +**Why:** LDAP requires binding before any queries. Simple bind uses credentials; external bind uses OS-level Kerberos tickets. + +## Next + + + + Hierarchies and display name patterns + + + RawId annotation and per-connector ID formats + + diff --git a/developer/recipes-caching.mdx b/developer/recipes-caching.mdx new file mode 100644 index 0000000..726a9ce --- /dev/null +++ b/developer/recipes-caching.mdx @@ -0,0 +1,308 @@ +--- +title: "Caching recipes" +sidebarTitle: "Caching recipes" +description: "Battle-tested patterns for caching data across resource types, with critical anti-patterns to avoid." +--- + +Each recipe includes the problem, solution code, and rationale. + +## When to cache + +**Problem:** You need data from one resource type when processing another (e.g., resolving user IDs to emails when emitting grants). + +**When caching helps:** + +| Scenario | Cache? | Why | +|----------|--------|-----| +| Grants() needs user details from List() | Yes | Avoids N+1 API calls | +| Entitlements() needs role definitions | Yes | Role metadata is stable | +| List() needs parent context | Maybe | Often passed via parentID | +| Any data across sync runs | No | Stale data causes drift | + +**When NOT to cache:** +- Across sync runs (connector restarts clear caches anyway) +- Large datasets that don't fit in memory +- Data that changes frequently during sync + +## Thread-safe caching with sync.Map + +**Problem:** Cache data that's populated in one method and read in another, possibly concurrently. + +**Solution:** + +```go +// pkg/connector/connector.go +type Connector struct { + client *client.Client + + // Thread-safe caches + userCache sync.Map // map[userID]User + groupCache sync.Map // map[groupID]Group +} + +// Populate cache during List() +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, next, err := u.client.ListUsers(ctx, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range users { + // Cache for later lookup + u.connector.userCache.Store(user.ID, user) + + r, _ := resource.NewUserResource(user.Name, userResourceType, user.ID, + resource.WithEmail(user.Email, true)) + resources = append(resources, r) + } + + return resources, next, nil, nil +} + +// Use cache during Grants() +func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + memberIDs, next, err := g.client.GetGroupMemberIDs(ctx, resource.Id.Resource) + if err != nil { + return nil, "", nil, err + } + + var grants []*v2.Grant + for _, memberID := range memberIDs { + // Look up cached user + if cached, ok := g.connector.userCache.Load(memberID); ok { + user := cached.(User) + gr := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "user", Resource: memberID}) + grants = append(grants, gr) + } + } + + return grants, next, nil, nil +} +``` + +**Why:** `sync.Map` is safe for concurrent reads and writes. The SDK may call different builders concurrently. + +## Cross-resource lookups + +**Problem:** When emitting grants, you have member IDs but need to determine if they're users or groups. + +**Solution:** + +```go +type Connector struct { + client *client.Client + + // Track which IDs are which type + knownUsers sync.Map // map[id]bool + knownGroups sync.Map // map[id]bool +} + +// In user List(), mark IDs as users +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, next, err := u.client.ListUsers(ctx, pToken.Token) + for _, user := range users { + u.connector.knownUsers.Store(user.ID, true) + // ... + } + return resources, next, nil, nil +} + +// In Grants(), determine principal type +func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + members, next, err := g.client.GetGroupMembers(ctx, resource.Id.Resource) + + var grants []*v2.Grant + for _, member := range members { + principalType := g.resolvePrincipalType(member.ID) + gr := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: principalType, Resource: member.ID}) + grants = append(grants, gr) + } + + return grants, next, nil, nil +} + +func (c *Connector) resolvePrincipalType(id string) string { + if _, ok := c.knownUsers.Load(id); ok { + return "user" + } + if _, ok := c.knownGroups.Load(id); ok { + return "group" + } + return "user" // Default +} +``` + +**Why:** Many APIs return member IDs without type information. Caching during List() avoids expensive lookups during Grants(). + +## Cache warming order + +**Problem:** Grants() runs before the cache is populated. + +**Solution:** The SDK processes resource types in the order they're registered. Register types that populate caches first: + +```go +func (c *Connector) ResourceSyncers(ctx context.Context) []connectorbuilder.ResourceSyncer { + return []connectorbuilder.ResourceSyncer{ + // Users first - populates userCache + newUserBuilder(c), + // Groups second - populates groupCache + newGroupBuilder(c), + // Roles last - can use both caches in Grants() + newRoleBuilder(c), + } +} +``` + +**Why:** SDK processes syncers in order. If roles need user lookups, users must be synced first. + +## Anti-pattern: package-level caches + + +**This is a critical anti-pattern.** Package-level `sync.Map` variables persist state across sync runs in daemon mode, causing data corruption and phantom grants. + + +**Real example found in production connectors:** + +```go +// pkg/connector/helper.go - ANTI-PATTERN (do not copy) +var userCache sync.Map // Package-level - persists across syncs! + +func lookupUser(id string) (*User, bool) { + if cached, ok := userCache.Load(id); ok { + return cached.(*User), true + } + return nil, false +} +``` + +**What goes wrong:** + +1. Sync 1 runs, populates cache with users A, B, C +2. User B is deleted from the target system +3. Sync 2 runs in daemon mode (same process) +4. Cache still contains user B +5. Grants referencing user B appear valid but point to deleted user +6. Access reviews show phantom access that doesn't exist + +**Correct pattern - struct-scoped cache:** + +```go +// pkg/connector/connector.go - CORRECT +type Connector struct { + client *client.Client + userCache sync.Map // Struct field - fresh per connector instance +} + +// Each sync creates new Connector instance with empty cache +func New(ctx context.Context, client *client.Client) *Connector { + return &Connector{ + client: client, + userCache: sync.Map{}, // Fresh cache + } +} +``` + +**How to verify your connector:** + +```bash +# Search for package-level sync.Map declarations +grep -r "^var.*sync\.Map" pkg/ +``` + +If you find any, refactor them to struct fields. + +## Cache lifetime in daemon mode + +**Problem:** In daemon mode, the connector runs continuously processing multiple syncs. Caches need explicit lifetime management. + +**Runtime modes:** + +| Mode | Cache lifetime | Risk | +|------|----------------|------| +| **One-shot (CLI)** | Process lifetime | Low - process exits after sync | +| **Daemon mode** | Must be managed | High - stale data persists across syncs | + +**Solution:** Clear or recreate caches at sync boundaries: + +```go +type Connector struct { + client *client.Client + userCache sync.Map + + // Track when cache was populated + cachePopulatedAt time.Time +} + +// Called at start of each sync cycle +func (c *Connector) PrepareForSync(ctx context.Context) error { + // Clear caches from previous sync + c.userCache = sync.Map{} + c.cachePopulatedAt = time.Time{} + return nil +} +``` + +**Cache lifetime expectations:** + +| Scenario | Expected behavior | +|----------|-------------------| +| CLI one-shot | Cache lives for single sync, then process exits | +| Daemon between syncs | Cache should be cleared before each new sync | +| Daemon during sync | Cache valid for duration of single sync cycle | +| Long-running sync (>5 min) | Consider time-based invalidation | + +## Memory-bounded caching + +**Problem:** Caching all users exhausts memory in large organizations. + +**Solution:** For very large datasets, use LRU cache or skip caching entirely: + +```go +import "github.com/hashicorp/golang-lru/v2" + +type Connector struct { + // LRU cache with max size + userCache *lru.Cache[string, User] +} + +func New(ctx context.Context) (*Connector, error) { + cache, err := lru.New[string, User](10000) // Max 10k entries + if err != nil { + return nil, err + } + return &Connector{userCache: cache}, nil +} +``` + +**Alternative:** For truly large datasets, accept the N+1 lookup cost or batch lookups: + +```go +// Batch lookup instead of caching everything +func (c *Connector) lookupUsers(ctx context.Context, ids []string) (map[string]User, error) { + return c.client.GetUsersByIDs(ctx, ids) // Single API call for batch +} +``` + +**Why:** Memory is finite. A connector that OOMs is worse than one that makes extra API calls. + +## Next + + + + Authentication patterns + + + Run your connector in production + + diff --git a/developer/recipes-id.mdx b/developer/recipes-id.mdx new file mode 100644 index 0000000..57c6d6e --- /dev/null +++ b/developer/recipes-id.mdx @@ -0,0 +1,106 @@ +--- +title: "ID correlation recipes" +sidebarTitle: "ID recipes" +description: "Battle-tested patterns for RawId annotation and per-connector ID formats." +--- + +Each recipe includes the problem, solution code, and rationale. + +## Per-connector ID formats + +**Problem:** Different external systems use different ID formats. Knowing what value to use for `RawId` annotation is critical for correlation. + +**Solution:** Use the external system's native stable identifier: + +| Connector | ID source | Format | Example | +|-----------|-----------|--------|---------| +| **Okta** | `app.Id` / `group.Id` | 20-char alphanumeric | `0oa1xyz789abcdef0h7` | +| **Azure AD** | Object ID (not App ID) | UUID | `12345678-1234-1234-1234-123456789012` | +| **GCP** | Resource path | `projects/{id}` | `projects/my-project-123` | +| **AWS** | ARN | Full ARN | `arn:aws:iam::123456789012:role/Admin` | +| **GitHub** | Node ID or numeric ID | Integer as string | `12345678` | +| **Salesforce** | Salesforce ID | 18-char ID | `00e3h000000bRQAAA2` | +| **Google Workspace** | Google Group ID | Variable | `00gjdgxs3x1h123` | + +**Why this matters:** ConductorOne uses these IDs to correlate resources across syncs. Using the wrong ID causes duplicate objects or failed correlations. + +**Key points:** +- Azure AD has two IDs: Object ID (use this) and Application ID (client ID for OAuth) +- AWS uses full ARNs, not account IDs alone +- GitHub has numeric IDs and GraphQL node IDs; either works but be consistent + +## Setting RawId annotation + +**Problem:** Ensure ConductorOne can correlate your resources across syncs and match resources created via Terraform. + +**What is RawId?** A string annotation containing the external system's native identifier. ConductorOne uses this to match resources across syncs and to merge Terraform-created objects with connector-discovered ones. + +**Solution:** Add the `RawId` annotation when building resources: + +```go +// Inline with resource creation (preferred) +userResource, err := resource.NewUserResource( + user.DisplayName, + userResourceType, + user.ID, + userTraitOptions, + resource.WithAnnotation(&v2.RawId{Id: user.ID}), +) + +// Or add after creation +groupResource, err := resource.NewGroupResource( + group.Name, + groupResourceType, + group.ID, + groupTraitOptions, +) +groupResource.WithAnnotation(&v2.RawId{Id: group.Email}) +``` + +**What value to use:** The external system's native, stable identifier. Choose an ID that: +- Won't change when the resource is renamed or modified +- Is unique within that resource type +- Is what an admin would recognize from the external system + +## Examples from production connectors + +| System | Resource | RawId value | Why | +|--------|----------|-------------|-----| +| Okta | User | `user.Id` (`00u1abc...`) | Okta's internal ID, stable across renames | +| Okta | App | `app.Id` (`0oa1xyz...`) | Okta's internal app ID | +| Google Workspace | User | `user.PrimaryEmail` | Email is the canonical identifier | +| Google Workspace | Group | `group.Email` | Group email is stable and recognizable | +| GCP | Project | `project.ProjectId` | Project ID (not display name) | +| AWS | Role | Full ARN | ARNs are globally unique | +| Azure AD | User | Object ID (UUID) | Not the UPN, which can change | + +## Common mistakes + +| Mistake | Problem | Fix | +|---------|---------|-----| +| Using display names | Change when renamed | Use system-generated IDs | +| Using array indices | Change with ordering | Use stable identifiers | +| Different IDs for same resource | Correlation fails | Use consistent ID source | +| Using mutable fields | Resource appears as new each sync | Use immutable system IDs | + +## When to use RawId + +**Should have RawId:** +- Apps, groups, roles, and any resource that might be pre-created via Terraform +- Resources that need stable correlation across syncs +- Any resource type where admins might reference objects by external ID + +**May not need RawId:** +- Ephemeral or derived resources +- Resources only used internally by the connector + +## Next + + + + Mock servers and testability patterns + + + Thread-safe caching and anti-patterns + + diff --git a/developer/recipes-modeling.mdx b/developer/recipes-modeling.mdx new file mode 100644 index 0000000..d263b0b --- /dev/null +++ b/developer/recipes-modeling.mdx @@ -0,0 +1,292 @@ +--- +title: "Resource modeling recipes" +sidebarTitle: "Modeling recipes" +description: "Battle-tested patterns for structuring resources, hierarchies, and display names." +--- + +Each recipe includes the problem, solution code, and rationale. + +## Parent-child hierarchies + +**Problem:** Model resources that exist within other resources (projects in organizations, repos in orgs). + +**Solution:** + +```go +// Define parent type with child annotation +var orgResourceType = &v2.ResourceType{ + Id: "organization", + DisplayName: "Organization", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_GROUP}, +} + +var projectResourceType = &v2.ResourceType{ + Id: "project", + DisplayName: "Project", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_GROUP}, +} + +// In org List(), declare children +func (o *orgBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + orgs, err := o.client.ListOrgs(ctx) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, org := range orgs { + r, _ := resource.NewResource(org.Name, orgResourceType, org.ID, + // Declare that this org has project children + resource.WithAnnotation(&v2.ChildResourceType{ + ResourceTypeId: projectResourceType.Id, + }), + ) + resources = append(resources, r) + } + + return resources, "", nil, nil +} + +// In project List(), reference parent +func (p *projectBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + // parentID is the org ID when SDK calls this for each org + if parentID == nil { + return nil, "", nil, nil // Projects only exist within orgs + } + + projects, err := p.client.ListProjects(ctx, parentID.Resource) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, proj := range projects { + r, _ := resource.NewResource(proj.Name, projectResourceType, proj.ID, + resource.WithParentResourceID(parentID), + ) + resources = append(resources, r) + } + + return resources, "", nil, nil +} +``` + +**Why:** Parent-child relationships let the SDK scope `List()` calls. The UI can show hierarchical navigation. Entitlements inherit context from their parent. + +## Display name fallbacks + +**Problem:** Ensure resources always have a human-readable name. + +**Solution:** + +```go +func displayNameFor(user User) string { + if user.DisplayName != "" { + return user.DisplayName + } + if user.Name != "" { + return user.Name + } + if user.Email != "" { + return user.Email + } + // Last resort - never return empty + return user.ID +} + +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, _ := u.client.ListUsers(ctx) + + var resources []*v2.Resource + for _, user := range users { + r, _ := resource.NewUserResource( + displayNameFor(user), // Never empty + userResourceType, + user.ID, + ) + resources = append(resources, r) + } + + return resources, "", nil, nil +} +``` + +**Why:** Empty display names break UIs and access reviews. Reviewers can't approve access to "(blank)". + +## Error handling + +### Wrap errors with context + +**Problem:** Make errors traceable to their source. + +**Solution:** + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.ListUsers(ctx) + if err != nil { + // Include connector name and operation + return nil, "", nil, fmt.Errorf("baton-example: failed to list users: %w", err) + } + + // ... +} +``` + +**Why:** The connector name prefix makes it clear which connector produced the error. The `%w` verb preserves the error chain for `errors.Is()` and `errors.As()`. + +### Distinguish retryable vs fatal errors + +**Problem:** Let the SDK know which errors are worth retrying. + +**Solution:** + +```go +import "github.com/conductorone/baton-sdk/pkg/connectorbuilder" + +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.ListUsers(ctx) + if err != nil { + // Check for retryable errors + if isRateLimitError(err) || isNetworkError(err) { + // SDK will retry automatically + return nil, "", nil, err + } + + // Fatal errors (bad credentials, permission denied) + if isAuthError(err) { + return nil, "", nil, fmt.Errorf("baton-example: authentication failed (check credentials): %w", err) + } + + return nil, "", nil, err + } + // ... +} + +func isRateLimitError(err error) bool { + var httpErr *HTTPError + if errors.As(err, &httpErr) { + return httpErr.StatusCode == 429 + } + return false +} +``` + +**Why:** The SDK handles retries for transient errors. Clear error messages for fatal errors help users fix configuration issues. + +### Check context cancellation in loops + +**Problem:** Respect timeouts and cancellation in long-running operations. + +**Solution:** + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.ListUsers(ctx) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range users { + // Check for cancellation + select { + case <-ctx.Done(): + return nil, "", nil, ctx.Err() + default: + } + + r, err := resource.NewUserResource(user.Name, userResourceType, user.ID) + if err != nil { + return nil, "", nil, err + } + resources = append(resources, r) + } + + return resources, "", nil, nil +} +``` + +**Why:** A cancelled context means "stop now." Ignoring it wastes resources and can cause timeouts in subsequent operations. + +## Anti-patterns + +### Don't buffer entire datasets + +```go +// WRONG: Loading everything into memory +allUsers, _ := client.GetAllUsers(ctx) // Could be millions + +// CORRECT: Paginate +users, nextCursor, _ := client.ListUsers(ctx, cursor, 100) +``` + +### Don't swallow errors + +```go +// WRONG: Ignoring errors +users, _ := client.ListUsers(ctx) + +// CORRECT: Return errors +users, err := client.ListUsers(ctx) +if err != nil { + return nil, "", nil, fmt.Errorf("baton-example: failed to list users: %w", err) +} +``` + +### Don't log sensitive data + +```go +// WRONG: Logging tokens +l.Info("authenticating", zap.String("token", token)) + +// CORRECT: Never log credentials +l.Info("authenticating", zap.String("user", username)) +``` + +### Don't mix resource types in grants + +```go +// WRONG: Grant with mismatched types +grant.NewGrant( + groupResource, + "member", + &v2.ResourceId{ + ResourceType: "app", // Wrong! + Resource: userID, + }, +) + +// CORRECT: Consistent resource types +grant.NewGrant( + groupResource, + "member", + &v2.ResourceId{ + ResourceType: userResourceType.Id, + Resource: userID, + }, +) +``` + +## Next + + + + RawId annotation and per-connector ID formats + + + Mock servers and testability patterns + + diff --git a/developer/recipes-testing.mdx b/developer/recipes-testing.mdx new file mode 100644 index 0000000..c39fe58 --- /dev/null +++ b/developer/recipes-testing.mdx @@ -0,0 +1,242 @@ +--- +title: "Testing recipes" +sidebarTitle: "Testing recipes" +description: "Battle-tested patterns for testing connectors without hitting production." +--- + +Each recipe includes the problem, solution code, and rationale. + +## Configurable base URL for mocks + +**Problem:** Test connector against mock server without hitting production. + +**Solution:** + +```go +// pkg/config/config.go +type Config struct { + APIKey string `mapstructure:"api-key"` + BaseURL string `mapstructure:"base-url"` // Required for testability +} + +// cmd/baton-example/main.go +func main() { + fields := []field.SchemaField{ + field.StringField("api-key", + field.WithRequired(true), + field.WithDescription("API key for authentication"), + ), + field.StringField("base-url", + field.WithDefaultValue("https://api.example.com"), + field.WithDescription("Base URL (use http://localhost:8080 for testing)"), + ), + } + // ... +} + +// pkg/client/client.go +func New(cfg *Config) *Client { + baseURL := cfg.BaseURL + if baseURL == "" { + baseURL = "https://api.example.com" + } + return &Client{baseURL: baseURL} +} +``` + +**Usage:** + +```bash +# Production +./baton-example --api-key $KEY + +# Testing +./baton-example --api-key test --base-url http://localhost:8080 +``` + +**Why:** Without configurable base URL, you can't test without hitting production. This breaks CI/CD and risks exposing production data. + +## Insecure TLS for local testing + +**Problem:** Test against mock server with self-signed certificate. + +**Solution:** + +```go +// pkg/config/config.go +type Config struct { + APIKey string `mapstructure:"api-key"` + BaseURL string `mapstructure:"base-url"` + Insecure bool `mapstructure:"insecure"` // Skip TLS verification +} + +// pkg/client/client.go +func New(ctx context.Context, cfg *Config) (*Client, error) { + opts := []uhttp.Option{} + + if cfg.Insecure { + transport := &http.Transport{ + TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, + } + opts = append(opts, uhttp.WithTransport(transport)) + } + + httpClient, err := uhttp.NewBaseHttpClient(ctx, opts...) + if err != nil { + return nil, err + } + + return &Client{http: httpClient, baseURL: cfg.BaseURL}, nil +} +``` + +**Usage:** + +```bash +# Testing with self-signed cert +./baton-example --api-key test --base-url https://localhost:8443 --insecure +``` + +**Why:** Mock servers often use self-signed certificates. The `--insecure` flag enables testing without installing custom CAs. + + +Never use `--insecure` in production. This flag is strictly for local testing with mock servers. + + +## LDAP server-side paging + +**Problem:** Paginate large LDAP result sets. + +**Solution:** + +```go +import ( + "encoding/base64" + "github.com/go-ldap/ldap/v3" +) + +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + const pageSize = 1000 + + // Create paging control + pagingControl := ldap.NewControlPaging(uint32(pageSize)) + + // Restore cookie from token if continuing + if pToken.Token != "" { + cookie, err := base64.StdEncoding.DecodeString(pToken.Token) + if err != nil { + return nil, "", nil, fmt.Errorf("invalid pagination token: %w", err) + } + pagingControl.SetCookie(cookie) + } + + // Execute search with paging control + searchRequest := ldap.NewSearchRequest( + "dc=example,dc=com", + ldap.ScopeWholeSubtree, + ldap.NeverDerefAliases, + 0, 0, false, + "(objectClass=user)", + []string{"cn", "mail", "sAMAccountName"}, + []ldap.Control{pagingControl}, + ) + + result, err := u.conn.Search(searchRequest) + if err != nil { + return nil, "", nil, err + } + + // Convert entries to resources + var resources []*v2.Resource + for _, entry := range result.Entries { + r, _ := resource.NewUserResource( + entry.GetAttributeValue("cn"), + userResourceType, + entry.GetAttributeValue("sAMAccountName"), + resource.WithEmail(entry.GetAttributeValue("mail"), true), + ) + resources = append(resources, r) + } + + // Extract next page cookie + nextToken := "" + pagingResult := ldap.FindControl(result.Controls, ldap.ControlTypePaging) + if pc, ok := pagingResult.(*ldap.ControlPaging); ok && len(pc.Cookie) > 0 { + nextToken = base64.StdEncoding.EncodeToString(pc.Cookie) + } + + return resources, nextToken, nil, nil +} +``` + +**Why:** LDAP servers handle pagination server-side with cookies. The cookie must be base64-encoded to fit in the string token. + +## Nested pagination with Bag + +**Problem:** Paginate children within paginated parents (e.g., members within each group). + +**Solution:** + +```go +func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + bag := &pagination.Bag{} + if err := bag.Unmarshal(pToken.Token); err != nil { + return nil, "", nil, err + } + + // On first call for this resource, push initial state + if bag.Current() == nil { + bag.Push(pagination.PageState{ + ResourceID: resource.Id.Resource, + ResourceTypeID: resource.Id.ResourceType, + }) + } + + // Get current page of members + members, nextCursor, err := g.client.GetGroupMembers(ctx, + bag.Current().ResourceID, + bag.PageToken()) + if err != nil { + return nil, "", nil, err + } + + // Convert to grants + var grants []*v2.Grant + for _, member := range members { + g := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "user", Resource: member.ID}) + grants = append(grants, g) + } + + // Update bag with next page cursor + if nextCursor != "" { + bag.Next(nextCursor) + } else { + bag.Pop() // Done with this resource + } + + nextToken, err := bag.Marshal() + if err != nil { + return nil, "", nil, err + } + + return grants, nextToken, nil, nil +} +``` + +**Why:** The `Bag` type is a stack that manages nested pagination state. Push when entering a nested level, pop when done. + +## Next + + + + Thread-safe caching and anti-patterns + + + Grant and revoke operations + + diff --git a/developer/sql-authoring.mdx b/developer/sql-authoring.mdx new file mode 100644 index 0000000..62bc03e --- /dev/null +++ b/developer/sql-authoring.mdx @@ -0,0 +1,306 @@ +--- +title: "baton-sql: database integration" +sidebarTitle: "SQL authoring" +description: "Integrate any SQL database without writing Go code. Configuration only." +--- + +Use baton-sql for any SQL database: internal user directories, custom applications, or legacy systems with database access. + +## Connection configuration + +```yaml +version: "1" +app_name: "Internal User Directory" +app_description: "Syncs users and groups from PostgreSQL" + +connect: + scheme: "postgres" # or "mysql", "oracle", "sqlserver", "sqlite" + host: "${DB_HOST}" + port: "5432" + database: "${DB_NAME}" + user: "${DB_USER}" + password: "${DB_PASS}" +``` + +**Supported databases:** +- PostgreSQL (`postgres`) +- MySQL (`mysql`) +- Oracle (`oracle`) +- SQL Server (`sqlserver`) +- SAP HANA (`hana`) +- SQLite (`sqlite`) + +## Listing resources + +```yaml +resource_types: + user: + name: "User" + list: + query: | + SELECT id, username, email, status, created_at + FROM users + WHERE active = true + pagination: + strategy: "offset" + primary_key: "id" + map: + id: ".id" + display_name: ".username" + traits: + user: + emails: + - ".email" + status: ".status" +``` + +## Grants discovery + +```yaml +grants: + - query: | + SELECT u.username as user_id, g.name as group_name + FROM users u + JOIN group_members gm ON u.id = gm.user_id + JOIN groups g ON gm.group_id = g.id + WHERE g.id = ? + map: + - principal_id: ".user_id" + principal_type: "user" + entitlement_id: "member" +``` + +The `?` syntax binds the current resource ID to the query parameter. + +## Entitlements + +### Static entitlements + +```yaml +resource_types: + group: + static_entitlements: + - id: "member" + display_name: "'Member'" + purpose: "assignment" + grantable_to: + - "user" +``` + +### Dynamic entitlements + +Discover entitlements from the database: + +```yaml +entitlements: + query: | + SELECT id, name, description FROM permissions + map: + id: ".id" + display_name: ".name" + description: ".description" +``` + +## Provisioning + +### Grant and revoke + +```yaml +static_entitlements: + - id: "member" + display_name: "'Member'" + purpose: "assignment" + grantable_to: + - "user" + provisioning: + vars: + user_id: "principal.ID" + group_id: "resource.ID" + grant: + queries: + - | + INSERT INTO group_members (user_id, group_id, created_at) + VALUES (?, ?, NOW()) + revoke: + queries: + - | + DELETE FROM group_members + WHERE user_id = ? AND group_id = ? +``` + +### Account creation (JIT provisioning) + +```yaml +resource_types: + user: + account_provisioning: + schema: + - name: "username" + type: "string" + required: true + - name: "email" + type: "string" + required: true + + credentials: + random_password: + min_length: 16 + max_length: 32 + preferred: true + + create: + vars: + username: "input.username" + email: "input.email" + password: "password" + queries: + - | + INSERT INTO users (username, email, password_hash) + VALUES (?, ?, crypt(?, gen_salt('bf'))) +``` + +### Credential rotation + +```yaml +credential_rotation: + credentials: + random_password: + min_length: 16 + max_length: 32 + preferred: true + update: + vars: + user_id: "resource_id" + password: "password" + queries: + - | + UPDATE users SET password_hash = crypt(?, gen_salt('bf')) + WHERE id = ? +``` + +## Complete example + +```yaml +version: "1" +app_name: "Internal User Directory" +app_description: "Syncs users and groups from internal PostgreSQL database" + +connect: + scheme: "postgres" + host: "${DB_HOST}" + port: "5432" + database: "directory" + user: "${DB_USER}" + password: "${DB_PASSWORD}" + +resource_types: + user: + name: "User" + description: "Internal directory user" + + list: + query: | + SELECT id, username, email, first_name, last_name, + status, department, created_at + FROM users + WHERE deleted_at IS NULL + pagination: + strategy: "offset" + primary_key: "id" + map: + id: ".id" + display_name: ".first_name + ' ' + .last_name" + traits: + user: + emails: + - ".email" + status: ".status == 'active' ? 'enabled' : 'disabled'" + profile: + user_id: ".username" + first_name: ".first_name" + last_name: ".last_name" + + group: + name: "Group" + description: "User group for access control" + + list: + query: | + SELECT id, name, description FROM groups + pagination: + strategy: "offset" + primary_key: "id" + map: + id: ".id" + display_name: ".name" + description: ".description" + + static_entitlements: + - id: "member" + display_name: "'Member'" + purpose: "assignment" + grantable_to: + - "user" + provisioning: + vars: + user_id: "principal.ID" + group_id: "resource.ID" + grant: + queries: + - | + INSERT INTO group_members (user_id, group_id, added_at) + VALUES (?, ?, NOW()) + ON CONFLICT DO NOTHING + revoke: + queries: + - | + DELETE FROM group_members + WHERE user_id = ? AND group_id = ? + + grants: + - query: | + SELECT gm.user_id, g.id as group_id + FROM group_members gm + JOIN groups g ON gm.group_id = g.id + WHERE g.id = ? + map: + - principal_id: ".user_id" + principal_type: "user" + entitlement_id: "member" +``` + +## Running baton-sql + +### Validate configuration + +```bash +baton-sql --config-path ./config.yaml --validate-config-only +``` + +### One-shot mode (local testing) + +```bash +baton-sql --config-path ./config.yaml -f sync.c1z +baton resources -f sync.c1z +baton grants -f sync.c1z +``` + +### Service mode with provisioning + +```bash +baton-sql --config-path ./config.yaml \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" \ + --provisioning +``` + +## Related + + + + baton-http for REST API integration + + + Data transformation with CEL + + diff --git a/developer/submit.mdx b/developer/submit.mdx new file mode 100644 index 0000000..9b86d25 --- /dev/null +++ b/developer/submit.mdx @@ -0,0 +1,197 @@ +--- +title: "Publishing connectors" +sidebarTitle: "Publishing" +description: "Share your connector with the community. Make access management better for everyone." +--- + +You've built a connector that works. Publishing makes it available to everyone using ConductorOne: +- Other organizations can deploy it +- It appears in the Connector Hub +- It becomes eligible for hosted mode (ConductorOne runs it for customers) +- The community can contribute improvements + +## Publishing flow + +``` +Your Connector Connector Registry Connector Hub + | | | + | 1. Create Connector | | + |--------------------------->| | + | | | + | 2. Create Version | | + |--------------------------->| | + | | | + | 3. Upload Binaries | | + |--------------------------->| | + | | | + | 4. Finalize Version | | + |--------------------------->| | + | | 5. Published | + | |-------------------------->| +``` + +### Version lifecycle + +``` +PENDING -> UPLOADING -> VALIDATING -> PUBLISHED + | + +------> FAILED (validation error) + +PUBLISHED -> YANKED (deprecated/broken) +``` + +| State | Meaning | +|-------|---------| +| `PENDING` | Version created, awaiting asset uploads | +| `UPLOADING` | Assets are being uploaded | +| `VALIDATING` | Assets uploaded, validation in progress | +| `PUBLISHED` | Version available for download | +| `YANKED` | Version withdrawn (still visible but marked) | +| `FAILED` | Validation failed | + +## Publishing paths + +### Contributing to existing connectors + +Before building a new connector, check if one already exists. If it does but lacks features you need: + +**Option A: Contribute upstream** +- Fork the repository +- Add your changes +- Submit a pull request +- Maintainers review and merge +- New version published with your changes + +**Option B: Fork and maintain** +- Fork the repository +- Publish under your organization +- Maintain independently + +### New connectors + +**Option A: Open source under ConductorOne** +- Work with ConductorOne to host in their GitHub org +- Benefits from existing CI/CD and publishing infrastructure + +**Option B: Open source under your organization** +- Host in your own GitHub organization +- Publish to registry under your org name +- Full control, full responsibility + +**Option C: Internal only** +- Don't publish to the public registry +- Deploy in daemon mode on your infrastructure +- Suitable for proprietary internal systems + +## Supported platforms + +| Platform | GOOS | GOARCH | +|----------|------|--------| +| darwin-amd64 | darwin | amd64 | +| darwin-arm64 | darwin | arm64 | +| linux-amd64 | linux | amd64 | +| linux-arm64 | linux | arm64 | +| windows-amd64 | windows | amd64 | + +### Signing keys + +Connectors can be signed with: + +| Type | Description | +|------|-------------| +| `GPG` | GnuPG signatures | +| `COSIGN` | Sigstore cosign signatures | + +## Contributing a connector + +### Before you start + +1. **Check for existing connectors**: Search the Connector Hub for your target system. If one exists, consider contributing improvements rather than building from scratch. + +2. **Review the SDK**: Familiarize yourself with the [baton-sdk](https://github.com/ConductorOne/baton-sdk) and the building connectors guide. + +3. **Understand the target system's API**: You'll need API credentials with sufficient permissions to list users, groups, roles, and other access-related resources. + +### Implementation requirements + +Your connector must meet these requirements before submission: + +| Requirement | Details | +|-------------|---------| +| **ResourceSyncer interface** | Implement `ResourceType()`, `List()`, `Entitlements()`, `Grants()` | +| **Lint checks pass** | Run `make lint` with standard golangci-lint configuration | +| **Tests included** | Unit tests for resource builders; integration tests recommended | +| **README.md** | Document API permissions required, configuration options, usage examples | +| **License** | Apache 2.0 (standard for Baton ecosystem) | +| **Capabilities manifest** | Include `baton_capabilities.json` declaring supported operations | + +### Pull request process + +#### For existing connectors + +```bash +# Fork the repository +gh repo fork ConductorOne/baton-yourservice +cd baton-yourservice + +# Create a feature branch +git checkout -b feature/add-role-support + +# Make your changes, then run local validation +make build +make lint +make test +./dist/*/baton-yourservice --help + +# Submit a pull request +``` + +#### For new connectors + +1. Use the standard project structure +2. Ensure CI/CD is configured with standard GitHub workflows +3. Contact ConductorOne if you want official hosting: + - Open an issue on [baton-sdk](https://github.com/ConductorOne/baton-sdk/issues) + - Include: target system name, API documentation link, your use case +4. Or publish independently under your organization + +### Review criteria + +| Criterion | What reviewers check | +|-----------|---------------------| +| **Correctness** | Does it accurately model the target system's access structure? | +| **Completeness** | Are all relevant resource types included? | +| **Code quality** | Does it follow SDK patterns and pass lint? | +| **Test coverage** | Are there tests for the key behaviors? | +| **Documentation** | Is the README clear and complete? | +| **Security** | No credentials logged, proper error handling | + +### Security reporting + +If you discover a security issue: + +- **DO NOT** file a public issue +- **DO** send a private report to [security@conductorone.com](mailto:security@conductorone.com) + +Security researchers are publicly thanked for responsible disclosure. + +## Quick reference + +### Registry API methods + +| Method | Description | +|--------|-------------| +| `CreateConnector` | Register a new connector | +| `CreateVersion` | Create a new version | +| `GetUploadURLs` | Get presigned upload URLs | +| `FinalizeVersion` | Complete upload and validate | +| `YankVersion` | Mark version as yanked | + + + + Implementation guide + + + Get help and contribute + + diff --git a/developer/syncing.mdx b/developer/syncing.mdx new file mode 100644 index 0000000..e5d468a --- /dev/null +++ b/developer/syncing.mdx @@ -0,0 +1,455 @@ +--- +title: "Building connectors" +sidebarTitle: "Syncing resources" +description: "Build a connector that produces clean resources, entitlements, and grants - and can be tested without touching production." +--- + +You are building a Go module that implements the `ResourceSyncer` contract. The interface is focused: four methods per resource type, and the SDK handles everything else. + +Your connector answers three questions: + +1. **What exists?** Users, groups, roles, applications +2. **What permissions are available?** Entitlements that can be granted +3. **Who has what?** Grants connecting users to permissions + +The Baton SDK handles orchestration, output format, pagination coordination, and communication with ConductorOne. You focus on translating your system's API into the Resource/Entitlement/Grant model. + +## Project structure + +### Directory layout + +A common structure for Baton connectors: + +``` +baton-{service}/ + cmd/baton-{service}/ + main.go # Entry point, config setup + pkg/ + config/ + config.go # Configuration fields (API keys, URLs, etc.) + connector/ + connector.go # Register resource types with the SDK + users.go # User resource builder + groups.go # Group resource builder + roles.go # Role resource builder (if applicable) + resource_types.go # Shared resource type definitions + client/ # Optional: API wrapper + client.go # HTTP client for target system API + .github/workflows/ # CI/release automation + ci.yaml # Build, lint, test on PRs + release.yaml # Build and publish releases + .golangci.yml # Lint configuration + baton_capabilities.json # Capability manifest (what operations are supported) + go.mod # Dependencies (includes baton-sdk) + go.sum # Dependency checksums + Makefile # Build targets + README.md # Usage documentation + LICENSE # Apache 2.0 (standard for Baton) +``` + + +Not all connectors follow this exact structure. Some organize code differently based on their needs. The structure above is a common starting point, not a requirement. + + +The naming convention is `baton-{service}` - for example, `baton-github`, `baton-okta`, `baton-salesforce`. + +### Key files + +| File | Purpose | +|------|---------| +| `cmd/.../main.go` | Entry point. Parses config, creates connector, runs CLI | +| `pkg/connector/connector.go` | Registers all resource builders with the SDK | +| `pkg/connector/*.go` | One file per resource type implementing ResourceSyncer | +| `pkg/client/client.go` | Wraps target system API with Go methods | +| `baton_capabilities.json` | Declares what operations (sync, grant, revoke) are supported | + +### Makefile targets + +Standard connectors include these make targets: + +```makefile +# Build the connector binary +make build +# Output: dist/{os}_{arch}/baton-{service} + +# Run golangci-lint +make lint + +# Update dependencies +make update-deps +``` + +### Setting up a new connector + + + + ```bash + mkdir baton-yourservice + cd baton-yourservice + go mod init github.com/your-org/baton-yourservice + ``` + + + ```bash + go get github.com/conductorone/baton-sdk + ``` + + + ```bash + mkdir -p cmd/baton-yourservice pkg/connector pkg/client + ``` + + + From an existing connector: + - `.golangci.yml` (lint configuration) + - `Makefile` (build targets) + - `.github/workflows/ci.yaml` (CI workflow) + - `.github/workflows/release.yaml` (release workflow) + + + Following the patterns in this guide + + + +### Capability manifest + +The capability manifest declares what operations your connector supports. This file is **auto-generated** by running: + +```bash +./dist/*/baton-yourservice capabilities > baton_capabilities.json +``` + +Example manifest: + +```json +{ + "@type": "type.googleapis.com/c1.connector.v2.ConnectorCapabilities", + "resourceTypeCapabilities": [ + { + "resourceType": { + "id": "user", + "displayName": "User", + "traits": ["TRAIT_USER"] + }, + "capabilities": ["CAPABILITY_SYNC"] + }, + { + "resourceType": { + "id": "group", + "displayName": "Group", + "traits": ["TRAIT_GROUP"] + }, + "capabilities": ["CAPABILITY_SYNC", "CAPABILITY_PROVISION"] + } + ] +} +``` + +| Capability | Meaning | +|------------|---------| +| `CAPABILITY_SYNC` | Resource type participates in sync | +| `CAPABILITY_TARGETED_SYNC` | Supports fetching specific resources by ID | +| `CAPABILITY_PROVISION` | Supports Grant/Revoke operations | + + +Do not write this file manually. Always generate it from the connector binary to ensure accuracy. + + +## Implementing ResourceSyncer + +The `ResourceSyncer` interface is the heart of your connector. Per resource type, implement four methods: +- `ResourceType(ctx) *v2.ResourceType` +- `List(ctx, parentResourceID, token) ([]*v2.Resource, nextToken, annotations, error)` +- `Entitlements(ctx, resource, token) ([]*v2.Entitlement, nextToken, annotations, error)` +- `Grants(ctx, resource, token) ([]*v2.Grant, nextToken, annotations, error)` + +### ResourceType() + +Defines what this resource is: + +```go +func (u *userBuilder) ResourceType(ctx context.Context) *v2.ResourceType { + return &v2.ResourceType{ + Id: "user", + DisplayName: "User", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_USER}, + } +} +``` + +Traits tell ConductorOne how to interpret the resource. Use `TRAIT_USER` for people, `TRAIT_GROUP` for collections, `TRAIT_ROLE` for permission bundles. + +### List() + +Fetches all instances of this resource type: + +```go +func (u *userBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + // Call your API + users, nextPage, err := u.client.GetUsers(ctx, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + // Convert to Baton resources + var resources []*v2.Resource + for _, user := range users { + r, err := resource.NewUserResource(user.Name, userResourceType, user.ID, + resource.WithEmail(user.Email, true)) + if err != nil { + return nil, "", nil, err + } + resources = append(resources, r) + } + + return resources, nextPage, nil, nil +} +``` + +Return a page of resources plus a token for the next page. Empty token means you're done. The SDK calls you repeatedly until you return an empty token. + +### The RawId annotation + +Always include a `RawId` annotation with the external system's stable identifier: + +```go +r, err := resource.NewUserResource(user.Name, userResourceType, user.ID, + resource.WithEmail(user.Email, true)) +if err != nil { + return nil, "", nil, err +} +// Add the external system's ID for correlation +r.WithAnnotation(&v2.RawId{Id: user.ID}) +``` + +**Why this matters:** ConductorOne uses the `RawId` to: +- **Correlate resources across syncs** - Same ID = same resource, not a duplicate +- **Track provenance** - Know which connector discovered which resource +- **Enable pre-sync patterns** - Support reservation mechanisms that create placeholders before sync + +| System | RawId value | Example | +|--------|-------------|---------| +| Okta | `app.Id` | `0oa1xyz789abcdef0` | +| AWS | ARN | `arn:aws:iam::123456789:user/alice` | +| GCP | Resource name | `projects/my-project-123` | +| Azure AD | Object ID | `550e8400-e29b-41d4-a716-446655440000` | +| GitHub | Node ID or numeric ID | `MDQ6VXNlcjE=` or `12345` | + +### Entitlements() + +Defines what permissions this resource offers: + +```go +func (g *groupBuilder) Entitlements(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Entitlement, string, annotations.Annotations, error) { + + // Groups typically offer membership + membership := entitlement.NewAssignmentEntitlement(resource, "member", + entitlement.WithDisplayName("Member"), + entitlement.WithDescription("Member of this group")) + + return []*v2.Entitlement{membership}, "", nil, nil +} +``` + +Users typically return empty here - they receive grants, they don't offer entitlements. + +### Grants() + +Reports who has each entitlement: + +```go +func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + // Get group members from API + members, nextPage, err := g.client.GetGroupMembers(ctx, resource.Id.Resource, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var grants []*v2.Grant + for _, member := range members { + g := grant.NewGrant(resource, "member", + &v2.ResourceId{ResourceType: "user", Resource: member.ID}) + grants = append(grants, g) + } + + return grants, nextPage, nil, nil +} +``` + + +**Pagination must progress**: the SDK detects and errors if your "next page token" repeats the input token. + + +## Modeling decisions + +How you structure resources and entitlements determines what ConductorOne can manage. + +### What to sync as a resource? + +| Good candidates | Why | +|-----------------|-----| +| Users | People who have access | +| Groups | Collections that grant access | +| Roles | Permission bundles | +| Teams | Organizational units with permissions | +| Projects/Workspaces | Scoped containers | + +| Skip these | Why | +|------------|-----| +| Business data | Customers, orders, tickets - not access control | +| Logs/events | Operational data, not identity | +| Configurations | Unless they control who can do what | + +### Entitlement granularity + +**Fine-grained:** Separate entitlements for read, write, admin +- Pro: More control in access reviews +- Con: More complexity, more grants to manage + +**Coarse-grained:** Single "access" entitlement +- Pro: Simpler model +- Con: Can't revoke admin without revoking everything + +Choose based on how access decisions are made. If "can this person admin the database?" is a real question, admin should be a separate entitlement. + +### Parent-child relationships + +Some systems have hierarchies: organization -> project -> resource. + +```go +// Parent declares it has children +orgResource, _ := resource.NewResource("Acme Corp", orgType, "org-123", + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "project"})) + +// Child references parent +projectResource, _ := resource.NewResource("Platform", projectType, "proj-456", + resource.WithParentResourceID(orgResource.Id)) +``` + +Use hierarchies when: +- Child resources only make sense within a parent context +- You need to scope List() calls to a parent +- The target API is organized hierarchically + +See [Pagination patterns](/developer/pagination) for handling hierarchical data with nested pagination. + +## Definition of done + +Your connector is ready when: + +- **Sync works deterministically** (same inputs produce stable IDs and consistent results across runs) +- **Pagination works** (no token loops; handles large datasets) +- **You can run without production ConductorOne credentials** (local testing story exists) + +### Build and test + +```bash +make build +./dist/baton-yourservice --api-key $KEY --log-level debug + +# Inspect results +baton resources -f sync.c1z +baton grants -f sync.c1z +``` + +## Common mistakes + +### Resource type mismatches + +A grant references a principal by `ResourceId` (type + id). If your principal type id doesn't match what you used in `ResourceType()`, you will create dangling edges. + +### Implicit capability claims + +A connector may have a `--provisioning` flag but still not implement specific provisioners. Treat "flag exists" as necessary, not sufficient. + +### API clients + +If the service has an official Go SDK, use it. Otherwise, the SDK's `uhttp` package handles rate limiting and retries: + +```go +import "github.com/conductorone/baton-sdk/pkg/uhttp" + +httpClient, _ := uhttp.NewBaseHttpClient(ctx) +``` + +### Error handling + +Return errors, don't panic. Wrap errors with context: + +```go +if err != nil { + return nil, "", nil, fmt.Errorf("baton-yourservice: failed to list users: %w", err) +} +``` + +### Credentials + +Never log credentials. Use the SDK's `SecretString` type for sensitive config fields: + +```go +type Config struct { + APIKey types.SecretString `mapstructure:"api-key"` +} +``` + +## Next steps + + + + Cursor, offset, LDAP paging, and nested pagination with Bag + + + Grant, revoke, and account lifecycle operations + + + Run modes, credentials, monitoring + + + Real-world patterns and solutions + + + +## Quick reference + +### Resource traits + +| Trait | Use for | +|-------|---------| +| `TRAIT_USER` | Individual accounts | +| `TRAIT_GROUP` | Collections of users | +| `TRAIT_ROLE` | Permission bundles | +| `TRAIT_APP` | Applications | + +### Method return signatures + +```go +// List returns: resources, nextPageToken, annotations, error +([]*v2.Resource, string, annotations.Annotations, error) + +// Entitlements returns: entitlements, nextPageToken, annotations, error +([]*v2.Entitlement, string, annotations.Annotations, error) + +// Grants returns: grants, nextPageToken, annotations, error +([]*v2.Grant, string, annotations.Annotations, error) +``` + +### SDK helpers + +```go +// Create user resource +resource.NewUserResource(name, resourceType, id, ...options) + +// Create group resource +resource.NewGroupResource(name, resourceType, id, ...options) + +// Create entitlement +entitlement.NewAssignmentEntitlement(resource, slug, ...options) + +// Create grant +grant.NewGrant(resource, entitlementSlug, principalID) +``` diff --git a/docs.json b/docs.json index 3933f21..a7694cf 100644 --- a/docs.json +++ b/docs.json @@ -443,10 +443,70 @@ "tab": "Developer", "icon": "code", "pages": [ - "developer/intro", - "developer/sdk", - "developer/postman", - "developer/terraform" + { + "group": "Getting started", + "pages": [ + "developer/intro", + "developer/concepts" + ] + }, + { + "group": "Building connectors", + "pages": [ + "developer/syncing", + "developer/pagination", + "developer/provisioning" + ] + }, + { + "group": "Cookbook", + "pages": [ + "developer/recipes-auth", + "developer/recipes-modeling", + "developer/recipes-id", + "developer/recipes-testing", + "developer/recipes-caching" + ] + }, + { + "group": "Meta-connectors", + "pages": [ + "developer/http-authoring", + "developer/sql-authoring", + "developer/cel-expressions" + ] + }, + { + "group": "Troubleshooting", + "pages": [ + "developer/debugging", + "developer/error-codes" + ] + }, + { + "group": "Reference", + "pages": [ + "developer/baton-sdk", + "developer/config-schema", + "developer/c1-api", + "developer/glossary" + ] + }, + { + "group": "Publishing", + "pages": [ + "developer/submit", + "developer/community" + ] + }, + { + "group": "Tools", + "pages": [ + "developer/sdk", + "developer/postman", + "developer/terraform" + ] + } ] }, { From f4177b7b45238e68ef3d8bde063009fbfb1f6b96 Mon Sep 17 00:00:00 2001 From: rch Date: Thu, 22 Jan 2026 19:31:30 -0800 Subject: [PATCH 2/3] Add RAP documentation for AI agents - 27 skill files in rap/ for Retrieval Augmented Prompt usage - INDEX.md with selection guidelines - Breadcrumb in developer/intro.mdx pointing to rap/INDEX.md --- developer/intro.mdx | 6 ++ rap/INDEX.md | 153 +++++++++++++++++++++++++++ rap/build-pagination.md | 183 ++++++++++++++++++++++++++++++++ rap/build-setup.md | 169 ++++++++++++++++++++++++++++++ rap/build-syncer.md | 214 +++++++++++++++++++++++++++++++++++++ rap/community.md | 139 ++++++++++++++++++++++++ rap/concepts-ids.md | 67 ++++++++++++ rap/concepts-overview.md | 74 +++++++++++++ rap/concepts-resources.md | 95 +++++++++++++++++ rap/concepts-sync.md | 76 ++++++++++++++ rap/debug-errors.md | 123 ++++++++++++++++++++++ rap/debug-workflow.md | 96 +++++++++++++++++ rap/meta-cel.md | 128 +++++++++++++++++++++++ rap/meta-http.md | 147 ++++++++++++++++++++++++++ rap/meta-sql.md | 157 ++++++++++++++++++++++++++++ rap/ops-modes.md | 100 ++++++++++++++++++ rap/provision-grant.md | 145 +++++++++++++++++++++++++ rap/publish-submit.md | 160 ++++++++++++++++++++++++++++ rap/recipes-auth.md | 139 ++++++++++++++++++++++++ rap/recipes-caching.md | 192 ++++++++++++++++++++++++++++++++++ rap/recipes-errors.md | 158 ++++++++++++++++++++++++++++ rap/recipes-modeling.md | 184 ++++++++++++++++++++++++++++++++ rap/recipes-testing.md | 215 ++++++++++++++++++++++++++++++++++++++ rap/ref-c1api.md | 184 ++++++++++++++++++++++++++++++++ rap/ref-config.md | 202 +++++++++++++++++++++++++++++++++++ rap/ref-faq.md | 156 +++++++++++++++++++++++++++ rap/ref-glossary.md | 71 +++++++++++++ rap/ref-sdk.md | 211 +++++++++++++++++++++++++++++++++++++ 28 files changed, 3944 insertions(+) create mode 100644 rap/INDEX.md create mode 100644 rap/build-pagination.md create mode 100644 rap/build-setup.md create mode 100644 rap/build-syncer.md create mode 100644 rap/community.md create mode 100644 rap/concepts-ids.md create mode 100644 rap/concepts-overview.md create mode 100644 rap/concepts-resources.md create mode 100644 rap/concepts-sync.md create mode 100644 rap/debug-errors.md create mode 100644 rap/debug-workflow.md create mode 100644 rap/meta-cel.md create mode 100644 rap/meta-http.md create mode 100644 rap/meta-sql.md create mode 100644 rap/ops-modes.md create mode 100644 rap/provision-grant.md create mode 100644 rap/publish-submit.md create mode 100644 rap/recipes-auth.md create mode 100644 rap/recipes-caching.md create mode 100644 rap/recipes-errors.md create mode 100644 rap/recipes-modeling.md create mode 100644 rap/recipes-testing.md create mode 100644 rap/ref-c1api.md create mode 100644 rap/ref-config.md create mode 100644 rap/ref-faq.md create mode 100644 rap/ref-glossary.md create mode 100644 rap/ref-sdk.md diff --git a/developer/intro.mdx b/developer/intro.mdx index 4f87bf7..a5e2d07 100644 --- a/developer/intro.mdx +++ b/developer/intro.mdx @@ -259,3 +259,9 @@ Plus: | Sync | Reading access data from a system | | Provision | Writing access changes back (grant, revoke, create, delete) | | Reconciliation | Comparing actual vs desired access and correcting drift | + + diff --git a/rap/INDEX.md b/rap/INDEX.md new file mode 100644 index 0000000..7f5a20d --- /dev/null +++ b/rap/INDEX.md @@ -0,0 +1,153 @@ +# Baton Connector Documentation Index + +Documentation for building ConductorOne Baton connectors. Request relevant sections based on user's question. + +## How to Use + +1. Read user's question +2. Identify relevant sections (up to 3-4) +3. Request those files +4. Answer using retrieved content + +## Available Sections + +### Conceptual + +| Section | File | Covers | +|---------|------|--------| +| What connectors do | `concepts-overview.md` | Problem solved, sync vs provision, reconciliation loop | +| Resource model | `concepts-resources.md` | Resources, entitlements, grants, traits | +| Sync lifecycle | `concepts-sync.md` | Four stages, pagination | +| ID correlation | `concepts-ids.md` | RawId, external_id, matching across syncs | + +### Building Connectors + +| Section | File | Covers | +|---------|------|--------| +| Project setup | `build-setup.md` | Directory structure, go.mod, main.go | +| ResourceSyncer | `build-syncer.md` | ResourceType(), List(), Entitlements(), Grants() | +| Pagination | `build-pagination.md` | Token vs Bag, cursor vs offset, nested | + +### Provisioning + +| Section | File | Covers | +|---------|------|--------| +| Grant and Revoke | `provision-grant.md` | ResourceProvisionerV2, Grant/Revoke | + +### Recipes (Cookbook) + +| Section | File | Covers | +|---------|------|--------| +| Authentication | `recipes-auth.md` | API key, OAuth2, JWT, LDAP, basic auth | +| Error handling | `recipes-errors.md` | Retryable vs fatal, context cancellation | +| Resource modeling | `recipes-modeling.md` | Hierarchies, traits, entitlements, grants | +| Testing | `recipes-testing.md` | Configurable base URL, mock servers, unit tests | +| Caching | `recipes-caching.md` | sync.Map, LRU, daemon mode, anti-patterns | + +### Meta-Connectors + +| Section | File | Covers | +|---------|------|--------| +| baton-http | `meta-http.md` | REST API via YAML config | +| baton-sql | `meta-sql.md` | Database via YAML config | +| CEL expressions | `meta-cel.md` | Data transformation | + +### Operations + +| Section | File | Covers | +|---------|------|--------| +| Run modes | `ops-modes.md` | One-shot vs daemon | + +### Troubleshooting + +| Section | File | Covers | +|---------|------|--------| +| Debugging workflow | `debug-workflow.md` | Step-by-step process | +| Common errors | `debug-errors.md` | Pagination loops, auth, rate limits | + +### Reference + +| Section | File | Covers | +|---------|------|--------| +| SDK interfaces | `ref-sdk.md` | ConnectorBuilder, ResourceSyncer, Provisioner | +| Configuration | `ref-config.md` | Flags, env vars, field types | +| C1 API | `ref-c1api.md` | Task types, lifecycle, heartbeat | +| FAQ | `ref-faq.md` | Common questions | +| Glossary | `ref-glossary.md` | Term definitions | + +### Publishing & Community + +| Section | File | Covers | +|---------|------|--------| +| Publishing | `publish-submit.md` | Registry, versioning, signing | +| Community | `community.md` | Getting help, contributing, reporting | + +--- + +## Selection Guidelines + +**"How do I..."** +- Build a connector -> `build-setup.md`, `build-syncer.md` +- Add pagination -> `build-pagination.md` +- Support provisioning -> `provision-grant.md` +- Connect REST API -> `meta-http.md` +- Connect database -> `meta-sql.md` +- Authenticate -> `recipes-auth.md` +- Handle errors -> `recipes-errors.md` +- Model resources -> `recipes-modeling.md` +- Test my connector -> `recipes-testing.md` +- Cache data -> `recipes-caching.md` +- Debug a problem -> `debug-workflow.md`, `debug-errors.md` +- Run in production -> `ops-modes.md` +- Publish my connector -> `publish-submit.md` +- Contribute -> `community.md` + +**"What is..."** +- A connector -> `concepts-overview.md` +- An entitlement/grant -> `concepts-resources.md` +- The sync lifecycle -> `concepts-sync.md` +- RawId -> `concepts-ids.md` +- CEL -> `meta-cel.md` +- daemon mode -> `ops-modes.md` +- c1z file -> `ref-faq.md` + +**Code with errors** +- Error message -> `debug-errors.md` +- Interface issue -> `build-syncer.md`, `ref-sdk.md` +- Pagination issue -> `build-pagination.md` +- Auth failure -> `recipes-auth.md`, `debug-errors.md` +- Cache problems -> `recipes-caching.md` + +**Configuration** +- CLI flags -> `ref-config.md` +- Environment variables -> `ref-config.md` +- Config files -> `ref-config.md` + +**Architecture** +- SDK interfaces -> `ref-sdk.md` +- C1 communication -> `ref-c1api.md` + +--- + +## Usage Examples + +User: "How do I implement pagination?" +Retrieve: `build-pagination.md` + +User: "Pagination loop error" +Retrieve: `debug-errors.md`, `build-pagination.md` + +User: "Connect REST API without Go" +Retrieve: `meta-http.md`, `meta-cel.md` + +User: "OAuth2 authentication" +Retrieve: `recipes-auth.md` + +User: "Cache users between List and Grants" +Retrieve: `recipes-caching.md` + +User: "What SDK interfaces do I implement?" +Retrieve: `ref-sdk.md` + +User: "How does daemon mode work?" +Retrieve: `ops-modes.md`, `ref-c1api.md` diff --git a/rap/build-pagination.md b/rap/build-pagination.md new file mode 100644 index 0000000..a874ea3 --- /dev/null +++ b/rap/build-pagination.md @@ -0,0 +1,183 @@ +# build-pagination + +Token vs Bag, cursor vs offset, nested pagination patterns. + +--- + +## Two Pagination Strategies + +**Cursor-based**: API returns opaque token for next page +``` +GET /users?cursor=abc123 +Response: { users: [...], next_cursor: "def456" } +``` + +**Offset-based**: You track page number or offset +``` +GET /users?offset=100&limit=50 +Response: { users: [...], total: 500 } +``` + +## Simple Pagination with Token + +For flat lists where API returns a next-page token: + +```go +func (b *userBuilder) List( + ctx context.Context, + parentResourceID *v2.ResourceId, + token *pagination.Token, +) ([]*v2.Resource, string, annotations.Annotations, error) { + pageToken := "" + if token != nil && token.Token != "" { + pageToken = token.Token + } + + users, nextToken, err := b.client.ListUsers(ctx, pageToken, 100) + if err != nil { + return nil, "", nil, err + } + + // Convert users to resources... + + // Return nextToken directly - empty string means done + return resources, nextToken, nil, nil +} +``` + +## Offset-Based Pagination + +When API uses offset/limit instead of cursors: + +```go +func (b *userBuilder) List( + ctx context.Context, + parentResourceID *v2.ResourceId, + token *pagination.Token, +) ([]*v2.Resource, string, annotations.Annotations, error) { + offset := 0 + if token != nil && token.Token != "" { + var err error + offset, err = strconv.Atoi(token.Token) + if err != nil { + return nil, "", nil, err + } + } + + pageSize := 100 + users, total, err := b.client.ListUsers(ctx, offset, pageSize) + if err != nil { + return nil, "", nil, err + } + + // Convert users to resources... + + // Calculate next token + nextOffset := offset + len(users) + nextToken := "" + if nextOffset < total { + nextToken = strconv.Itoa(nextOffset) + } + + return resources, nextToken, nil, nil +} +``` + +## Nested Pagination with Bag + +When paginating children within parents (e.g., members within groups): + +```go +func (b *groupBuilder) Grants( + ctx context.Context, + resource *v2.Resource, + token *pagination.Token, +) ([]*v2.Grant, string, annotations.Annotations, error) { + bag := &pagination.Bag{} + if token != nil && token.Token != "" { + err := bag.Unmarshal(token.Token) + if err != nil { + return nil, "", nil, err + } + } + + // Get or initialize page state + pageState := bag.Current() + if pageState == nil { + pageState = &pagination.PageState{ + ResourceTypeID: "member", + ResourceID: resource.Id.Resource, + } + bag.Push(pageState) + } + + // Fetch members using page state token + members, nextCursor, err := b.client.GetGroupMembers( + ctx, + resource.Id.Resource, + pageState.Token, + ) + if err != nil { + return nil, "", nil, err + } + + // Convert to grants... + + // Update pagination state + if nextCursor != "" { + pageState.Token = nextCursor + } else { + bag.Pop() + } + + nextToken, err := bag.Marshal() + if err != nil { + return nil, "", nil, err + } + + return grants, nextToken, nil, nil +} +``` + +## Meta-Connector Pagination + +In YAML config, declare pagination strategy: + +```yaml +# Cursor-based +pagination: + strategy: "cursor" + primary_key: "id" + +# Offset-based +pagination: + strategy: "offset" + limit_param: "limit" + offset_param: "offset" + page_size: 100 +``` + +For baton-sql, use query placeholders: +```sql +SELECT * FROM users +WHERE id > ? +ORDER BY id ASC +LIMIT ? +``` + +## Critical Invariant + +**Your next token must progress.** The SDK detects loops where the same token is returned twice in a row and errors. + +```go +// WRONG - returns same token, causes infinite loop +if hasMore { + return resources, currentToken, nil, nil // Bug! +} + +// RIGHT - return new token or empty string +if hasMore { + return resources, newToken, nil, nil +} +return resources, "", nil, nil // Done +``` diff --git a/rap/build-setup.md b/rap/build-setup.md new file mode 100644 index 0000000..bb8e1af --- /dev/null +++ b/rap/build-setup.md @@ -0,0 +1,169 @@ +# build-setup + +Directory structure, go.mod, main.go, Makefile for a new connector. + +--- + +## Project Structure + +``` +baton-myservice/ + cmd/ + baton-myservice/ + main.go # Entry point + pkg/ + connector/ + connector.go # Connector interface implementation + users.go # User resource syncer + groups.go # Group resource syncer + client.go # API client wrapper + go.mod + go.sum + Makefile + .golangci.yaml +``` + +## go.mod + +```go +module github.com/conductorone/baton-myservice + +go 1.21 + +require ( + github.com/conductorone/baton-sdk v0.2.x + github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 +) +``` + +Check baton-sdk for the current minimum Go version. + +## main.go + +```go +package main + +import ( + "context" + "fmt" + "os" + + "github.com/conductorone/baton-sdk/pkg/config" + "github.com/conductorone/baton-sdk/pkg/connectorbuilder" + "github.com/conductorone/baton-sdk/pkg/field" + "github.com/conductorone/baton-sdk/pkg/types" + "github.com/conductorone/baton-myservice/pkg/connector" +) + +var version = "dev" + +func main() { + ctx := context.Background() + + cfg := &config.Config{ + Fields: []field.SchemaField{ + field.StringField("api-token", + field.WithRequired(true), + field.WithDescription("API token for authentication"), + ), + field.StringField("base-url", + field.WithDescription("API base URL"), + field.WithDefaultValue("https://api.myservice.com"), + ), + }, + } + + cb, err := connector.New(ctx, cfg) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + + c, err := connectorbuilder.NewConnector(ctx, cb) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + + connectorbuilder.Run(ctx, c, cfg) +} +``` + +## Connector Constructor + +```go +// pkg/connector/connector.go +package connector + +type Connector struct { + client *MyServiceClient +} + +func New(ctx context.Context, cfg *config.Config) (*Connector, error) { + token := cfg.GetString("api-token") + baseURL := cfg.GetString("base-url") + + client, err := NewClient(baseURL, token) + if err != nil { + return nil, err + } + + return &Connector{client: client}, nil +} + +func (c *Connector) ResourceSyncers(ctx context.Context) []connectorbuilder.ResourceSyncer { + return []connectorbuilder.ResourceSyncer{ + newUserBuilder(c.client), + newGroupBuilder(c.client), + } +} + +func (c *Connector) Metadata(ctx context.Context) (*v2.ConnectorMetadata, error) { + return &v2.ConnectorMetadata{ + DisplayName: "My Service", + Description: "Syncs users and groups from My Service", + }, nil +} + +func (c *Connector) Validate(ctx context.Context) (annotations.Annotations, error) { + // Test the connection + _, err := c.client.GetCurrentUser(ctx) + if err != nil { + return nil, fmt.Errorf("failed to validate credentials: %w", err) + } + return nil, nil +} +``` + +## Makefile + +```makefile +GOOS = $(shell go env GOOS) +GOARCH = $(shell go env GOARCH) +BUILD_DIR = dist + +.PHONY: build +build: + go build -o $(BUILD_DIR)/baton-myservice ./cmd/baton-myservice + +.PHONY: lint +lint: + golangci-lint run + +.PHONY: test +test: + go test -v ./... + +.PHONY: update-deps +update-deps: + go get -u ./... + go mod tidy +``` + +## Required Dependencies + +| Tool | Purpose | +|------|---------| +| Go | See go.mod for version | +| golangci-lint | Code quality | +| make | Build automation | diff --git a/rap/build-syncer.md b/rap/build-syncer.md new file mode 100644 index 0000000..d0f2526 --- /dev/null +++ b/rap/build-syncer.md @@ -0,0 +1,214 @@ +# build-syncer + +Implementing ResourceType(), List(), Entitlements(), Grants() methods. + +--- + +## The ResourceSyncer Interface + +```go +type ResourceSyncer interface { + ResourceType(ctx context.Context) *v2.ResourceType + List(ctx context.Context, parentResourceID *v2.ResourceId, token *pagination.Token) ( + []*v2.Resource, string, annotations.Annotations, error) + Entitlements(ctx context.Context, resource *v2.Resource, token *pagination.Token) ( + []*v2.Entitlement, string, annotations.Annotations, error) + Grants(ctx context.Context, resource *v2.Resource, token *pagination.Token) ( + []*v2.Grant, string, annotations.Annotations, error) +} +``` + +## Complete User Builder Example + +```go +package connector + +import ( + "context" + + v2 "github.com/conductorone/baton-sdk/pb/c1/connector/v2" + "github.com/conductorone/baton-sdk/pkg/annotations" + "github.com/conductorone/baton-sdk/pkg/pagination" + rs "github.com/conductorone/baton-sdk/pkg/types/resource" +) + +var userResourceType = &v2.ResourceType{ + Id: "user", + DisplayName: "User", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_USER}, +} + +type userBuilder struct { + client *MyServiceClient +} + +func newUserBuilder(client *MyServiceClient) *userBuilder { + return &userBuilder{client: client} +} + +func (b *userBuilder) ResourceType(ctx context.Context) *v2.ResourceType { + return userResourceType +} + +func (b *userBuilder) List( + ctx context.Context, + parentResourceID *v2.ResourceId, + token *pagination.Token, +) ([]*v2.Resource, string, annotations.Annotations, error) { + // Parse page token + pageToken := "" + if token != nil && token.Token != "" { + pageToken = token.Token + } + + // Fetch from API + users, nextToken, err := b.client.ListUsers(ctx, pageToken, 100) + if err != nil { + return nil, "", nil, err + } + + // Convert to resources + var resources []*v2.Resource + for _, user := range users { + resource, err := rs.NewUserResource( + user.Name, + userResourceType, + user.ID, + []rs.UserTraitOption{ + rs.WithEmail(user.Email, true), + rs.WithStatus(v2.UserTrait_Status_STATUS_ENABLED), + rs.WithUserLogin(user.Username), + }, + ) + if err != nil { + return nil, "", nil, err + } + resources = append(resources, resource) + } + + return resources, nextToken, nil, nil +} + +func (b *userBuilder) Entitlements( + ctx context.Context, + resource *v2.Resource, + token *pagination.Token, +) ([]*v2.Entitlement, string, annotations.Annotations, error) { + // Users typically don't offer entitlements + return nil, "", nil, nil +} + +func (b *userBuilder) Grants( + ctx context.Context, + resource *v2.Resource, + token *pagination.Token, +) ([]*v2.Grant, string, annotations.Annotations, error) { + // Users typically don't have grants on them + return nil, "", nil, nil +} +``` + +## Group Builder (With Entitlements and Grants) + +```go +var groupResourceType = &v2.ResourceType{ + Id: "group", + DisplayName: "Group", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_GROUP}, +} + +type groupBuilder struct { + client *MyServiceClient +} + +func (b *groupBuilder) ResourceType(ctx context.Context) *v2.ResourceType { + return groupResourceType +} + +func (b *groupBuilder) List( + ctx context.Context, + parentResourceID *v2.ResourceId, + token *pagination.Token, +) ([]*v2.Resource, string, annotations.Annotations, error) { + groups, nextToken, err := b.client.ListGroups(ctx, token.Token, 100) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, group := range groups { + resource, err := rs.NewGroupResource( + group.Name, + groupResourceType, + group.ID, + []rs.GroupTraitOption{}, + ) + if err != nil { + return nil, "", nil, err + } + resources = append(resources, resource) + } + + return resources, nextToken, nil, nil +} + +func (b *groupBuilder) Entitlements( + ctx context.Context, + resource *v2.Resource, + token *pagination.Token, +) ([]*v2.Entitlement, string, annotations.Annotations, error) { + // Groups offer membership + entitlement := &v2.Entitlement{ + Id: "member", + DisplayName: "Member", + Description: "Member of " + resource.DisplayName, + Resource: resource, + GrantableTo: []*v2.ResourceType{userResourceType}, + Purpose: v2.Entitlement_PURPOSE_VALUE_ASSIGNMENT, + Slug: "member", + } + + return []*v2.Entitlement{entitlement}, "", nil, nil +} + +func (b *groupBuilder) Grants( + ctx context.Context, + resource *v2.Resource, + token *pagination.Token, +) ([]*v2.Grant, string, annotations.Annotations, error) { + groupID := resource.Id.Resource + + members, nextToken, err := b.client.GetGroupMembers(ctx, groupID, token.Token) + if err != nil { + return nil, "", nil, err + } + + var grants []*v2.Grant + for _, member := range members { + grant := &v2.Grant{ + Entitlement: &v2.Entitlement{ + Id: "member", + Resource: resource, + }, + Principal: &v2.Resource{ + Id: &v2.ResourceId{ + ResourceType: "user", + Resource: member.UserID, + }, + }, + } + grants = append(grants, grant) + } + + return grants, nextToken, nil, nil +} +``` + +## Key Points + +- **ResourceType()**: Called once per sync to learn resource type metadata +- **List()**: May be called multiple times for pagination +- **Entitlements()**: Called once per resource instance +- **Grants()**: Called once per resource instance, may paginate + +Return empty string for nextToken when done paginating. diff --git a/rap/community.md b/rap/community.md new file mode 100644 index 0000000..a8fc887 --- /dev/null +++ b/rap/community.md @@ -0,0 +1,139 @@ +# community + +Getting help and contributing to the Baton connector ecosystem. + +--- + +## Getting help + +### GitHub Discussions + +| Repository | Use For | +|------------|---------| +| [baton-sdk](https://github.com/ConductorOne/baton-sdk/discussions) | SDK questions, general development | +| Specific connector repos | Issues with that connector | + +Before asking: +1. Search existing issues +2. Check documentation +3. Review Cookbook patterns + +### Good question format + +Include: +- What you're trying to accomplish +- What you've tried +- Error messages (full text) +- Relevant code +- Connector version + +Bad: "My connector doesn't work" + +Good: "Sync fails with 'unauthorized' when listing users. Using baton-okta v0.5.2, API token auth, read_users scope. Error: [full text]" + +--- + +## Support channels + +| Channel | Response | Use For | +|---------|----------|---------| +| GitHub Issues | Days | Bugs, features | +| GitHub Discussions | Days | Questions | +| ConductorOne Support | Hours | Production (customers) | + +--- + +## Reporting issues + +### Bug report template + +```markdown +**Connector:** baton-example v1.2.3 +**Environment:** macOS 14.0, Go 1.23 + +**Steps:** +1. Configure with API key +2. Run sync +3. Observe error + +**Expected:** Sync completes +**Actual:** Error: [paste] +``` + +### Feature requests + +Describe the problem, not just the solution: + +Good: "Need to sync custom Okta attributes for access reviews" + +Not: "Add custom attribute support" + +### Security issues + +DO NOT file publicly. + +Report to: security@conductorone.com + +--- + +## Contributing + +### Pre-PR checklist + +- [ ] Follows SDK patterns +- [ ] `make lint` passes +- [ ] `make test` passes +- [ ] README documents permissions +- [ ] No credentials in code + +### Flow + +``` +1. Fork repo +2. Create branch +3. Make changes +4. Run build/lint/test +5. Submit PR +6. Address feedback +7. Merge +``` + +--- + +## Maintainer guide + +### Responsibilities + +| Task | Frequency | +|------|-----------| +| Triage issues | Weekly | +| Review PRs | As submitted | +| Update deps | Monthly | +| Release versions | As needed | +| Security monitoring | Ongoing | + +### Release process + +```bash +git tag v1.2.3 +git push origin v1.2.3 +# GitHub Actions handles build and release +``` + +### Issue handling + +| Type | Response | +|------|----------| +| Bug with repro | Prioritize fix | +| Bug without repro | Request info | +| Feature request | Evaluate, label | +| Question | Point to docs | +| Security | Follow security process | + +--- + +## Code of conduct + +Follows Contributor Covenant. + +Report violations to: open-source@conductorone.com diff --git a/rap/concepts-ids.md b/rap/concepts-ids.md new file mode 100644 index 0000000..eb1d32e --- /dev/null +++ b/rap/concepts-ids.md @@ -0,0 +1,67 @@ +# concepts-ids + +RawId annotation, external_id, and how ConductorOne matches resources across syncs. + +--- + +## Why ID Correlation Matters + +ConductorOne needs to know if a resource in this sync is the same resource from a previous sync. This enables: +- Tracking changes over time +- Correlating resources across connectors +- Supporting pre-sync reservation patterns + +## The RawId Annotation + +When building a resource, include the external system's ID: + +```go +resource, _ := resourceBuilder.NewGroupResource( + group.Name, + groupResourceType, + group.Id, // Used internally by SDK + []resource.GroupTraitOption{}, +) +// Add external ID for correlation +resource.WithAnnotation(&v2.RawId{Id: group.Id}) +``` + +## ID Flow Through the System + +| Stage | Term | Purpose | +|-------|------|---------| +| Connector output | `RawId` annotation | External system's stable identifier | +| Sync storage | `external_id` | Same value on ConnectorResource records | +| Domain objects | `source_connector_ids` | Map of connector_id to external_id | +| Domain objects | `raw_baton_id` | Canonical external ID after merge | + +Flow: `RawId` (connector) -> `external_id` (sync) -> `source_connector_ids` (domain) + +## What Value to Use + +Use the external system's native, stable identifier: + +| System | Use | +|--------|-----| +| Okta | `app.Id` | +| AWS | ARN | +| GCP | Project ID | +| GitHub | Repository ID (numeric) | +| Database | Primary key | + +**Properties of good IDs:** +- Stable (doesn't change when resource is renamed) +- Unique within the system +- Native to the external system + +## Common Mistakes + +**Using display names:** Names change. `engineering-team` might become `platform-team`. + +**Using composite keys:** If you construct `org/repo`, changes to either part break correlation. + +**Omitting RawId:** Without it, ConductorOne can't correlate resources across syncs. + +## Pre-sync Reservation + +The `match_baton_id` field (in Terraform/API) allows creating ConductorOne objects before the connector discovers them. When the connector syncs, resources are matched by this ID. diff --git a/rap/concepts-overview.md b/rap/concepts-overview.md new file mode 100644 index 0000000..dfbaf71 --- /dev/null +++ b/rap/concepts-overview.md @@ -0,0 +1,74 @@ +# concepts-overview + +What connectors do, sync vs provision, the reconciliation loop. + +--- + +## What Problem Connectors Solve + +A **connector** answers: *who has access to what?* + +ConductorOne needs visibility into users, groups, roles, and permissions across all systems. Every system stores this differently - Okta has users and groups, AWS has IAM roles and policies, Salesforce has profiles and permission sets. + +A connector translates access data from any system into a common format. Once connected, you get unified visibility across your infrastructure. + +## What a Connector Does + +In Baton terms, a connector is a program that can: +- **List resources** (users, groups, roles, apps, projects) +- **Define entitlements** (permissions that can be granted) +- **Emit grants** (who currently has which entitlements) + +## Sync vs Provision + +**Sync** (read): Pull access data into ConductorOne +- Who exists? What groups? What roles? +- What permissions are available? +- Who has what access right now? + +**Provision** (write): Push access changes back +- **Grant**: Give someone approved access +- **Revoke**: Remove terminated access +- **Create Account**: JIT provisioning +- **Delete Resource**: Remove accounts entirely + +## The Reconciliation Loop + +Together, sync and provision create a reconciliation loop: + +1. ConductorOne sees what access exists (sync) +2. Compares to what access *should* exist (policy) +3. Corrects any drift (provision) + +Access controls become self-healing. + +## Identity Providers + +IdPs (Okta, Azure AD, Google Workspace) have a unique role - they're the **source of truth** for user identities. + +IdP connectors: +- Define canonical user identities +- Enable correlation of users across other systems +- Originate user lifecycle (join, move, leave) + +Connecting an IdP establishes the identity foundation other connectors build upon. + +## The Connector Binary + +When you build a connector, you produce a standalone binary (e.g., `baton-okta`): + +- Embeds the SDK at compile time +- Self-contained, no runtime dependencies +- Produces a `.c1z` file as output + +```bash +./baton-okta --domain example.okta.com --api-token $TOKEN +ls sync.c1z # Output file +``` + +## When to Build vs Reuse + +- **Use existing connector** when it meets your needs +- **Contribute upstream** when missing a capability you need +- **Build new** when target system is unsupported +- **Use meta-connectors** (baton-http, baton-sql) for quick REST/SQL integration diff --git a/rap/concepts-resources.md b/rap/concepts-resources.md new file mode 100644 index 0000000..a788112 --- /dev/null +++ b/rap/concepts-resources.md @@ -0,0 +1,95 @@ +# concepts-resources + +Resources, entitlements, grants, traits, and the access graph. + +--- + +## The Access Graph + +Your connector produces an access graph with three node types: + +``` +RESOURCES ENTITLEMENTS GRANTS +Things that exist Permissions that Who has what + can be assigned + +Users, Groups, -> Admin, Read, -> Alice has Admin +Roles, Apps Member on Database X +``` + +## Resources + +Resources are things that exist in the target system: +- Users (individual accounts) +- Groups (collections of users) +- Roles (permission bundles) +- Apps (applications or services) +- Custom types (projects, repositories, teams) + +Each resource has a **resource type** with optional **traits**. + +## Traits + +Traits tell ConductorOne how to interpret a resource: + +| Trait | Use For | +|-------|---------| +| `TRAIT_USER` | Individual accounts | +| `TRAIT_GROUP` | Collections of users | +| `TRAIT_ROLE` | Permission sets | +| `TRAIT_APP` | Applications or services | +| `TRAIT_SECRET` | Credentials or tokens | + +```go +var userResourceType = &v2.ResourceType{ + Id: "user", + DisplayName: "User", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_USER}, +} +``` + +Marking a resource with `TRAIT_USER` enables: +- Correlation with users from other systems +- Display in user-centric views +- User-specific policies + +## Entitlements + +Entitlements define what can be granted. They attach to resources. + +```go +entitlement := &v2.Entitlement{ + Id: "member", + DisplayName: "Member", + Resource: groupResource, +} +``` + +One resource can offer multiple entitlements. A GitHub repository might offer: read, write, maintain, admin. + +## Grants + +Grants record who has what: + +```go +grant := &v2.Grant{ + Principal: aliceResource, // Who + Entitlement: memberEntitlement, // What permission on what resource +} +``` + +A grant connects a **principal** (usually a user) to an **entitlement**. + +## Hierarchical Resources + +Resources can have parent-child relationships: +- GitHub: organizations contain repositories +- AWS: accounts contain services +- GCP: projects contain resources + +Express hierarchy through `parentResourceID` in your `List()` method. The SDK calls `List()` with no parent first (top-level), then for each parent that has children. + +This enables: +- Access shown in context (this role on *this* project) +- Scoped access reviews (all access within one org unit) +- Inheritance patterns where they exist diff --git a/rap/concepts-sync.md b/rap/concepts-sync.md new file mode 100644 index 0000000..c4652c4 --- /dev/null +++ b/rap/concepts-sync.md @@ -0,0 +1,76 @@ +# concepts-sync + +The four sync stages, pagination, and how the SDK orchestrates calls. + +--- + +## The Four Stages + +The SDK orchestrates sync in four stages: + +``` +Stage 1: ResourceType() + SDK learns what resource types exist (user, group, role) + +Stage 2: List() + SDK fetches all instances of each type + Returns: 127 users, 23 groups, 15 roles + +Stage 3: Entitlements() + SDK asks each resource what entitlements it offers + Returns: group-A offers "member", role-X offers "assigned" + +Stage 4: Grants() + SDK discovers who has each entitlement + Returns: alice has "member" on group-A +``` + +## Inversion of Control + +The SDK processes ALL resource types together for each stage, not one type completely before the next. + +You define what to sync. The SDK decides when to call your methods. + +This keeps connector code focused on data transformation, not orchestration. + +## The ResourceSyncer Interface + +Four methods give you a working connector: + +```go +type ResourceSyncer interface { + ResourceType(ctx) (*v2.ResourceType, error) + List(ctx, parentResourceID, token) ([]*v2.Resource, string, annotations, error) + Entitlements(ctx, resource, token) ([]*v2.Entitlement, string, annotations, error) + Grants(ctx, resource, token) ([]*v2.Grant, string, annotations, error) +} +``` + +## Pagination + +Each method returns: +- A list of results +- A `nextPageToken` string (empty when done) +- Optional annotations + +The SDK handles pagination orchestration. You return items and the next token. + +**Key invariant:** Your next token must progress. The SDK detects and errors on "same token" loops. + +## The Sync Pipeline + +``` +External Connector .c1z Sync Domain +System -> (yours) -> File -> Service -> Objects + | + "Uplift" +``` + +1. **Fetch** - Your connector calls external API +2. **Transform** - Create Resource/Entitlement/Grant objects +3. **Output** - SDK writes to .c1z file (gzip SQLite) +4. **Ingest** - ConductorOne reads the file +5. **Uplift** - Raw records become domain objects + +**You control:** Steps 1-3 +**ConductorOne controls:** Steps 4-5 diff --git a/rap/debug-errors.md b/rap/debug-errors.md new file mode 100644 index 0000000..cbf337b --- /dev/null +++ b/rap/debug-errors.md @@ -0,0 +1,123 @@ +# debug-errors + +Common errors and their solutions. + +--- + +## Pagination Loop Detected + +**Error:** `pagination loop detected: same token returned` + +**Cause:** Your List/Grants method returned the same next token twice. + +**Fix:** +```go +// Wrong +return resources, currentToken, nil, nil + +// Right - return NEW token or empty string +if hasMore { + return resources, newToken, nil, nil +} +return resources, "", nil, nil +``` + +## Authentication Failures + +**Error:** `401 Unauthorized` or `403 Forbidden` + +**Causes:** +- Token expired +- Token lacks required scopes +- Wrong API endpoint + +**Fix:** +1. Verify token is valid: `curl -H "Authorization: Bearer $TOKEN" $API_URL` +2. Check required scopes in target system docs +3. Regenerate token with correct permissions + +## Rate Limiting + +**Error:** `429 Too Many Requests` + +**Cause:** Hitting API rate limits. + +**Fix:** The SDK's uhttp client handles retries automatically. If still failing: +- Reduce page size +- Add delays between requests +- Request rate limit increase from API provider + +## Empty Sync Results + +**Symptom:** Connector runs successfully but `baton resources` shows nothing. + +**Causes:** +- Credentials lack read permissions +- API returns empty due to filters +- Wrong base URL + +**Fix:** +1. Test API directly with curl +2. Check if filters in code exclude everything +3. Verify base URL matches environment (prod vs staging) + +## Connection Refused + +**Error:** `connection refused` or `no such host` + +**Causes:** +- Wrong hostname +- Firewall blocking connection +- Service is down + +**Fix:** +1. Verify URL is correct +2. Test connectivity: `curl -v $API_URL` +3. Check firewall/VPN requirements + +## Certificate Errors + +**Error:** `x509: certificate signed by unknown authority` + +**Cause:** Self-signed or internal CA certificate. + +**Fix for testing:** +```go +// Add to client setup +client.WithInsecureSkipVerify(true) +``` + +**Production fix:** Add CA certificate to trust store. + +## JSON Parsing Errors + +**Error:** `json: cannot unmarshal...` + +**Cause:** API response doesn't match expected structure. + +**Fix:** +1. Log raw response: `--log-level debug` +2. Compare actual response to struct definition +3. Handle optional/nullable fields + +## Missing Entitlements + +**Symptom:** Resources exist but no entitlements appear. + +**Cause:** Entitlements() method returns empty. + +**Fix:** Verify Entitlements() returns at least one entitlement for resources that should offer permissions. + +## Grants Not Appearing + +**Symptom:** Entitlements exist but no grants. + +**Causes:** +- Grants() not implemented +- Principal resource type doesn't exist +- Wrong entitlement ID in grant + +**Fix:** +1. Verify Grants() is called (add logging) +2. Check principal resource type matches a synced type +3. Verify entitlement ID matches what Entitlements() returns diff --git a/rap/debug-workflow.md b/rap/debug-workflow.md new file mode 100644 index 0000000..1ed97e4 --- /dev/null +++ b/rap/debug-workflow.md @@ -0,0 +1,96 @@ +# debug-workflow + +Step-by-step process for debugging connector issues. + +--- + +## 1. Run Locally First + +```bash +./baton-myservice --api-token "$TOKEN" -f sync.c1z +``` + +Check exit code. Non-zero means error. + +## 2. Inspect the Output + +```bash +# What resources were synced? +baton resources -f sync.c1z + +# What grants exist? +baton grants -f sync.c1z + +# What entitlements are available? +baton entitlements -f sync.c1z + +# Stats overview +baton stats -f sync.c1z +``` + +## 3. Increase Log Verbosity + +```bash +./baton-myservice --api-token "$TOKEN" --log-level debug -f sync.c1z +``` + +Debug logs show: +- API requests and responses +- Pagination state +- Resource counts per type + +## 4. Check for Common Issues + +**Zero resources?** +- Credentials may lack read permissions +- API endpoint may be wrong +- Filter may be excluding everything + +**Missing grants?** +- Check Grants() method is implemented +- Verify entitlements exist on resources +- Check pagination isn't truncating results + +**Pagination loop?** +- Your next token isn't progressing +- See error: "pagination loop detected" + +## 5. Test Individual Endpoints + +If the connector calls multiple APIs, test each: + +```bash +# Test with curl first +curl -H "Authorization: Bearer $TOKEN" \ + https://api.example.com/v1/users | jq . +``` + +## 6. Review Resource Mapping + +Resources may exist but not match expected types: + +```bash +# List specific resource type +baton resources -f sync.c1z --resource-type user + +# Look for unexpected types +baton resources -f sync.c1z | sort | uniq -c +``` + +## 7. Validate Against Production + +Compare local sync to production: +- Same resource counts? +- Same grant patterns? +- Any missing resource types? + +## Debugging Checklist + +| Symptom | Check | +|---------|-------| +| No resources | Credentials, API endpoint, filters | +| No grants | Grants() implementation, entitlements exist | +| Pagination loop | Token progression logic | +| Missing users | User list endpoint, status filters | +| Wrong counts | Page size, pagination completeness | +| Auth failures | Token validity, required scopes | diff --git a/rap/meta-cel.md b/rap/meta-cel.md new file mode 100644 index 0000000..cee0c3f --- /dev/null +++ b/rap/meta-cel.md @@ -0,0 +1,128 @@ +# meta-cel + +Data transformation with Common Expression Language in meta-connectors. + +--- + +## What is CEL + +CEL (Common Expression Language) is used in baton-http and baton-sql for: +- Field mapping from API/SQL responses +- Conditional logic +- String transformation +- Type conversion + +Think of it as a safer alternative to embedding code in config. + +## Basic Syntax + +```yaml +map: + # Direct field access + id: ".id" + + # String concatenation + display_name: ".first_name + ' ' + .last_name" + + # Ternary conditional + status: ".is_active ? 'enabled' : 'disabled'" + + # Null handling + last_login: ".last_login != null ? string(.last_login) : ''" +``` + +## Available Functions + +| Function | Purpose | Example | +|----------|---------|---------| +| `lowercase()` | To lowercase | `lowercase(.email)` | +| `uppercase()` | To uppercase | `uppercase(.code)` | +| `titlecase()` | Title case | `titlecase(.name)` | +| `trim()` | Remove whitespace | `trim(.value)` | +| `match()` | Regex match | `match(.email, ".*@corp\\.com")` | +| `extract()` | Regex extract | `extract(.urn, "user-([0-9]+)")` | +| `replace()` | String replace | `replace(.name, {"old": "_", "new": "-"})` | +| `get()` | Get with default | `get(.optional, "default")` | +| `has()` | Check field exists | `has(input.employee_id)` | +| `parse_json()` | Parse JSON string | `parse_json(.metadata).type` | +| `json_path()` | Extract from JSON | `json_path(.data, "user.name")` | +| `string()` | Convert to string | `string(.numeric_id)` | + +## Context Variables + +| Variable | Available In | Description | +|----------|--------------|-------------| +| `.column` or `item` | List, Grants | Current row/item | +| `resource` | Grants, Provisioning | Current resource | +| `principal` | Provisioning | User being modified | +| `entitlement` | Provisioning | Entitlement being modified | +| `input` | Account creation | User-provided values | +| `password` | Account creation | Generated password | + +## Common Patterns + +**Account type from field:** +```yaml +account_type: ".type == 'employee' ? 'human' : 'service'" +``` + +**Email from multiple possible fields:** +```yaml +emails: + - "has(.primary_email) ? .primary_email : .email" +``` + +**Status normalization:** +```yaml +status: ".status == 'active' || .status == 'enabled' ? 'enabled' : 'disabled'" +``` + +**Skip system accounts in grants:** +```yaml +grants: + - query: "SELECT * FROM role_members WHERE role_id = ?" + map: + - skip_if: ".account_type == 'system'" + principal_id: ".user_id" + principal_type: "user" + entitlement_id: "member" +``` + +**Build display name:** +```yaml +display_name: "titlecase(.first_name) + ' ' + titlecase(.last_name)" +``` + +**Handle optional nested field:** +```yaml +department: "has(.profile) && has(.profile.department) ? .profile.department : 'Unknown'" +``` + +## Provisioning Variables + +```yaml +provisioning: + vars: + user_id: "principal.ID" # Principal's resource ID + group_id: "resource.ID" # Target resource ID + email: "principal.DisplayName" # Can access other fields + timestamp: "now()" # Current timestamp + grant: + queries: + - "INSERT INTO members VALUES (?, ?, ?)" +``` + +## Type Coercion + +CEL is type-aware. Convert explicitly when needed: + +```yaml +# Integer to string +id: "string(.numeric_id)" + +# String to bool (if needed) +active: ".status == 'active'" + +# Handle null +value: ".field != null ? .field : ''" +``` diff --git a/rap/meta-http.md b/rap/meta-http.md new file mode 100644 index 0000000..c3a5147 --- /dev/null +++ b/rap/meta-http.md @@ -0,0 +1,147 @@ +# meta-http + +REST API integration via YAML configuration, no Go code required. + +--- + +## When to Use + +- Target system has REST/HTTP endpoints +- Access model is straightforward +- You want ops teams to maintain the integration +- Quick integration matters more than flexibility + +## Basic Structure + +```yaml +version: "1" +app_name: "My SaaS App" +app_description: "Syncs users and roles" + +connect: + base_url: "https://api.example.com/v1" + auth: + type: "bearer" + token: "${API_KEY}" + +resource_types: + user: + name: "User" + list: + request: + url: "/users" + method: "GET" + response: + items_path: "data.users" + map: + id: ".id" + display_name: ".name" + traits: + user: + emails: + - ".email" +``` + +## Authentication Types + +```yaml +# Bearer token +auth: + type: "bearer" + token: "${API_KEY}" + +# API key in header +auth: + type: "api_key" + header: "X-API-Key" + key: "${API_KEY}" + +# Basic auth +auth: + type: "basic" + username: "${USERNAME}" + password: "${PASSWORD}" + +# OAuth2 client credentials +auth: + type: "oauth2_client_credentials" + client_id: "${CLIENT_ID}" + client_secret: "${CLIENT_SECRET}" + token_url: "https://auth.example.com/oauth/token" +``` + +## Request Configuration + +```yaml +list: + request: + url: "/users" + method: "GET" + headers: + Accept: "application/json" + query_params: + status: "active" + response: + items_path: "data.users" # JSONPath to array +``` + +## URL Templates + +Dynamic URLs using Go templates: + +```yaml +grants: + - request: + url: "tmpl:/groups/{{.resource.id}}/members" + method: "GET" +``` + +Available variables: +- `.resource` - Current resource +- `.principal` - User for provisioning +- `.item` - Current item in iteration + +## Pagination + +```yaml +pagination: + strategy: "offset" + limit_param: "per_page" + offset_param: "page" + page_size: 100 + total_path: "meta.total" # Optional +``` + +Or cursor-based: +```yaml +pagination: + strategy: "cursor" + cursor_param: "cursor" + cursor_path: "meta.next_cursor" +``` + +## Response Mapping + +```yaml +map: + id: ".id" + display_name: ".attributes.name" + traits: + user: + emails: + - ".attributes.email" + status: ".attributes.active ? 'enabled' : 'disabled'" +``` + +## Running + +```bash +# Validate first +baton-http --config-path ./config.yaml --validate-config-only + +# One-shot sync +baton-http --config-path ./config.yaml -f sync.c1z + +# Inspect results +baton resources -f sync.c1z +``` diff --git a/rap/meta-sql.md b/rap/meta-sql.md new file mode 100644 index 0000000..90f085d --- /dev/null +++ b/rap/meta-sql.md @@ -0,0 +1,157 @@ +# meta-sql + +SQL database integration via YAML configuration. + +--- + +## Supported Databases + +- PostgreSQL (`postgres`) +- MySQL (`mysql`) +- SQL Server (`sqlserver`) +- Oracle (`oracle`) +- SAP HANA (`hana`) +- SQLite (`sqlite`) + +## Connection + +```yaml +connect: + scheme: "postgres" + host: "${DB_HOST}" + port: "5432" + database: "${DB_NAME}" + user: "${DB_USER}" + password: "${DB_PASS}" + params: + sslmode: "require" +``` + +Alternative DSN format: +```yaml +connect: + dsn: "postgres://${DB_HOST}:5432/${DB_NAME}?sslmode=require" + user: "${DB_USER}" + password: "${DB_PASS}" +``` + +## Listing Resources + +```yaml +resource_types: + user: + name: "User" + list: + query: | + SELECT id, username, email, status, department + FROM users + WHERE active = true + AND id > ? + ORDER BY id ASC + LIMIT ? + pagination: + strategy: "cursor" + primary_key: "id" + map: + id: ".id" + display_name: ".username" + traits: + user: + emails: + - ".email" + status: ".status == 'active' ? 'enabled' : 'disabled'" +``` + +## Query Placeholders + +- `?` - Current pagination cursor +- `?` - Page size +- `?` - Offset for offset-based pagination +- `?` - Current resource ID in grants query +- `?` - Named variable from `vars` block + +## Grants Discovery + +```yaml +grants: + - query: | + SELECT user_id, group_id + FROM group_members + WHERE group_id = ? + map: + - principal_id: ".user_id" + principal_type: "user" + entitlement_id: "member" + skip_if: ".user_type == 'system'" # Optional filter +``` + +## Provisioning + +```yaml +static_entitlements: + - id: "member" + display_name: "'Member'" + purpose: "assignment" + grantable_to: + - "user" + provisioning: + vars: + user_id: "principal.ID" + group_id: "resource.ID" + grant: + queries: + - | + INSERT INTO group_members (user_id, group_id) + VALUES (?, ?) + ON CONFLICT DO NOTHING + revoke: + queries: + - | + DELETE FROM group_members + WHERE user_id = ? AND group_id = ? +``` + +## Account Creation + +```yaml +account_provisioning: + schema: + - name: "username" + type: "string" + required: true + - name: "email" + type: "string" + required: true + + credentials: + random_password: + min_length: 16 + max_length: 32 + preferred: true + + create: + vars: + username: "input.username" + email: "input.email" + password: "password" + queries: + - | + INSERT INTO users (username, email, password_hash) + VALUES (?, ?, crypt(?, gen_salt('bf'))) +``` + +## Running + +```bash +# Validate +baton-sql --config-path ./config.yaml --validate-config-only + +# Sync +baton-sql --config-path ./config.yaml -f sync.c1z + +# With provisioning +baton-sql --config-path ./config.yaml \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" \ + --provisioning +``` diff --git a/rap/ops-modes.md b/rap/ops-modes.md new file mode 100644 index 0000000..4b2160e --- /dev/null +++ b/rap/ops-modes.md @@ -0,0 +1,100 @@ +# ops-modes + +One-shot vs daemon vs hosted mode. + +--- + +## Three Run Modes + +| Mode | Trigger | Behavior | +|------|---------|----------| +| **One-shot** | No `--client-id` | Run once, produce .c1z file, exit | +| **Daemon** | `--client-id` provided | Connect to C1, poll for tasks, run continuously | +| **Hosted** | ConductorOne infrastructure | Managed by ConductorOne, no local deployment | + +## One-Shot Mode + +Run once and produce a sync file: + +```bash +./baton-myservice \ + --api-token "$API_TOKEN" \ + -f sync.c1z + +# Inspect results +baton resources -f sync.c1z +baton grants -f sync.c1z +``` + +Use for: +- Local development and testing +- CI/CD pipelines +- Manual audits +- Debugging + +## Daemon Mode + +Connect to ConductorOne and process tasks continuously: + +```bash +./baton-myservice \ + --api-token "$API_TOKEN" \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" +``` + +The connector: +1. Authenticates to ConductorOne +2. Polls for sync/provisioning tasks +3. Executes tasks and reports results +4. Repeats until stopped + +Use for: +- Production deployments +- Automated syncs +- Provisioning workflows + +## Hosted Mode + +ConductorOne runs the connector for you: +- No infrastructure to manage +- Automatic updates +- Credentials stored in ConductorOne + +Check if your connector is available as hosted in the ConductorOne console. + +## Provisioning Flag + +Enable provisioning operations (grant/revoke/create/delete): + +```bash +./baton-myservice \ + --api-token "$API_TOKEN" \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" \ + --provisioning +``` + +Without `--provisioning`, the connector only syncs (read-only). + +## Environment Variables + +All flags have environment variable equivalents: + +| Flag | Environment Variable | +|------|---------------------| +| `--api-token` | `BATON_API_TOKEN` | +| `--client-id` | `BATON_CLIENT_ID` | +| `--client-secret` | `BATON_CLIENT_SECRET` | +| `--provisioning` | `BATON_PROVISIONING` | +| `--file` | `BATON_FILE` | +| `--log-level` | `BATON_LOG_LEVEL` | + +## Log Levels + +```bash +--log-level debug # Verbose, for troubleshooting +--log-level info # Default +--log-level warn # Warnings and errors only +--log-level error # Errors only +``` diff --git a/rap/provision-grant.md b/rap/provision-grant.md new file mode 100644 index 0000000..8f2a774 --- /dev/null +++ b/rap/provision-grant.md @@ -0,0 +1,145 @@ +# provision-grant + +GrantProvisionerV2 and RevokeProvisioner interfaces for Grant and Revoke operations. + +--- + +## The Interfaces + +```go +// V2 interface (recommended for new connectors) +type GrantProvisionerV2 interface { + Grant(ctx context.Context, principal *v2.Resource, entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) +} + +type RevokeProvisioner interface { + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} +``` + +## Implementation Pattern + +Add provisioning to your resource builder: + +```go +type groupBuilder struct { + client *MyServiceClient +} + +// Implement ResourceSyncer methods... + +func (b *groupBuilder) Grant( + ctx context.Context, + principal *v2.Resource, + entitlement *v2.Entitlement, +) ([]*v2.Grant, annotations.Annotations, error) { + groupID := entitlement.Resource.Id.Resource + userID := principal.Id.Resource + + err := b.client.AddGroupMember(ctx, groupID, userID) + if err != nil { + if isAlreadyExistsError(err) { + return nil, nil, nil // Idempotent success + } + return nil, nil, fmt.Errorf("failed to add member: %w", err) + } + + // Return created grant + grant := sdkGrant.NewGrant(entitlement.Resource, entitlement.Slug, principal.Id) + return []*v2.Grant{grant}, nil, nil +} + +func (b *groupBuilder) Revoke( + ctx context.Context, + grant *v2.Grant, +) (annotations.Annotations, error) { + groupID := grant.Entitlement.Resource.Id.Resource + userID := grant.Principal.Id.Resource + + err := b.client.RemoveGroupMember(ctx, groupID, userID) + if err != nil { + if isNotFoundError(err) { + return nil, nil // Already revoked + } + return nil, fmt.Errorf("failed to remove member: %w", err) + } + + return nil, nil +} +``` + +## Registering Provisioning + +In your connector's constructor: + +```go +func (c *Connector) ResourceSyncers(ctx context.Context) []connectorbuilder.ResourceSyncer { + return []connectorbuilder.ResourceSyncer{ + newUserBuilder(c.client), + newGroupBuilder(c.client), // Implements ResourceProvisionerV2 + } +} +``` + +The SDK detects if your builder implements `ResourceProvisionerV2` and enables provisioning automatically. + +## Multiple Entitlements + +Handle different entitlement types in a single resource: + +```go +func (b *groupBuilder) Grant( + ctx context.Context, + principal *v2.Resource, + entitlement *v2.Entitlement, +) ([]*v2.Grant, annotations.Annotations, error) { + groupID := entitlement.Resource.Id.Resource + userID := principal.Id.Resource + + switch entitlement.Slug { + case "member": + err := b.client.AddMember(ctx, groupID, userID) + // ... + case "admin": + err := b.client.AddAdmin(ctx, groupID, userID) + // ... + default: + return nil, nil, fmt.Errorf("unknown entitlement: %s", entitlement.Slug) + } + + grant := sdkGrant.NewGrant(entitlement.Resource, entitlement.Slug, principal.Id) + return []*v2.Grant{grant}, nil, nil +} +``` + +## Idempotency + +Provisioning operations should be idempotent: + +```go +// Good: handles already-exists gracefully +err := b.client.AddGroupMember(ctx, groupID, userID) +if err != nil && !isAlreadyMemberError(err) { + return nil, err +} +return nil, nil + +// Good: handles not-found gracefully +err := b.client.RemoveGroupMember(ctx, groupID, userID) +if err != nil && !isNotFoundError(err) { + return nil, err +} +return nil, nil +``` + +## Running with Provisioning + +The connector must be started with `--provisioning` flag: + +```bash +./baton-myservice \ + --api-token "$TOKEN" \ + --client-id "$C1_CLIENT_ID" \ + --client-secret "$C1_CLIENT_SECRET" \ + --provisioning +``` diff --git a/rap/publish-submit.md b/rap/publish-submit.md new file mode 100644 index 0000000..9ccb74c --- /dev/null +++ b/rap/publish-submit.md @@ -0,0 +1,160 @@ +# publish-submit + +Publishing connectors to the connector registry. + +--- + +## Publishing flow + +``` +1. Create Connector --> Registry +2. Create Version --> Registry +3. Upload Binaries --> Registry +4. Finalize Version --> Registry --> Connector Hub +``` + +--- + +## Version states + +``` +PENDING -> UPLOADING -> VALIDATING -> PUBLISHED + | + +-> FAILED +PUBLISHED -> YANKED +``` + +| State | Meaning | +|-------|---------| +| PENDING | Version created, awaiting uploads | +| UPLOADING | Assets being uploaded | +| VALIDATING | Validation in progress | +| PUBLISHED | Available for download | +| YANKED | Withdrawn (still visible) | +| FAILED | Validation failed | + +--- + +## Release manifest + +| Field | Purpose | +|-------|---------| +| org | Organization identifier | +| name | Connector name | +| version | Semantic version | +| description | Human-readable description | +| repository_url | Source code repo | +| license | License identifier | +| assets | Platform binaries | +| commit_sha | Git commit | + +--- + +## Asset structure + +| Field | Purpose | +|-------|---------| +| platform | Target (e.g., `linux-amd64`) | +| filename | Binary filename | +| size_bytes | File size | +| sha256 | Checksum | +| download_url | Download location | +| signature_url | Detached signature | + +--- + +## Supported platforms + +| Platform | Description | +|----------|-------------| +| darwin-amd64 | macOS Intel | +| darwin-arm64 | macOS Apple Silicon | +| linux-amd64 | Linux x86_64 | +| linux-arm64 | Linux ARM64 | +| windows-amd64 | Windows x86_64 | + +--- + +## Publishing paths + +### Contributing upstream (recommended) + +1. Fork existing connector repo +2. Add changes +3. Submit PR +4. Maintainers review and merge +5. New version published + +Benefits: community benefits, maintainers handle publishing. + +### Fork and maintain + +1. Fork repository +2. Publish under your org +3. Maintain independently + +Trade-off: Full control but ongoing maintenance burden. + +### Internal only + +1. Don't publish to public registry +2. Deploy in daemon mode on your infrastructure +3. Suitable for proprietary systems + +--- + +## Signing + +Connectors can be signed with: +- GPG signatures +- Cosign (sigstore) + +Signatures verified during download. + +--- + +## Pre-submission checklist + +Before publishing: + +- [ ] Connector syncs correctly +- [ ] Required permissions documented +- [ ] README complete +- [ ] No credentials in code +- [ ] License file present +- [ ] Tests pass +- [ ] Lint passes +- [ ] Builds for all platforms + +--- + +## Versioning + +Use semantic versioning: + +| Version | When | +|---------|------| +| Major (1.0 -> 2.0) | Breaking changes | +| Minor (1.0 -> 1.1) | New features, backward compatible | +| Patch (1.0.0 -> 1.0.1) | Bug fixes | + +--- + +## CI/CD + +Most connectors use GitHub Actions: + +```yaml +on: + push: + tags: + - 'v*' + +jobs: + release: + # Build all platforms + # Upload to registry + # Sign binaries +``` + +Tag push triggers release workflow. diff --git a/rap/recipes-auth.md b/rap/recipes-auth.md new file mode 100644 index 0000000..8ae0004 --- /dev/null +++ b/rap/recipes-auth.md @@ -0,0 +1,139 @@ +# recipes-auth + +Authentication patterns for connector API clients. + +--- + +## API key (Bearer token) + +```go +import "github.com/conductorone/baton-sdk/pkg/uhttp" + +func NewClient(ctx context.Context, apiKey string) (*Client, error) { + httpClient, err := uhttp.NewBaseHttpClient(ctx, + uhttp.WithBearerToken(apiKey)) + if err != nil { + return nil, err + } + return &Client{http: httpClient, baseURL: "https://api.example.com"}, nil +} +``` + +`WithBearerToken` sets `Authorization: Bearer `. The SDK handles retries and rate limiting. + +--- + +## OAuth2 client credentials + +```go +import "golang.org/x/oauth2/clientcredentials" + +func NewClient(ctx context.Context, clientID, clientSecret, tokenURL string) (*Client, error) { + config := &clientcredentials.Config{ + ClientID: clientID, + ClientSecret: clientSecret, + TokenURL: tokenURL, + Scopes: []string{"read", "write"}, + } + httpClient := config.Client(ctx) // Auto-refreshes tokens + return &Client{http: httpClient}, nil +} +``` + +The `clientcredentials` package handles token refresh automatically. + +--- + +## JWT service account (Google-style) + +```go +import ( + "google.golang.org/api/option" + admin "google.golang.org/api/admin/directory/v1" +) + +func NewGoogleClient(ctx context.Context, credentialsJSON []byte, adminEmail string) (*admin.Service, error) { + config, err := google.JWTConfigFromJSON(credentialsJSON, + admin.AdminDirectoryUserReadonlyScope, + admin.AdminDirectoryGroupReadonlyScope, + ) + if err != nil { + return nil, fmt.Errorf("failed to parse credentials: %w", err) + } + config.Subject = adminEmail // Impersonate domain admin + return admin.NewService(ctx, option.WithHTTPClient(config.Client(ctx))) +} +``` + +Domain-wide delegation requires `Subject` to specify which user to impersonate. + +--- + +## LDAP bind + +```go +import "github.com/go-ldap/ldap/v3" + +func NewLDAPClient(ctx context.Context, serverURL, bindDN, bindPassword string) (*ldap.Conn, error) { + conn, err := ldap.DialURL(serverURL) // ldaps://dc.example.com:636 + if err != nil { + return nil, fmt.Errorf("failed to connect to LDAP: %w", err) + } + err = conn.Bind(bindDN, bindPassword) + if err != nil { + conn.Close() + return nil, fmt.Errorf("failed to bind: %w", err) + } + return conn, nil +} +``` + +LDAP requires binding before queries. Use `ldaps://` (port 636) for TLS. + +--- + +## Basic auth + +```go +func NewClient(ctx context.Context, username, password string) (*Client, error) { + httpClient, err := uhttp.NewBaseHttpClient(ctx, + uhttp.WithBasicAuth(username, password)) + if err != nil { + return nil, err + } + return &Client{http: httpClient}, nil +} +``` + +--- + +## Custom header auth + +Some APIs use non-standard headers like `X-API-Key`: + +```go +func NewClient(ctx context.Context, apiKey string) (*Client, error) { + httpClient, err := uhttp.NewBaseHttpClient(ctx) + if err != nil { + return nil, err + } + // Add custom header to all requests + httpClient.Transport = &headerTransport{ + base: httpClient.Transport, + header: "X-API-Key", + value: apiKey, + } + return &Client{http: httpClient}, nil +} + +type headerTransport struct { + base http.RoundTripper + header string + value string +} + +func (t *headerTransport) RoundTrip(req *http.Request) (*http.Response, error) { + req.Header.Set(t.header, t.value) + return t.base.RoundTrip(req) +} +``` diff --git a/rap/recipes-caching.md b/rap/recipes-caching.md new file mode 100644 index 0000000..de7d444 --- /dev/null +++ b/rap/recipes-caching.md @@ -0,0 +1,192 @@ +# recipes-caching + +Caching patterns and anti-patterns for connectors. + +--- + +## When to cache + +| Scenario | Cache? | Reason | +|----------|--------|--------| +| Grants() needs user details from List() | Yes | Avoids N+1 API calls | +| Entitlements() needs role definitions | Yes | Role metadata is stable | +| List() needs parent context | Maybe | Often passed via parentID | +| Any data across sync runs | No | Stale data causes drift | + +Do NOT cache across sync runs. Connector restarts clear caches anyway. + +--- + +## Thread-safe caching with sync.Map + +```go +type Connector struct { + client *client.Client + userCache sync.Map // map[userID]User - struct field, not package-level +} + +// Populate during List() +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, next, err := u.client.ListUsers(ctx, pToken.Token) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range users { + u.connector.userCache.Store(user.ID, user) // Cache for later + r, _ := resource.NewUserResource(user.Name, userResourceType, user.ID) + resources = append(resources, r) + } + return resources, next, nil, nil +} + +// Use during Grants() +func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) { + + memberIDs, _ := g.client.GetGroupMemberIDs(ctx, resource.Id.Resource) + + var grants []*v2.Grant + for _, memberID := range memberIDs { + if cached, ok := g.connector.userCache.Load(memberID); ok { + user := cached.(User) + // Use cached user data + } + } + return grants, "", nil, nil +} +``` + +--- + +## ANTI-PATTERN: Package-level caches + +**Critical bug pattern.** Package-level `sync.Map` persists across syncs in daemon mode. + +```go +// WRONG - package-level cache persists across syncs +var userCache sync.Map + +func lookupUser(id string) (*User, bool) { + if cached, ok := userCache.Load(id); ok { + return cached.(*User), true + } + return nil, false +} +``` + +What goes wrong: +1. Sync 1 caches users A, B, C +2. User B deleted from target system +3. Sync 2 runs (same process in daemon mode) +4. Cache still has user B +5. Grants reference deleted user +6. Access reviews show phantom access + +**Correct pattern - struct-scoped:** + +```go +type Connector struct { + client *client.Client + userCache sync.Map // Fresh per connector instance +} + +func New(ctx context.Context, client *client.Client) *Connector { + return &Connector{ + client: client, + userCache: sync.Map{}, // Fresh cache + } +} +``` + +Find violations: +```bash +grep -r "^var.*sync\.Map" pkg/ +``` + +--- + +## Memory-bounded caching + +For large datasets, use LRU cache: + +```go +import "github.com/hashicorp/golang-lru/v2" + +type Connector struct { + userCache *lru.Cache[string, User] +} + +func New(ctx context.Context) (*Connector, error) { + cache, err := lru.New[string, User](10000) // Max 10k entries + if err != nil { + return nil, err + } + return &Connector{userCache: cache}, nil +} +``` + +Alternative for very large datasets - batch lookups instead of caching: + +```go +func (c *Connector) lookupUsers(ctx context.Context, ids []string) (map[string]User, error) { + return c.client.GetUsersByIDs(ctx, ids) // Single batch API call +} +``` + +--- + +## Cache lifetime in daemon mode + +| Mode | Cache Lifetime | Risk | +|------|----------------|------| +| One-shot (CLI) | Process lifetime | Low - exits after sync | +| Daemon mode | Must be managed | High - stale data persists | + +Clear caches at sync boundaries: + +```go +type Connector struct { + client *client.Client + userCache sync.Map +} + +func (c *Connector) PrepareForSync(ctx context.Context) error { + c.userCache = sync.Map{} // Clear from previous sync + return nil +} +``` + +Time-based invalidation for long syncs: + +```go +type Connector struct { + userCache sync.Map + cachePopulatedAt time.Time +} + +func (c *Connector) getCachedUser(id string) (*User, bool) { + if time.Since(c.cachePopulatedAt) > 5*time.Minute { + c.userCache = sync.Map{} + c.cachePopulatedAt = time.Now() + } + if cached, ok := c.userCache.Load(id); ok { + return cached.(*User), true + } + return nil, false +} +``` + +--- + +## Cache expectations by mode + +| Scenario | Expected Behavior | +|----------|-------------------| +| CLI one-shot | Cache lives for single sync, process exits | +| Daemon between syncs | Cache cleared before each sync | +| Daemon during sync | Cache valid for sync duration | +| Long sync (>5 min) | Consider time-based invalidation | diff --git a/rap/recipes-errors.md b/rap/recipes-errors.md new file mode 100644 index 0000000..cf000ae --- /dev/null +++ b/rap/recipes-errors.md @@ -0,0 +1,158 @@ +# recipes-errors + +Error handling patterns for connectors. + +--- + +## Distinguish retryable from fatal errors + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.ListUsers(ctx) + if err != nil { + if isRateLimitError(err) || isNetworkError(err) { + return nil, "", nil, err // SDK retries automatically + } + if isAuthError(err) { + return nil, "", nil, fmt.Errorf("baton-example: authentication failed (check credentials): %w", err) + } + return nil, "", nil, err + } + // ... +} + +func isRateLimitError(err error) bool { + var httpErr *HTTPError + if errors.As(err, &httpErr) { + return httpErr.StatusCode == 429 + } + return false +} +``` + +SDK handles retries for transient errors. Fatal errors need clear messages. + +--- + +## Context cancellation in loops + +```go +func (u *userBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + users, err := u.client.ListUsers(ctx) + if err != nil { + return nil, "", nil, err + } + + var resources []*v2.Resource + for _, user := range users { + select { + case <-ctx.Done(): + return nil, "", nil, ctx.Err() + default: + } + r, err := resource.NewUserResource(user.Name, userResourceType, user.ID) + if err != nil { + return nil, "", nil, err + } + resources = append(resources, r) + } + return resources, "", nil, nil +} +``` + +Check `ctx.Done()` in loops processing many items. Cancelled context means stop immediately. + +--- + +## Error prefix convention + +Prefix errors with connector name for debugging: + +```go +return nil, "", nil, fmt.Errorf("baton-example: failed to list users: %w", err) +``` + +Pattern: `baton-: : %w` + +--- + +## Wrapping vs returning errors + +**Wrap** when adding context: +```go +if err != nil { + return fmt.Errorf("failed to parse user %s: %w", userID, err) +} +``` + +**Return directly** when no additional context helps: +```go +if err != nil { + return err +} +``` + +--- + +## HTTP status code handling + +| Status | Meaning | Action | +|--------|---------|--------| +| 400 | Bad request | Fatal - log request details | +| 401 | Unauthorized | Fatal - check credentials | +| 403 | Forbidden | Fatal - check permissions | +| 404 | Not found | Usually skip, not error | +| 429 | Rate limited | Retry (SDK handles) | +| 500+ | Server error | Retry (SDK handles) | + +```go +func handleResponse(resp *http.Response) error { + switch resp.StatusCode { + case http.StatusOK, http.StatusCreated: + return nil + case http.StatusNotFound: + return nil // Resource doesn't exist, not an error + case http.StatusUnauthorized: + return fmt.Errorf("baton-example: unauthorized (check API key)") + case http.StatusForbidden: + return fmt.Errorf("baton-example: forbidden (check permissions)") + default: + if resp.StatusCode >= 500 { + return fmt.Errorf("server error: %d", resp.StatusCode) // Will retry + } + return fmt.Errorf("unexpected status: %d", resp.StatusCode) + } +} +``` + +--- + +## Partial success handling + +When processing multiple items, decide strategy upfront: + +**Fail fast** (default for sync): +```go +for _, item := range items { + if err := process(item); err != nil { + return err // Stop on first error + } +} +``` + +**Collect errors** (for provisioning): +```go +var errs []error +for _, item := range items { + if err := process(item); err != nil { + errs = append(errs, fmt.Errorf("item %s: %w", item.ID, err)) + } +} +if len(errs) > 0 { + return errors.Join(errs...) +} +``` diff --git a/rap/recipes-modeling.md b/rap/recipes-modeling.md new file mode 100644 index 0000000..432050e --- /dev/null +++ b/rap/recipes-modeling.md @@ -0,0 +1,184 @@ +# recipes-modeling + +Resource modeling patterns for connectors. + +--- + +## Parent-child hierarchies + +Model resources within other resources (repos in orgs, projects in accounts): + +```go +// Parent type declares children +var orgResourceType = &v2.ResourceType{ + Id: "organization", + DisplayName: "Organization", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_GROUP}, +} + +var projectResourceType = &v2.ResourceType{ + Id: "project", + DisplayName: "Project", + Traits: []v2.ResourceType_Trait{v2.ResourceType_TRAIT_GROUP}, +} + +// Org List() declares child relationship +func (o *orgBuilder) List(ctx context.Context, _ *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + orgs, _ := o.client.ListOrgs(ctx) + var resources []*v2.Resource + for _, org := range orgs { + r, _ := resource.NewResource(org.Name, orgResourceType, org.ID, + resource.WithAnnotation(&v2.ChildResourceType{ + ResourceTypeId: projectResourceType.Id, + }), + ) + resources = append(resources, r) + } + return resources, "", nil, nil +} + +// Project List() receives parent ID +func (p *projectBuilder) List(ctx context.Context, parentID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) { + + if parentID == nil { + return nil, "", nil, nil // Projects only exist within orgs + } + projects, _ := p.client.ListProjects(ctx, parentID.Resource) + var resources []*v2.Resource + for _, proj := range projects { + r, _ := resource.NewResource(proj.Name, projectResourceType, proj.ID, + resource.WithParentResourceID(parentID), + ) + resources = append(resources, r) + } + return resources, "", nil, nil +} +``` + +SDK calls child `List()` once per parent, passing `parentID`. + +--- + +## Deep hierarchies (3+ levels) + +AWS example: Account -> Region -> Resource + +```go +// Each level declares its children +accountType // WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "region"}) +regionType // WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "ec2_instance"}) +instanceType // Leaf node, no children +``` + +SDK traverses depth-first. Each level receives parent context. + +--- + +## User traits + +```go +r, _ := resource.NewUserResource( + user.DisplayName, + userResourceType, + user.ID, + resource.WithEmail(user.Email, true), // true = primary + resource.WithUserLogin(user.Username), + resource.WithStatus(v2.UserTrait_Status_STATUS_ENABLED), + resource.WithAccountType(v2.UserTrait_ACCOUNT_TYPE_HUMAN), +) +``` + +Common traits: +- `WithEmail(email, isPrimary)` - User's email +- `WithUserLogin(login)` - Username/login ID +- `WithStatus(status)` - ENABLED, DISABLED, DELETED +- `WithAccountType(type)` - HUMAN, SERVICE, SYSTEM + +--- + +## Group traits + +```go +r, _ := resource.NewGroupResource( + group.Name, + groupResourceType, + group.ID, + resource.WithGroupProfile(group.Description), +) +``` + +--- + +## Role traits + +```go +r, _ := resource.NewRoleResource( + role.Name, + roleResourceType, + role.ID, + resource.WithRoleProfile(role.Description), +) +``` + +--- + +## Custom resource types + +For resources that aren't users, groups, or roles: + +```go +var repositoryResourceType = &v2.ResourceType{ + Id: "repository", + DisplayName: "Repository", + Traits: []v2.ResourceType_Trait{}, // No standard trait +} + +r, _ := resource.NewResource( + repo.Name, + repositoryResourceType, + repo.ID, + // Add custom annotations as needed +) +``` + +--- + +## Entitlement patterns + +**Permission entitlements** (on a single resource): +```go +entitlement.NewPermissionEntitlement( + resource, // The resource this permission applies to + "admin", // Permission slug + entitlement.WithDisplayName("Administrator"), + entitlement.WithDescription("Full administrative access"), + entitlement.WithGrantableTo(userResourceType), +) +``` + +**Membership entitlements** (belonging to a group): +```go +entitlement.NewAssignmentEntitlement( + groupResource, + "member", + entitlement.WithDisplayName("Member"), + entitlement.WithGrantableTo(userResourceType), +) +``` + +--- + +## Grant patterns + +```go +grant.NewGrant( + resource, // Resource the entitlement belongs to + "admin", // Entitlement slug + principalID, // Who has the grant (usually user resource ID) +) +``` + +Principal is typically a `*v2.ResourceId` from a user resource. diff --git a/rap/recipes-testing.md b/rap/recipes-testing.md new file mode 100644 index 0000000..a46fd0c --- /dev/null +++ b/rap/recipes-testing.md @@ -0,0 +1,215 @@ +# recipes-testing + +Testing patterns for connectors. + +--- + +## Configurable base URL (required) + +Every connector must support configurable base URL for testing: + +```go +// cmd/baton-example/main.go +fields := []field.SchemaField{ + field.StringField("api-key", + field.WithRequired(true), + field.WithDescription("API key for authentication"), + ), + field.StringField("base-url", + field.WithDefaultValue("https://api.example.com"), + field.WithDescription("Base URL (use http://localhost:8080 for testing)"), + ), +} + +// pkg/client/client.go +func New(cfg *Config) *Client { + baseURL := cfg.BaseURL + if baseURL == "" { + baseURL = "https://api.example.com" + } + return &Client{baseURL: baseURL} +} +``` + +Usage: +```bash +./baton-example --api-key $KEY # Production +./baton-example --api-key test --base-url http://localhost:8080 # Testing +``` + +Without configurable base URL, you cannot test without hitting production. + +--- + +## Insecure TLS for mock servers + +For mock servers with self-signed certificates: + +```go +type Config struct { + APIKey string `mapstructure:"api-key"` + BaseURL string `mapstructure:"base-url"` + Insecure bool `mapstructure:"insecure"` +} + +func New(ctx context.Context, cfg *Config) (*Client, error) { + opts := []uhttp.Option{} + if cfg.Insecure { + transport := &http.Transport{ + TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, + } + opts = append(opts, uhttp.WithTransport(transport)) + } + httpClient, err := uhttp.NewBaseHttpClient(ctx, opts...) + if err != nil { + return nil, err + } + return &Client{http: httpClient, baseURL: cfg.BaseURL}, nil +} +``` + +Usage: +```bash +./baton-example --api-key test --base-url https://localhost:8443 --insecure +``` + +--- + +## Unit testing resource builders + +```go +func TestUserBuilder_List(t *testing.T) { + // Mock client + mockClient := &MockClient{ + users: []User{ + {ID: "u1", Name: "Alice", Email: "alice@example.com"}, + {ID: "u2", Name: "Bob", Email: "bob@example.com"}, + }, + } + + builder := &userBuilder{client: mockClient} + resources, nextToken, _, err := builder.List(context.Background(), nil, &pagination.Token{}) + + require.NoError(t, err) + require.Len(t, resources, 2) + require.Empty(t, nextToken) + + assert.Equal(t, "u1", resources[0].Id.Resource) + assert.Equal(t, "Alice", resources[0].DisplayName) +} +``` + +--- + +## Integration testing with mock server + +```go +func TestConnector_Integration(t *testing.T) { + // Start mock server + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/users": + json.NewEncoder(w).Encode(map[string]any{ + "users": []map[string]string{ + {"id": "u1", "name": "Alice"}, + }, + }) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + defer server.Close() + + // Create connector pointing to mock + cfg := &Config{ + APIKey: "test-key", + BaseURL: server.URL, + } + connector, err := NewConnector(context.Background(), cfg) + require.NoError(t, err) + + // Test sync + // ... +} +``` + +--- + +## Testing pagination + +```go +func TestUserBuilder_Pagination(t *testing.T) { + mockClient := &MockClient{ + pages: map[string][]User{ + "": {{ID: "u1"}, {ID: "u2"}}, // First page + "page2": {{ID: "u3"}, {ID: "u4"}}, // Second page + "page3": {{ID: "u5"}}, // Last page + }, + nextTokens: map[string]string{ + "": "page2", + "page2": "page3", + "page3": "", // Empty = no more pages + }, + } + + builder := &userBuilder{client: mockClient} + + // First page + r1, next1, _, _ := builder.List(ctx, nil, &pagination.Token{Token: ""}) + assert.Len(t, r1, 2) + assert.Equal(t, "page2", next1) + + // Second page + r2, next2, _, _ := builder.List(ctx, nil, &pagination.Token{Token: "page2"}) + assert.Len(t, r2, 2) + assert.Equal(t, "page3", next2) + + // Last page + r3, next3, _, _ := builder.List(ctx, nil, &pagination.Token{Token: "page3"}) + assert.Len(t, r3, 1) + assert.Empty(t, next3) +} +``` + +--- + +## Testing grants + +```go +func TestGroupBuilder_Grants(t *testing.T) { + mockClient := &MockClient{ + groupMembers: map[string][]string{ + "g1": {"u1", "u2"}, + }, + } + + builder := &groupBuilder{client: mockClient} + groupResource := makeGroupResource("g1", "Admins") + + grants, _, _, err := builder.Grants(ctx, groupResource, &pagination.Token{}) + + require.NoError(t, err) + require.Len(t, grants, 2) + assert.Equal(t, "u1", grants[0].Principal.Id.Resource) + assert.Equal(t, "u2", grants[1].Principal.Id.Resource) +} +``` + +--- + +## Makefile test targets + +```makefile +.PHONY: test +test: + go test -v ./... + +.PHONY: test-integration +test-integration: + go test -v -tags=integration ./... + +.PHONY: test-coverage +test-coverage: + go test -coverprofile=coverage.out ./... + go tool cover -html=coverage.out +``` diff --git a/rap/ref-c1api.md b/rap/ref-c1api.md new file mode 100644 index 0000000..4fdf64e --- /dev/null +++ b/rap/ref-c1api.md @@ -0,0 +1,184 @@ +# ref-c1api + +How connectors communicate with ConductorOne platform. SDK handles this; understanding helps debugging. + +--- + +## Architecture + +``` +ConductorOne Platform Connector (Daemon Mode) ++---------------------+ +------------------------+ +| Task Queue |<--- gRPC ---------| BatonServiceClient | +| (Sync, Grant, etc) |--- Task --------->| Task Handler | +| Upload Service |<--- c1z upload ---| ConnectorBuilderV2 | +| Heartbeat |<--- keepalive ----| HeartbeatLoop | ++---------------------+ +------------------------+ +``` + +- Connectors poll for tasks (pull model) +- Results upload via streaming gRPC (512KB chunks) +- Heartbeats keep tasks alive during long operations + +--- + +## BatonServiceClient interface + +```go +type BatonServiceClient interface { + Hello(ctx, req) (*HelloResponse, error) // Initial handshake + GetTask(ctx, req) (*GetTaskResponse, error) // Poll for work + Heartbeat(ctx, req) (*HeartbeatResponse, error) // Keep task alive + FinishTask(ctx, req) (*FinishTaskResponse, error) // Report completion + Upload(ctx, task, reader) error // Upload c1z file +} +``` + +--- + +## Task types + +### SyncTask + +```go +type SyncTask struct { + SyncId string // Unique identifier + ResourceTypes []string // Types to sync (empty = all) + ResourceIds []string // Specific resources (targeted) + SkipEntitlements bool + SkipGrants bool +} +``` + +Produces: c1z file uploaded to C1 + +### GrantTask + +```go +type GrantTask struct { + Entitlement *v2.Entitlement // What to grant + Principal *v2.Resource // Who receives it +} +``` + +Calls: Your `Grant()` implementation + +### RevokeTask + +```go +type RevokeTask struct { + Grant *v2.Grant // Grant to revoke +} +``` + +Calls: Your `Revoke()` implementation + +### CreateAccountTask + +```go +type CreateAccountTask struct { + AccountInfo *v2.AccountInfo + CredentialOptions *v2.LocalCredentialOptions +} +``` + +Calls: Your `CreateAccount()` implementation + +### DeleteResourceTask + +```go +type DeleteResourceTask struct { + ResourceId *v2.ResourceId + ParentResourceId *v2.ResourceId +} +``` + +Calls: Your `Delete()` implementation + +### RotateCredentialTask + +```go +type RotateCredentialTask struct { + ResourceId *v2.ResourceId + CredentialOptions *v2.LocalCredentialOptions +} +``` + +Calls: Your `Rotate()` implementation + +--- + +## Task lifecycle + +``` +Connector C1 Platform + |--- Hello (identify) -------------->| + |<-- HelloResponse (config) ---------| + | | + |--- GetTask (poll) ---------------->| + |<-- Task or NoTask -----------------| + | | + [If task] | + |--- Heartbeat (every 30s) --------->| + | | + [Process task] | + | | + |--- FinishTask (result) ----------->| +``` + +--- + +## Heartbeat + +Tasks have 60-second default timeout. Connector sends heartbeats every 30 seconds during processing. + +```go +// SDK handles automatically +go func() { + ticker := time.NewTicker(30 * time.Second) + for range ticker.C { + client.Heartbeat(ctx, &HeartbeatRequest{TaskId: task.Id}) + } +}() +``` + +Missing heartbeats = task considered failed, may be reassigned. + +--- + +## c1z upload + +Sync results upload via streaming: + +```go +// SDK handles chunking (512KB) +err := client.Upload(ctx, task, c1zFileReader) +``` + +Large syncs may take several minutes to upload. Heartbeats continue during upload. + +--- + +## Authentication + +Daemon mode uses OAuth2 client credentials: + +```bash +./baton-example --client-id $CLIENT_ID --client-secret $CLIENT_SECRET +``` + +SDK exchanges credentials for access token, refreshes automatically. + +--- + +## Error handling + +| Error | SDK Behavior | +|-------|--------------| +| Network timeout | Retry with backoff | +| 401 Unauthorized | Refresh token, retry | +| 429 Rate limited | Backoff, retry | +| Task timeout | Task marked failed | +| Upload failure | Retry upload | + +Connector code should return errors; SDK handles retry logic. diff --git a/rap/ref-config.md b/rap/ref-config.md new file mode 100644 index 0000000..6c6aa83 --- /dev/null +++ b/rap/ref-config.md @@ -0,0 +1,202 @@ +# ref-config + +Configuration system reference for baton connectors. + +--- + +## Precedence + +1. CLI flags (`--domain example.com`) +2. Environment variables (`BATON_DOMAIN=example.com`) +3. Config files (YAML) + +CLI wins over env, env wins over file. + +--- + +## Standard flags (all connectors) + +### Output & logging + +| Flag | Default | Description | +|------|---------|-------------| +| `--file`, `-f` | `sync.c1z` | Output file path | +| `--log-level` | `info` | `debug`, `info`, `warn`, `error` | +| `--log-format` | auto | `json` or `console` | + +### Daemon mode + +| Flag | Description | +|------|-------------| +| `--client-id` | C1 OAuth client ID (enables daemon mode) | +| `--client-secret` | C1 OAuth client secret | +| `--skip-full-sync` | Disable full sync in daemon mode | + +### Provisioning + +| Flag | Description | +|------|-------------| +| `--provisioning` | Enable provisioning mode | +| `--grant-entitlement` | Entitlement ID to grant | +| `--grant-principal` | Resource ID receiving grant | +| `--revoke-grant` | Grant ID to revoke | + +### Account management + +| Flag | Description | +|------|-------------| +| `--create-account-login` | Login for new account | +| `--create-account-email` | Email for new account | +| `--delete-resource` | Resource ID to delete | +| `--rotate-credentials` | Resource ID for rotation | + +### Targeted sync + +| Flag | Description | +|------|-------------| +| `--sync-resources` | Specific resource IDs to sync | +| `--sync-resource-types` | Resource types to sync | +| `--skip-entitlements-and-grants` | Skip E&G during sync | + +--- + +## Environment variables + +All flags map to `BATON_` prefixed env vars: + +```bash +--domain -> BATON_DOMAIN +--api-key -> BATON_API_KEY +--base-dn -> BATON_BASE_DN +--client-id -> BATON_CLIENT_ID +``` + +Rules: +- Prefix: `BATON_` +- Dashes become underscores +- Case insensitive + +--- + +## Config file + +```yaml +# ~/.baton/config.yaml +domain: example.okta.com +api-token: "00abc123..." +log-level: debug + +# Arrays +skip-groups: + - "Test Group" + - "Temp Users" +``` + +Checked in order: +1. `--config` flag path +2. `./baton.yaml` +3. `~/.baton/config.yaml` + +--- + +## Field types (for connector authors) + +### StringField + +```go +field.StringField("domain", + field.WithRequired(true), + field.WithDescription("Your Okta domain"), +) +``` + +### BoolField + +```go +field.BoolField("ldaps", + field.WithDefaultValue(false), +) +``` + +### IntField + +```go +field.IntField("port", + field.WithDefaultValue(389), +) +``` + +### StringSliceField + +```go +field.StringSliceField("skip-groups", + field.WithDescription("Groups to exclude"), +) +``` + +CLI: `--skip-groups "Group1" --skip-groups "Group2"` +Env: `BATON_SKIP_GROUPS="Group1,Group2"` + +### SelectField (enum) + +```go +field.SelectField("auth-type", []string{"token", "oauth", "basic"}, + field.WithDefaultValue("token"), +) +``` + +--- + +## Field options + +| Option | Purpose | +|--------|---------| +| `WithRequired(true)` | Must be provided | +| `WithIsSecret(true)` | Masked in logs/UI | +| `WithDefaultValue(v)` | Default if not set | +| `WithHidden(true)` | Hidden from help | +| `WithPlaceholder(s)` | GUI placeholder text | +| `WithDescription(s)` | Help text | + +--- + +## Defining custom fields + +```go +func main() { + ctx := context.Background() + + fields := []field.SchemaField{ + field.StringField("domain", + field.WithRequired(true), + field.WithDescription("Okta domain (e.g., example.okta.com)"), + ), + field.StringField("api-token", + field.WithRequired(true), + field.WithIsSecret(true), + field.WithDescription("Okta API token"), + ), + field.StringSliceField("skip-groups"), + } + + cfg := &Config{} + c, err := cli.NewCobra(ctx, "baton-okta", cfg, + cli.WithSchema(fields...), + ) + // ... +} + +type Config struct { + Domain string `mapstructure:"domain"` + APIToken string `mapstructure:"api-token"` + SkipGroups []string `mapstructure:"skip-groups"` +} +``` + +Access in connector: +```go +func NewConnector(ctx context.Context, cfg *Config) (*Connector, error) { + client := NewClient(cfg.Domain, cfg.APIToken) + return &Connector{client: client, skipGroups: cfg.SkipGroups}, nil +} +``` diff --git a/rap/ref-faq.md b/rap/ref-faq.md new file mode 100644 index 0000000..f4e0ec8 --- /dev/null +++ b/rap/ref-faq.md @@ -0,0 +1,156 @@ +# ref-faq + +Common questions about baton connectors. + +--- + +## SDK tools + +**Q: What is baton-sdk vs cone vs conductorone-sdk-go?** + +| Tool | Purpose | User | +|------|---------|------| +| baton-sdk | Go SDK for building connectors | Connector developers | +| cone | CLI for C1 platform ops | End users (requests, approvals) | +| conductorone-sdk-go | Go SDK for C1 API | App integrators | + +Building a connector? Use baton-sdk. + +--- + +## Capabilities + +**Q: Do all connectors support provisioning?** + +No. Many are sync-only. Check specific connector's capability manifest. + +**Q: Where is the authoritative source for capabilities?** + +| Source | Purpose | +|--------|---------| +| Docs capabilities index | Human discovery | +| Connector binary + manifests | Runtime contract | + +--- + +## Run modes + +**Q: What are the run modes?** + +| Mode | Trigger | Behavior | +|------|---------|----------| +| One-shot | No `--client-id` | Runs once, produces c1z, exits | +| Daemon | `--client-id` provided | Continuous task processing | + +**Q: What is "service mode"?** + +Overloaded term: +1. Daemon mode (SDK feature) +2. OS service (deployment concern) +3. Marketing ("runs continuously") + +**Q: How to determine mode?** + +``` +--client-id provided? + No --> One-shot + Yes --> Daemon +``` + +--- + +## Data + +**Q: What is a .c1z file?** + +SQLite + gzip containing access graph data from sync. Inspect with `baton` CLI. + +**Q: What happens during sync?** + +| Stage | Method | Purpose | +|-------|--------|---------| +| 1 | ResourceType() | Learn what types exist | +| 2 | List() | Fetch all instances | +| 3 | Entitlements() | What permissions each offers | +| 4 | Grants() | Who has each permission | + +SDK processes ALL types per stage, not type-by-type. + +--- + +## Authentication + +**Q: Which auth method to use?** + +Whatever target system requires: + +| Method | Use Case | +|--------|----------| +| API Key | Token-based APIs | +| Bearer Token | OAuth2 APIs | +| OAuth2 Client Credentials | Service-to-service | +| JWT Service Account | Google-style | +| LDAP Bind | Directory services | + +--- + +## Pagination + +**Q: How to handle pagination?** + +| Helper | Use Case | +|--------|----------| +| pagination.Token | Linear pagination | +| pagination.Bag | Nested (children within parents) | + +**Q: bag.Current() returns nil?** + +Expected on first call. Nil-check before accessing fields. + +--- + +## Caching + +**Q: Can I use sync.Map?** + +Yes. Recommended for thread-safe caching. Cache users in List(), use in Grants(). + +**Q: Package-level sync.Map?** + +ANTI-PATTERN. Use struct fields. Package-level persists across syncs in daemon mode. + +--- + +## Meta-connectors + +**Q: What are baton-http and baton-sql?** + +Configuration-driven connectors using YAML + CEL instead of Go code. + +| Connector | Target | +|-----------|--------| +| baton-http | Any REST API | +| baton-sql | Any SQL database | + +**Q: When to use?** + +Standard patterns, don't want to write Go. Won't handle all edge cases. + +--- + +## Troubleshooting + +**Q: Pagination loop detected?** + +Returning same token causes infinite loop. Check: +- Returning empty string when done +- Not modifying cursor +- API actually returning different cursors + +**Q: Empty sync results?** + +Check: +- Credentials valid +- Permissions sufficient +- Filters not too restrictive +- API returning data diff --git a/rap/ref-glossary.md b/rap/ref-glossary.md new file mode 100644 index 0000000..b50546c --- /dev/null +++ b/rap/ref-glossary.md @@ -0,0 +1,71 @@ +# ref-glossary + +Term definitions for Baton connector development. + +--- + +## Core Terms + +| Term | Definition | +|------|------------| +| **Baton** | The connector framework: Go SDK + individual connectors | +| **baton-sdk** | Go library that handles sync orchestration and pagination | +| **Connector** | Go binary that syncs access data from a system | +| **c1z** | Compressed sync output file (gzip SQLite) | +| **Meta-connector** | Configuration-driven connector (baton-http, baton-sql) | + +## Access Model + +| Term | Definition | +|------|------------| +| **Resource** | Entity in target system: user, group, role, app | +| **Resource Type** | Classification with traits (e.g., "user" with TRAIT_USER) | +| **Trait** | Resource classification: TRAIT_USER, TRAIT_GROUP, TRAIT_ROLE, TRAIT_APP | +| **Entitlement** | Permission that can be granted (e.g., "admin" on a group) | +| **Grant** | Assignment of entitlement to principal | +| **Principal** | Entity receiving grants (typically users) | + +## SDK Concepts + +| Term | Definition | +|------|------------| +| **ResourceSyncer** | Interface: ResourceType, List, Entitlements, Grants methods | +| **pagination.Token** | SDK type for page cursors | +| **pagination.Bag** | SDK type for nested pagination state | +| **uhttp** | SDK HTTP client with retries and rate limiting | +| **RawId** | Annotation carrying external system's identifier | + +## Sync Lifecycle + +| Term | Definition | +|------|------------| +| **Sync** | Reading access data from a system | +| **Uplift** | ConductorOne process transforming raw records to domain objects | +| **ID Correlation** | Matching resources across syncs using RawId | + +## Provisioning + +| Term | Definition | +|------|------------| +| **Provision** | Writing access changes (grant, revoke, create, delete) | +| **Grant** (operation) | Add entitlement to principal | +| **Revoke** | Remove entitlement from principal | +| **CreateAccount** | JIT provisioning - create user in target system | +| **DeleteResource** | Remove resource from target system | + +## Run Modes + +| Term | Definition | +|------|------------| +| **One-shot** | Run once, produce c1z file, exit | +| **Daemon** | Long-running, polls ConductorOne for tasks | +| **Hosted** | Run by ConductorOne infrastructure | + +## Meta-Connector Terms + +| Term | Definition | +|------|------------| +| **CEL** | Common Expression Language for data transformation | +| **items_path** | JSONPath to array in API response | +| **primary_key** | Column used for cursor pagination | +| **skip_if** | CEL condition to filter grants | diff --git a/rap/ref-sdk.md b/rap/ref-sdk.md new file mode 100644 index 0000000..f4e57c5 --- /dev/null +++ b/rap/ref-sdk.md @@ -0,0 +1,211 @@ +# ref-sdk + +SDK interface reference for baton-sdk. Interfaces your connector implements. + +--- + +## Core interfaces + +### ConnectorBuilder (entry point) + +```go +type ConnectorBuilder interface { + MetadataProvider + ValidateProvider + ResourceSyncers(ctx context.Context) []ResourceSyncer +} + +type ConnectorBuilderV2 interface { // Preferred + MetadataProvider + ValidateProvider + ResourceSyncers(ctx context.Context) []ResourceSyncerV2 +} + +type MetadataProvider interface { + Metadata(ctx context.Context) (*v2.ConnectorMetadata, error) +} + +type ValidateProvider interface { + Validate(ctx context.Context) (annotations.Annotations, error) +} +``` + +--- + +### ResourceSyncer (sync data) + +```go +// V1 (legacy) +type ResourceSyncer interface { + ResourceType(ctx context.Context) *v2.ResourceType + List(ctx context.Context, parentResourceID *v2.ResourceId, + pToken *pagination.Token) ([]*v2.Resource, string, annotations.Annotations, error) + Entitlements(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Entitlement, string, annotations.Annotations, error) + Grants(ctx context.Context, resource *v2.Resource, + pToken *pagination.Token) ([]*v2.Grant, string, annotations.Annotations, error) +} + +// V2 (preferred) +type ResourceSyncerV2 interface { + ResourceType(ctx context.Context) *v2.ResourceType + List(ctx context.Context, parentResourceID *v2.ResourceId, + opts resource.SyncOpAttrs) ([]*v2.Resource, *resource.SyncOpResults, error) + Entitlements(ctx context.Context, resource *v2.Resource, + opts resource.SyncOpAttrs) ([]*v2.Entitlement, *resource.SyncOpResults, error) + Grants(ctx context.Context, resource *v2.Resource, + opts resource.SyncOpAttrs) ([]*v2.Grant, *resource.SyncOpResults, error) +} +``` + +V2 differences: receives `SyncOpAttrs` with session store, returns structured `SyncOpResults`. + +--- + +### ResourceProvisioner (grant/revoke) + +```go +// V2 (preferred) +type ResourceProvisionerV2 interface { + ResourceSyncer + Grant(ctx context.Context, resource *v2.Resource, + entitlement *v2.Entitlement) ([]*v2.Grant, annotations.Annotations, error) + Revoke(ctx context.Context, grant *v2.Grant) (annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_PROVISION`. + +--- + +### AccountManager (user provisioning) + +```go +type AccountManager interface { + ResourceSyncer + CreateAccount(ctx context.Context, + accountInfo *v2.AccountInfo, + credentialOptions *v2.LocalCredentialOptions, + ) (CreateAccountResponse, []*v2.PlaintextData, annotations.Annotations, error) + CreateAccountCapabilityDetails(ctx context.Context, + ) (*v2.CredentialDetailsAccountProvisioning, annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_ACCOUNT_PROVISIONING`. + +--- + +### ResourceManager (create/delete) + +```go +type ResourceManagerV2 interface { + ResourceSyncer + Create(ctx context.Context, resource *v2.Resource) (*v2.Resource, annotations.Annotations, error) + Delete(ctx context.Context, resourceId *v2.ResourceId, + parentResourceID *v2.ResourceId) (annotations.Annotations, error) +} +``` + +Enables `CAPABILITY_RESOURCE_CREATE`, `CAPABILITY_RESOURCE_DELETE`. + +--- + +## Pagination + +### Token (simple cursor) + +```go +type Token struct { + Token string // Opaque cursor, empty for first page + Size int // Requested page size +} +``` + +Usage in List(): +```go +resp, _ := client.ListUsers(ctx, pToken.Token, 100) +return resources, resp.NextCursor, nil, nil // Return next cursor +``` + +### Bag (nested pagination) + +```go +bag := &pagination.Bag{} +bag.Push(pagination.PageState{Token: "cursor", ResourceID: "parent-123"}) +state := bag.Pop() + +// Encode for return +nextToken, _ := bag.Marshal() +``` + +Use when paginating within nested resources. + +--- + +## Resource helpers + +```go +// Create user resource +resource.NewUserResource(name, resourceType, id, + resource.WithEmail(email, isPrimary), + resource.WithUserLogin(login), + resource.WithStatus(v2.UserTrait_Status_STATUS_ENABLED), +) + +// Create group resource +resource.NewGroupResource(name, resourceType, id, + resource.WithGroupProfile(description), +) + +// Create generic resource +resource.NewResource(name, resourceType, id, + resource.WithParentResourceID(parentID), + resource.WithAnnotation(&v2.ChildResourceType{ResourceTypeId: "child-type"}), +) +``` + +--- + +## Entitlement helpers + +```go +// Permission on a resource +entitlement.NewPermissionEntitlement(resource, "admin", + entitlement.WithDisplayName("Administrator"), + entitlement.WithGrantableTo(userResourceType), +) + +// Membership in a group +entitlement.NewAssignmentEntitlement(groupResource, "member", + entitlement.WithDisplayName("Member"), +) +``` + +--- + +## Grant helpers + +```go +grant.NewGrant( + resource, // Resource the entitlement belongs to + "permission", // Entitlement slug + principalID, // Who has the grant (*v2.ResourceId) +) +``` + +--- + +## Capabilities + +| Capability | Interface Required | +|------------|-------------------| +| `CAPABILITY_PROVISION` | ResourceProvisioner | +| `CAPABILITY_ACCOUNT_PROVISIONING` | AccountManager | +| `CAPABILITY_RESOURCE_CREATE` | ResourceManager | +| `CAPABILITY_RESOURCE_DELETE` | ResourceManager | +| `CAPABILITY_TARGETED_SYNC` | ResourceTargetedSyncer | +| `CAPABILITY_CREDENTIAL_ROTATION` | CredentialManager | +| `CAPABILITY_ACTIONS` | CustomActionManager | + +Capabilities declared in connector metadata, enabled by implementing interfaces. From 4bd79611a7a932cbb315450966c5ef0401b6f843 Mon Sep 17 00:00:00 2001 From: rch Date: Thu, 22 Jan 2026 19:51:11 -0800 Subject: [PATCH 3/3] Fix MDX comment syntax for breadcrumb Use {/* */} instead of for MDX compatibility --- developer/intro.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/developer/intro.mdx b/developer/intro.mdx index a5e2d07..f837c97 100644 --- a/developer/intro.mdx +++ b/developer/intro.mdx @@ -260,8 +260,8 @@ Plus: | Provision | Writing access changes back (grant, revoke, create, delete) | | Reconciliation | Comparing actual vs desired access and correcting drift | - +*/}