AspireOtelResearch

A .NET Aspire distributed application demonstrating comprehensive OpenTelemetry instrumentation, saga orchestration with Wolverine, and event sourcing with Marten.

Overview

This project showcases a cloud-native, distributed .NET 9.0 application built with .NET Aspire, featuring:

• Distributed Tracing: Full OpenTelemetry instrumentation across all services with context propagation
• Saga Orchestration: Long-running order processing workflow using Wolverine
• Event Sourcing: Marten-backed event store with PostgreSQL
• Message-Driven Architecture: Asynchronous communication via RabbitMQ
• Service Orchestration: .NET Aspire AppHost for infrastructure provisioning and service discovery

Architecture

Service Topology

graph TB
    subgraph "Infrastructure"
        PG[PostgreSQL Server]
        RMQ[RabbitMQ]
        PGA[pgAdmin]
    end

    subgraph "Databases"
        PG --> ODB[(ordersdb)]
        PG --> IDB[(inventorydb)]
        PG --> RDB[(researchdb)]
    end

    subgraph "Services"
        AH[AppHost<br/>Orchestrator]
        OS[OrderService<br/>Saga Coordinator]
        IS[InventoryService<br/>Inventory Manager]
        AS[ApiService<br/>API Gateway]
        WA[WebApp<br/>UI Frontend]
        WS[WorkerService<br/>Background Worker]
    end

    AH -.provisions.-> PG
    AH -.provisions.-> RMQ
    AH -.orchestrates.-> OS
    AH -.orchestrates.-> IS
    AH -.orchestrates.-> AS
    AH -.orchestrates.-> WA
    AH -.orchestrates.-> WS

    OS --> ODB
    OS --> RMQ
    IS --> IDB
    IS --> RMQ
    AS --> RDB
    AS --> RMQ
    WS --> RDB
    WS --> RMQ
    WA --> AS
    WA --> OS

    style AH fill:#e1f5ff
    style OS fill:#ffe1e1
    style IS fill:#e1ffe1
    style AS fill:#ffe1ff
    style WA fill:#fff5e1
    style WS fill:#f5e1ff

Loading

Saga Orchestration Flow

The OrderService coordinates a distributed saga for order processing:

sequenceDiagram
    participant C as Client
    participant OS as OrderService
    participant MQ as RabbitMQ
    participant IS as InventoryService
    participant DB as PostgreSQL

    C->>OS: POST /orders (CreateOrder)
    activate OS
    OS->>DB: Store Order (Pending)
    OS->>MQ: Publish VerifyInventory
    deactivate OS

    MQ->>IS: Deliver VerifyInventory
    activate IS
    IS->>DB: Query InventoryItem
    IS->>DB: Check Available Quantity
    IS->>MQ: Publish InventoryVerified
    deactivate IS

    MQ->>OS: Deliver InventoryVerified
    activate OS
    OS->>DB: Update Order (InventoryVerified)
    OS->>MQ: Publish ReserveInventory
    deactivate OS

    MQ->>IS: Deliver ReserveInventory
    activate IS
    IS->>DB: Update ReservedQuantity
    IS->>MQ: Publish InventoryReserved
    deactivate IS

    MQ->>OS: Deliver InventoryReserved
    activate OS
    OS->>DB: Update Order (Completed)
    OS->>MQ: Publish OrderCompleted
    deactivate OS

Loading

Message Flow

graph LR
    subgraph "Order Saga Messages"
        CO[CreateOrder]
        VI[VerifyInventory]
        IV[InventoryVerified]
        RI[ReserveInventory]
        IR[InventoryReserved]
        OC[OrderCompleted]
        OF[OrderFailed]
    end

    CO --> VI
    VI --> IV
    IV --> RI
    RI --> IR
    IR --> OC
    IR -.failure.-> OF
    IV -.failure.-> OF

    style CO fill:#e1f5ff
    style VI fill:#ffe1e1
    style IV fill:#e1ffe1
    style RI fill:#ffe1ff
    style IR fill:#fff5e1
    style OC fill:#c8e6c9
    style OF fill:#ffcdd2

Loading

Technologies & Resources

Core Frameworks

• .NET 9.0 - Latest .NET framework
• .NET Aspire 9.5.2 - Distributed application orchestration
• C# 13 - Latest language features

Service Communication

• Wolverine 3.13.0/3.13.3 - Mediator and saga orchestration framework
• RabbitMQ - Message broker for async communication
• Aspire Service Discovery - Built-in DNS resolution for inter-service communication

Data & Persistence

• Marten 7.40.0/7.40.3 - Event sourcing and document database for PostgreSQL
• PostgreSQL - Relational database (3 separate databases for bounded contexts)
• Npgsql - PostgreSQL driver for .NET

Observability

• OpenTelemetry - Distributed tracing and metrics
• System.Diagnostics.DiagnosticSource 10.0.0 - Activity and trace APIs
• Aspire Dashboard - Built-in monitoring and telemetry visualization

Package References

Service	Key Packages
OrderService	WolverineFx 3.13.0, Marten 7.40.0, Aspire.Npgsql 9.5.2
InventoryService	WolverineFx.Marten 3.13.3, Marten 7.40.0
ApiService	Aspire.Npgsql.EntityFrameworkCore.PostgreSQL 9.5.2
ServiceDefaults	Aspire.RabbitMQ.Client 9.5.2, OpenTelemetry.Exporter.OpenTelemetryProtocol 1.10.0

Getting Started

Prerequisites

• .NET 9.0 SDK or later
• Docker Desktop (for PostgreSQL and RabbitMQ containers)
• Visual Studio 2022 or VS Code with C# extension
• Git (for cloning the repository)

Installation

1. Clone the repository

   git clone https://github.com/amccool/WolverinePlayground.git
   cd WolverinePlayground

2. Restore dependencies

   dotnet restore AspireOtelResearch.sln

3. Build the solution

   dotnet build AspireOtelResearch.sln

Running the Application

Option 1: Visual Studio 2022

1. Open AspireOtelResearch.sln
2. Set AspireOtelResearch.AppHost as the startup project
3. Press F5 or click "Start Debugging"

The Aspire Dashboard will automatically open in your browser at https://localhost:17240

Option 2: Command Line

cd src/AspireOtelResearch.AppHost
dotnet run

Navigate to the Aspire Dashboard URL shown in the console output (typically https://localhost:17240).

Accessing Services

Once running, the following endpoints are available:

Service	Purpose	Default URL
Aspire Dashboard	Monitoring & telemetry visualization	https://localhost:17240
OrderService	Order management API	https://localhost:7xxx
ApiService	Research API	https://localhost:7xxx
WebApp	Frontend UI	https://localhost:7xxx
pgAdmin	PostgreSQL admin interface	http://localhost:5050
RabbitMQ Management	Message broker admin	http://localhost:15672

Note: Actual ports are dynamically assigned by Aspire and shown in the dashboard.

Project Structure

AspireOtelResearch/
├── src/
│   ├── AspireOtelResearch.AppHost/           # Aspire orchestration host
│   ├── AspireOtelResearch.ServiceDefaults/   # Shared service configuration
│   ├── AspireOtelResearch.Contracts/         # Shared message contracts
│   ├── AspireOtelResearch.OrderService/      # Saga coordinator service
│   ├── AspireOtelResearch.InventoryService/  # Inventory management service
│   ├── AspireOtelResearch.ApiService/        # REST API service
│   ├── AspireOtelResearch.WebApp/            # Frontend web application
│   └── AspireOtelResearch.WorkerService/     # Background worker service
└── AspireOtelResearch.sln

Key Features

1. OpenTelemetry Integration

All services include comprehensive OpenTelemetry instrumentation:

Distributed Tracing

• ActivitySource instances in each service
• Automatic context propagation across service boundaries
• Baggage propagation for cross-cutting concerns (order.id, order.customer)
• Activity tagging with business-relevant metadata

Metrics Collection

• ASP.NET Core instrumentation for HTTP server metrics
• HTTP client instrumentation for outbound request metrics
• Runtime instrumentation for .NET performance counters

Structured Logging

• ILogger with OpenTelemetry integration
• Automatic TraceId and SpanId correlation in logs
• Console and OTLP exporters for dual-export pattern

2. Saga Pattern with Wolverine

The OrderService implements a long-running saga for order processing:

Workflow Steps

1. CreateOrder - Client initiates order creation
2. VerifyInventory - Check if requested product quantity is available
3. InventoryVerified - Inventory service confirms availability
4. ReserveInventory - Reserve the verified inventory
5. InventoryReserved - Inventory service confirms reservation
6. OrderCompleted - Order successfully processed

Failure Handling

• If inventory verification fails → OrderFailed
• If inventory reservation fails → OrderFailed (with compensation logic)

3. Event Sourcing with Marten

Both OrderService and InventoryService use Marten for:

• Document storage (Order, InventoryItem models)
• Transactional outbox pattern with Wolverine integration
• Automatic schema creation and management
• Separate databases for bounded contexts

4. Service Discovery

Aspire automatically configures service discovery:

• Services reference each other by logical name (e.g., "apiservice")
• Connection strings injected via configuration
• Health checks and readiness probes
• Graceful startup ordering with .WaitFor()

Testing the Saga Workflow

Create an Order

curl -X POST https://localhost:7xxx/orders \
  -H "Content-Type: application/json" \
  -d '{
    "customerName": "John Doe",
    "productId": "SAMPLE-PRODUCT-1",
    "quantity": 5
  }'

Replace 7xxx with the actual port from the Aspire Dashboard.

Query Order Status

curl https://localhost:7xxx/orders/{order-id}

Monitor in Aspire Dashboard

1. Navigate to Traces tab

2. Find the trace for your CreateOrder request

3. Observe the full distributed trace showing:

   • Order creation in OrderService
   • VerifyInventory message published to RabbitMQ
   • InventoryService processing
   • ReserveInventory flow
   • Order completion

4. Check Metrics tab for:

   • HTTP request rates and durations
   • Message processing metrics
   • Database query performance

5. View Logs tab for correlated structured logs across all services

Troubleshooting

Services Exit Immediately

Symptom: OrderService or InventoryService shows "Exited" status in Aspire Dashboard

Causes & Solutions:

1. Missing connection strings - Aspire injects database-specific names ("ordersdb", "inventorydb")

   • Verify AppHost references: .WithReference(ordersDb) and .WithReference(inventoryDb)
   • Check Program.cs falls back correctly: GetConnectionString("ordersdb") ?? GetConnectionString("postgres")

2. Missing Id property on models - Marten requires Id field

   • Ensure public Guid Id { get; set; } exists on InventoryItem and Order models

3. Database not created - Marten auto-create not working

   • Verify AutoCreateSchemaObjects = Weasel.Core.AutoCreate.All in Marten configuration

DNS Resolution Errors

Symptom: System.Net.Sockets.SocketException: No such host is known. (apiservice:80)

Solution: Add proper service references in AppHost:

builder.AddProject<Projects.AspireOtelResearch_WebApp>("webapp")
    .WithReference(apiService)    // Required for service discovery
    .WaitFor(apiService);          // Ensures proper startup order

Version Conflict Warnings

Symptom: NU1107: Version conflict detected for System.Diagnostics.DiagnosticSource

Solution: This is expected and safe. The explicit package reference to version 10.0.0 overrides Wolverine's overly-restrictive constraint. Functionality is not affected.

RabbitMQ Connection Failures

Symptom: RabbitMQ.Client.Exceptions.BrokerUnreachableException

Solutions:

1. Ensure Docker Desktop is running
2. Verify RabbitMQ container is healthy in Aspire Dashboard
3. Add .WaitFor(rabbitmq) to service references in AppHost

PostgreSQL Connection Failures

Symptom: Npgsql.NpgsqlException: Connection refused

Solutions:

1. Check PostgreSQL container status in Aspire Dashboard
2. Verify database references in AppHost: .WithReference(ordersDb)
3. Ensure .WaitFor(postgresServer) is configured for dependent services

OpenTelemetry Observability Patterns

Activity Tagging Best Practices

using var activity = ActivitySource.StartActivity("OperationName", ActivityKind.Consumer);

// Add business context from baggage
activity?.SetTag("order.id", Activity.Current?.GetBaggageItem("order.id"));
activity?.SetTag("order.customer", Activity.Current?.GetBaggageItem("order.customer"));

// Add operation-specific tags
activity?.SetTag("inventory.productId", message.ProductId);
activity?.SetTag("inventory.requestedQuantity", message.Quantity);

// Add result tags
activity?.SetTag("inventory.isAvailable", isAvailable);

// Add events for significant milestones
activity?.AddEvent(new ActivityEvent("InventoryVerificationComplete"));

Baggage Propagation

Baggage allows passing cross-cutting context through distributed traces:

// Set baggage in upstream service
Activity.Current?.SetBaggage("order.id", orderId.ToString());
Activity.Current?.SetBaggage("order.customer", customerName);

// Read baggage in downstream service
var orderId = Activity.Current?.GetBaggageItem("order.id");
var customer = Activity.Current?.GetBaggageItem("order.customer");

Dual-Export Configuration

All services export telemetry to both console (development) and OTLP (production):

services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddConsoleExporter()      // For local debugging
        .AddOtlpExporter())        // For observability backends
    .WithMetrics(metrics => metrics
        .AddConsoleExporter()
        .AddOtlpExporter());

Database Schema

ordersdb (OrderService)

orders.orders table (managed by Marten):

• id (uuid, PK) - Order identifier
• data (jsonb) - Order document with CustomerName, ProductId, Quantity, Status
• mt_version (int) - Document version for optimistic concurrency

inventorydb (InventoryService)

inventory.inventory table (managed by Marten):

• id (uuid, PK) - Inventory item identifier
• data (jsonb) - InventoryItem document with ProductId, AvailableQuantity, ReservedQuantity
• mt_version (int) - Document version

researchdb (ApiService/WorkerService)

Domain-specific research data schema (application-specific).

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Commit changes: git commit -am 'Add your feature'
4. Push to branch: git push origin feature/your-feature
5. Submit a pull request

Resources & References

Official Documentation

• .NET Aspire Documentation
• Wolverine Framework
• Marten Event Store
• OpenTelemetry .NET

Key Concepts

• Saga Pattern
• Transactional Outbox
• Event Sourcing
• Distributed Tracing

Related Projects

• .NET Aspire Samples
• OpenTelemetry Demo

License

This project is provided as-is for educational and research purposes.

Acknowledgments

Built with Claude Code

Co-Authored-By: Claude noreply@anthropic.com