Observability

The EventStoreDB gRPC clients are designed with observability in mind, offering support for OpenTelemetry. This integration provides a set of distributed traces, enabling developers to gain deeper insights into their system.

WARNING

Currently, OpenTelemetry observability support is not available for all clients. Moreover, instrumentation is only provided for append and subscribe operations, which includes both 'Catchup' and 'Persistent' modes.

You can click on the links below to view the full code for each client:

• NodeJS
• Java
• C#

Required Packages

# Yarn
$ yarn add @eventstore/opentelemetry

# NPM
$ npm install --save @eventstore/opentelemetry

# TypeScript Declarations are included in the package.

# Yarn
$ yarn add @eventstore/opentelemetry

# NPM
$ npm install --save @eventstore/opentelemetry

OpenTelemetry is supported out of the box in the Java client.

dotnet add package EventStore.Client.Extensions.OpenTelemetry

Instrumentation

To emit trace data, you must first install and use the dedicated package, as instructed in the Required Packages section, if provided. This package includes the necessary instrumentation that needs to be registered with the client.

const provider = new NodeTracerProvider();

const instrumentation = new EventStoreDBInstrumentation();

registerInstrumentations({
  instrumentations: [instrumentation],
  tracerProvider: provider,
});

const provider = new NodeTracerProvider();

const instrumentation = new EventStoreDBInstrumentation();

registerInstrumentations({
  instrumentations: [instrumentation],
  tracerProvider: provider,
});

// Instrumentation is enabled by default in the Java client.

OpenTelemetrySdk.builder()
        .setTracerProvider(sdkTracerProvider)
        .buildAndRegisterGlobal();

tracerProviderBuilder
    .AddEventStoreClientInstrumentation();

Traces

Traces provide a clear picture of how operations are carried out in a distributed system, making it easier to maintain and enhance the system over time. Traces from the clients can be exported to any compatible collector that supports the OpenTelemetry protocol (OTLP).

In order for the client to emit traces, you need to need to enable instrumentation as described in Instrumentation.

For more guidance on setting up and utilizing tracing, refer to the OpenTelemetry documentation.

An example of a trace is shown below:

Activity.TraceId:            8da04787239dbb85c1f9c6fba1b1f0d6
Activity.SpanId:             4352ec4a66a20b95
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName: eventstoredb
Activity.DisplayName:        streams.append
Activity.Kind:               Client
Activity.StartTime:          2024-05-29T06:50:41.2519016Z
Activity.Duration:           00:00:00.1500707
Activity.Tags:
    db.eventstoredb.stream: d7caa2a5-1e19-4108-9541-58d5fba02d42
    server.address: localhost
    server.port: 2113
    db.system: eventstoredb
    db.operation: streams.append
StatusCode: Ok
Resource associated with Activity:
    service.name: sample
    service.instance.id: 7316ef20-c354-4e64-97da-c1b99c2c28b0
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.8.1

In this case, the trace is for an append operation on a stream. The trace includes the trace ID, span ID, trace flags, activity source name, display name, kind, start time, duration, tags, status code, and resource associated with the activity.

NOTE

The structure of the trace may vary depending on the client and the operation being performed but will generally include the same information.

Exporting Traces

You can set up various exporters to send traces to different destinations. Additionally, you have the option to export these traces to a collector of your choice, such as Jaeger or Seq.

For instance, if you choose to use Jaeger as your backend of choice, you can view your traces in the Jaeger UI, which provides a powerful interface for querying and visualizing your trace data.

The code snippets below demonstrate how to set up one or more exporters for each client:

const memoryExporter = new InMemorySpanExporter();
const otlpExporter = new OTLPTraceExporter({ url: "http://localhost:4317" }); // change this to your OTLP receiver address
const consoleExporter = new ConsoleSpanExporter();

provider.addSpanProcessor(new SimpleSpanProcessor(memoryExporter));
provider.addSpanProcessor(new SimpleSpanProcessor(consoleExporter));
provider.addSpanProcessor(new SimpleSpanProcessor(otlpExporter));

const memoryExporter = new InMemorySpanExporter();
const otlpExporter = new OTLPTraceExporter({ url: "http://localhost:4317" }); // change this to your OTLP receiver address
const consoleExporter = new ConsoleSpanExporter();

provider.addSpanProcessor(new SimpleSpanProcessor(memoryExporter));
provider.addSpanProcessor(new SimpleSpanProcessor(consoleExporter));
provider.addSpanProcessor(new SimpleSpanProcessor(otlpExporter));

LoggingSpanExporter consoleExporter = LoggingSpanExporter.create();

OtlpGrpcSpanExporter jaegarExporter = OtlpGrpcSpanExporter.builder()
        .setEndpoint("http://localhost:4317")
        .build();

SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(SimpleSpanProcessor.create(jaegarExporter))
        .addSpanProcessor(SimpleSpanProcessor.create(consoleExporter))
        .setResource(resource)
        .build();

tracerProviderBuilder
    .AddConsoleExporter()
    .AddJaegerExporter(
        options => {
            options.Endpoint = new Uri("http://localhost:4317");
            options.Protocol = JaegerExportProtocol.UdpCompactThrift;
        }
    );

For more details on configuring exporters for specific programming languages, refer to the OpenTelemetry documentation.