The ability to restore trace data when calling the scheduled method again

[mfilatov]

Hi everyone, I use this framework a lot and I would like to propose adding a feature.

I need to save traceId and spanId from the current trace and when repeating the method call, restore trace data.
Of course, we can use MDC context, but it won't work, if we are using OTEL framework.

My proposals for improvement:
POC: https://github.com/mfilatov/transaction-outbox/pull/1/files
Add Tracing object to Invocation
Add TracingInterceptor interface to implement store and restore trace

Example of implementation TracingInterceptor:

package org.example;

import com.gruelbox.transactionoutbox.TransactionOutboxEntry;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanContext;
import io.opentelemetry.api.trace.SpanId;
import io.opentelemetry.api.trace.TraceFlags;
import io.opentelemetry.api.trace.TraceId;
import io.opentelemetry.api.trace.TraceState;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;

import java.util.Objects;
import java.util.function.Consumer;

public class TracingInterceptorImpl implements TracingInterceptor {
    
    @Override
    public Tracing getTracing() {
        SpanContext spanContext = Span.current().getSpanContext();
        var traceId = spanContext.getTraceId();
        var spanId = spanContext.getSpanId();

        return new Tracing(traceId, spanId);
    }

    @Override
    public Consumer<TransactionOutboxEntry> wrapTrace(Tracing tracing, Consumer<TransactionOutboxEntry> localExecutor) {
        Consumer<TransactionOutboxEntry> executor = localExecutor;
        
        SpanContext currentSpanContext = Span.current().getSpanContext();
        
        String storedTraceId = tracing.getTraceId();
        String storedSpanId = tracing.getSpanId();
        
        if (TraceId.isValid(storedTraceId) && SpanId.isValid(storedSpanId)
                && !Objects.equals(storedTraceId, currentSpanContext.getTraceId())) {
            Span restoredSpan = restoreSpan(storedTraceId, storedSpanId, currentSpanContext);
            executor = wrapExecutor(localExecutor, restoredSpan);
        }
        
        return executor;
    }

    Span restoreSpan(String storedTraceId, String storedSpanId, SpanContext currentSpanContext) {
        SpanContext parentSpanContext = SpanContext.createFromRemoteParent(
                storedTraceId,
                storedSpanId,
                TraceFlags.getSampled(),
                TraceState.getDefault()
        );

        Span parentSpanNoOp = Span.wrap(parentSpanContext);

        OpenTelemetry openTelemetry = GlobalOpenTelemetry.get();
        Tracer tracer = openTelemetry.getTracer("transaction-outbox", "1.0.0");

        return tracer.spanBuilder("restore-trace")
                .addLink(currentSpanContext)
                .setParent(Context.current().with(parentSpanNoOp))
                .startSpan();
    }

    Consumer<TransactionOutboxEntry> wrapExecutor(Consumer<TransactionOutboxEntry> localExecutor, Span span) {
        return (
                transactionOutboxEntry -> {
                    try (Scope ignored = span.makeCurrent()) {
                        localExecutor.accept(transactionOutboxEntry);
                    } finally {
                        span.end();
                    }
                }
        );
    }
}

[badgerwithagun]

Hey @mfilatov! Thanks for this.

I'm not very hot on OTEL but I've referred this to the development teams here for some feedback.

[badgerwithagun]

Why do you need to store the span id?

[mfilatov]

Hi @badgerwithagun,

We have several services that sequentially pass data to each other.
For example, a chain:
service 1 -> service 2 -> service 3 -> service 4.

I would like to see how the trace goes through all four services, without failures it will be something like this:
service 1: (trace1, span1) -> service 2: (trace1, span2) -> service 3: (trace1, span3) -> service 4: (trace1, span4)
The spans will be different, but the trace will be the same.

If a failure occurs, the original trace will be lost:
service 1: (trace1, span1) -> service 2: (trace1, span2) failure and retry with new trace -> service 3: (trace2, span3) -> service 4: (trace2, span4)

But we can save the trace data and restore it when we call method again:
service 1: (trace1, span1) -> service 2: (trace1, span2) failure and retry with restored trace -> service 3: (trace1, span3) -> service 4: (trace1, span4)

[mfilatov]

If you need, I can prepare more clear diagrams and examples

[mfilatov]

If it helps, I'm ready to make a full functionality PR with unit tests.