FEAT: .span_format / format_span: Why is span context suppression/reformatting so difficult? 💵 Production cost implications!

[tgrushka]

Feature Request

Is there a design philosophy behind making span context suppression so difficult? Many users in production environments need to control log volume for cost reasons, especially in cloud environments where verbose logging directly impacts billing. The inability to easily suppress span context seems to run counter to user control and cost management needs.

In addition, verbose logs full of extra Span clutter from libraries over which users have no control, makes the logs unergonomic, difficult and frustrating to read, and less accessible for developers with visual impairments, making troubleshooting problems more difficult.

This lack of ergonomics in such an important and ubiquitous logging framework as tracing seems to go against the grain of Rust's design philosophy to help developers "spend their time focusing on the program’s logic rather than chasing down bugs".

Example

kube-rs hard-codes Spans on Controller reconcile loops, forcing an annoying and verbose display in every info! or higher logging call:
let reconciler_span = info_span!("reconciling object"... runtime/src/controller/mod.rs#L393

A single kube-rs reconciliation can generate hundreds of log lines with verbose span context, making it difficult to quickly identify actual application logic vs. framework noise.

Additionally, the only way to access the reconciliation reason is in the Span context, and tracing provides no way to transform the Span into a custom format of the user's choice; it gives 100% of the control to the upstream developer and narrows the choice of end users to either keep the verbose logging, or suppress the logs and entirely lose the information they carry.

The following snippet illustrates that to read the actual log lines, one must either wrap to several lines per "line", or scroll way to the right, cutting off the timestamp information:

05:14:53.592  INFO reconciling object{object.ref=MyCustomResource.v1.mydomain.example.com/my-resource-test-instance.my-test-namespace object.reason=object updated}: my-kubernetes-operator/src/controller/my-resource_controller.rs:105: 🆕 Detected newly created MyCustomResource my-resource-test-instance, fast-tracking initial setup
05:14:54.092  INFO reconciling object{object.ref=MyCustomResource.v1.mydomain.example.com/my-resource-test-instance.my-test-namespace object.reason=object updated}: my-kubernetes-operator/src/controller/my-resource_controller.rs:327: 🏷️  Adding instance labels to newly created MyCustomResource my-resource-test-instance
05:14:54.190  WARN reconciling object{object.ref=MyCustomResource.v1.mydomain.example.com/my-resource-test-instance.my-test-namespace object.reason=object updated}: my-kubernetes-operator/src/controller/my-resource_controller.rs:334: Patching instance labels for MyCustomResource my-resource-test-instance: {"app": "my-resource", "app.kubernetes.io/instance": "my-resource-test-instance", "app.kubernetes.io/managed-by": "operator.my-resource.mydomain.example.com", "app.kubernetes.io/name": "my-resource"}
...

Crates

tracing-subscriber

Motivation

The current design forces a binary choice: accept verbose, uncontrollable span context or lose valuable information entirely. This creates several problems:

1. Production Cost Impact: Cloud logging bills scale with log volume. Verbose span contexts can increase costs significantly for high-throughput applications.

2. Developer Experience: Cluttered logs make debugging harder, especially for developers with visual impairments who rely on screen readers, because it prevents them from customizing the logging format for crates they do not own.

3. Library Lock-in: Popular libraries like kube-rs force their span formatting choices on all downstream users, with no escape hatch. And it certainly is not deliberate -- they are just trying to help users by providing basic logging. But tracing requires all upstream authors to put in way more effort than should be required to allow their tracing formats to be user-customizable.

4. Information Accessibility: Critical data (like reconciliation reasons) is trapped in span formatting with no way to extract or reformat it.

Proposal

Add a .span_format() method to the fmt::layer() builder, following the existing pattern of .event_format() and .fmt_fields():

tracing_subscriber::fmt::layer()
    .with_target(config.target)
    .with_file(config.file)
    .span_format(KubeSpanFormatter)  // <-- New method
    .with_timer(timer)

New trait:

pub trait FormatSpan<S> {
    fn format_span(
        &self,
        span: &SpanRef<S>,
        fields: &FormattedFields<DefaultFields>,
        writer: fmt::Writer<'_>
    ) -> Option<fmt::Result>
    where
        S: Subscriber + for<'a> LookupSpan<'a>;
}

Example usage:

use std::fmt;
use std::sync::LazyLock;
use regex::Regex;

static OBJECT_REGEX: LazyLock<Regex> = LazyLock::new(|| {
    Regex::new(r"^(?<kind>[^.]+)\..*?/(?<name>[^.]+)\.(?<namespace>.+)$").unwrap()
});

pub struct KubeSpanFormatter;

// UNOPTIMIZED CODE - for illustrative purposes only:
impl<S> FormatSpan<S> for KubeSpanFormatter {
    fn format_span(&self, span: &SpanRef<S>, fields: &FormattedFields<DefaultFields>, mut writer: fmt::Writer<'_>) -> Option<fmt::Result> {
        if span.name() == "reconciling object" {
            let fields_str = fields.as_str();
            
            if let Some(obj_ref_start) = fields_str.find("object.ref=") {
                let remaining = &fields_str[obj_ref_start + 11..];
                let obj_ref = if let Some(end) = remaining.find(' ') {
                    &remaining[..end]
                } else {
                    remaining
                }.trim_matches('"');
                
                if let Some(caps) = OBJECT_REGEX.captures(obj_ref) {
                    let kind = caps.name("kind").map(|m| m.as_str()).unwrap_or("Unknown");
                    let name = caps.name("name").map(|m| m.as_str()).unwrap_or("unknown");
                    
                    let reason = if let Some(reason_start) = fields_str.find("object.reason=") {
                        let remaining = &fields_str[reason_start + 14..];
                        if let Some(end) = remaining.find(' ') {
                            &remaining[..end]
                        } else {
                            remaining
                        }.trim_matches('"')
                    } else {
                        "unknown"
                    };
                    
                    return Some(write!(writer, "[{kind}] {name} ({reason})"));
                }
            }
        }
        
        None  // Use default formatting for all other spans
    }
}

Alternatives

1. Custom Event Formatters
Currently requires implementing the entire FormatEvent trait and duplicating existing formatting logic just to modify span context. This approach:

• ❌ Complex: Must reimplement timestamp, level, target, and message formatting
• ❌ Formatter Duplication: Requires separate formatters for each library's spans, making consistency difficult to maintain
• ❌ Fragile: Tightly couples user code to internal formatting details
• ❌ Incomplete: Cannot easily extract span field data for transformation

2. Layer-based Event Transformation
Attempted using custom layers to intercept and replace events, but layers cannot suppress the original verbose events from reaching the formatter:

• ❌ Duplication: Both original and transformed events appear in logs
• ❌ Architectural limitation: Layers observe but cannot replace events
• ❌ Performance overhead: Creates additional events without removing originals

3. Field-level Formatting (fmt_fields)
Using custom field formatters to transform individual field values:

• ❌ Limited scope: Can only modify field values, not the overall span structure
• ❌ Still verbose: Retains the reconciling object{...}: wrapper
• ❌ Cannot change span names: Field formatters don't control span naming

4. Post-processing with Regex
Filtering log output using regex patterns in log aggregators:

• ❌ Loses structured data: Regex filtering destroys semantic information
• ❌ Fragile: Breaks when internal formatting changes
• ❌ External dependency: Requires log aggregation infrastructure
• ❌ Not portable: Different regex needed for each deployment environment

5. Fork tracing-subscriber
Creating a custom fork with span context modifications:

• ❌ Ecosystem fragmentation: Diverges from standard tooling
• ❌ Maintenance burden: Must track upstream changes indefinitely
• ❌ Community impact: Other users cannot benefit from improvements

Why the Proposed Solution?

The .span_format() approach was chosen because it:

• ✅ Follows existing patterns: Consistent with event_format() and fmt_fields()
• ✅ Surgical precision: Only affects span context, preserves all other formatting
• ✅ Performance conscious: Zero cost when unused, minimal overhead when used

[hawkw]

You can use filters to control which spans are recorded --- if the concern is that unwanted span contexts are included in log lines, filtering out the spans you're not interested in is the recommended approach, as the filtered-out spans will be completely disabled, and have the least possible runtime overhead. Choosing not to format some spans in the formatter will still pay the runtime performance cost of recording those spans and passing them to the formatter, but filtering them out avoids this.

If the goal is to format particular spans in a special way, though, I think adding an API for configuring how spans are formatted in the tracing_subscriber::fmt module is not a bad idea. The interface you're proposing seems a little strange though: it's operating on the span's fields when they have already been formatted and are being added to an event, and is post-processing the previously formatted fields using a regex. That seems like a pretty inefficient way to format specific fields in a custom way, when the field formatting API already allows determining how fields are handled based on the field's name and type --- so using that mechanism would be much more efficient than formatting the fields in the default way and then re-parsing them with a regex every time an event is formatted. If we add something for customizing how spans are formatted, I think we should consider how it interacts with the existing field-formatting APIs to avoid situations like this.

I think we'd definitely be open to a proposal for an API for configuring how spans aer formatted separately from writing a custom event formatter. But, again, I'd encourage that you use filters for eliminating spans you don't want to record at all.

---

As an aside, I note that this issue shows some pretty clear signs of having been written by an LLM. Honestly, I found it somewhat hard to follow --- I'm not completely sure if the model totally understands what you want to do here. In the future, I'd encourage using your own words when opening issues like this, as it will be easier for maintainers of libraries to understand what you're actually asking for.

[tgrushka]

Can you please point out what was hard to follow so I can rewrite it and make it easier to follow?