Skip to main content
High-throughput services can produce millions of spans per second. Storing every span is expensive, so teams commonly run the OpenTelemetry Collector’s tail-sampling processor to keep only 1-in-N spans. Each kept span carries a SampleRate attribute recording N. Once data is sampled, naive aggregations are wrong: count() returns N-times fewer events than actually occurred, sum() and avg() are biased, and percentiles shift. Dashboards show misleadingly low request counts, throughput, and error rates. ClickStack solves this with a sampling-aware query engine. When you configure a sample rate expression on a trace source, the query builder rewrites SQL aggregations to weight each span by its sample rate — across dashboards, alerts, and ad-hoc searches.

​
How it works

When a trace source has a sampleRateExpression configured, ClickStack wraps it as:
Spans without a SampleRate attribute default to weight 1, so unsampled data produces identical results to the original queries. The query builder then rewrites aggregations:
Percentiles under sampling use quantileTDigestWeighted, an approximate T-Digest sketch. Results are close but not exact.

​
Configuring the sample rate expression

Open your trace source in Source Settings and enter the ClickHouse expression that evaluates to the per-span sample rate in the Sample Rate Expression field. For example, if your OpenTelemetry tail-sampling processor writes the rate into SpanAttributes['SampleRate']: Once configured, all charts, dashboards, alerts, and service dashboard panels automatically apply sample-weighted aggregations. No changes to individual queries are needed.
Last modified on June 12, 2026