Spans¶

Everything below is what Flare actually emits. Attributes marked conditional are absent rather than zero when the underlying value was never observed. A missing attribute means "not measured", which is different from a measured zero.

spark.application¶

SpanKind SERVER, trace root.

Opened when SparkContext initialises, closed at SparkContext.stop().

spark.sql.N¶

SpanKind INTERNAL, child of spark.application.

One per SparkListenerSQLExecutionStart. N is the execution id.

spark.job.N¶

SpanKind INTERNAL, child of spark.sql.N or spark.application.

spark.stage.N¶

SpanKind INTERNAL, child of spark.job.N.

Metrics below are Spark's own sums across every task in the stage, so they are directly comparable with each other. All are set at onStageCompleted from StageInfo.taskMetrics, except spark.stage.scheduler.delay_ms, which Spark does not report and Flare derives from task ends; see below.

Why spark.stage.name is often useless, and what to use instead. Spark derives it from RDD.creationSite. For an async subquery or broadcast, the RDD is created on a Spark thread-pool thread, so the name resolves to something like $anonfun$withThreadLocalCaptured$2 at CompletableFuture.java:1768. The Spark UI shows the same string. In one four-stage SkewedJob run, three of the four stages looked like that.

StageInfo.details cannot recover it either: that stack was captured on the same pool thread and contains no user frame at all, only org.apache.spark.* and java.base/*. The stages whose names are useful are the only ones whose stacks contain user code, so walking the stack adds nothing where it is needed.

spark.stage.sql.description is the answer: for any stage under a SQL execution, Spark's own execution description names the user code exactly. spark.stage.name is left untouched so Spark UI correlation still works. A pure-RDD stage belonging to no SQL execution gets neither attribute: there is no source for the information in that case.

spark.stage.scheduler.delay_ms is the only stage attribute Spark does not report. It is derived per task as duration − executorRunTime − deserializeTime − resultSerializationTime − resultFetchTime and summed over the stage, matching Spark's own AppStatusUtils.schedulerDelay. The clamp at zero is applied per task, never to the sum, because driver and executor clocks differ and a fast task can report more run time than its own wall clock. The attribute is omitted when no task ends were observed, because a 0 there would read as "no queueing" rather than "not measured".

spark.task.executor¶

SpanKind INTERNAL, child of spark.stage.N, emitted on the executor JVM.

Subject to FLARE_TRACE_GRANULARITY, FLARE_SLOW_TASK_MS, FLARE_RETRY_TASKS_ONLY and FLARE_MAX_SPANS_PER_TRACE. This span is built from TaskContext, so driver-only TaskInfo fields (host, locality, speculative) are not available to it.

Under FLARE_SLOW_TASK_MS, spans made inside a task, such as a JDBC query, and the task's log lines are children of the stage span, not the task span. Whether a task span is kept is only known when the task ends, and anything pointing at a dropped one would point at nothing.

Failures¶

error.type, error.message and the exception event.

A failed job, stage or task span carries error.message and status ERROR, and error.type when the failure's type can be recovered, which on stage spans it sometimes cannot (see below). Where a stack trace is available it is attached as an OTEL exception span event with exception.type, exception.message and exception.stacktrace (capped at 8000 chars).

Spark reports failures three different ways, which is why the detail differs by span:

error.type is the key you group failures by, so it is left unset rather than guessed. On a stage span Spark hands Flare only a formatted string, and that string names the wrapper before it names the cause:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0
failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7) (executor 1):
java.lang.ArithmeticException: / by zero

Reading the first class name here would report SparkException for practically every failed stage. Flare instead searches after the last most recent failure:, or after the innermost Caused by: when that is absent, so the reported type is java.lang.ArithmeticException. When neither anchor is present it falls back to the first fully-qualified name ending in Exception, Error or Throwable.

Spark 3.4+ also formats many failures as an error class with no exception name anywhere in the text:

[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 …
== SQL (line 1, position 1) ==
id div divisor

When no class name is found, that leading error class is used instead, so this stage reports error.type = DIVIDE_BY_ZERO. Spark error classes are stable, low-cardinality identifiers, which makes them a good grouping key rather than a lesser one. An exception class still wins when both are present. Only when neither is available is the attribute omitted.

A missing error.type means "not recoverable here", not "no error": error.message and spark.stage.failure_reason are still populated. Under FLARE_TRACE_GRANULARITY=tasks or all the task span for the same failure carries the structured version, read from ExceptionFailure's own fields rather than from text.

The four byte counts and peak memory come from TaskContext.taskMetrics(), which is private[spark] and can throw from outside org.apache.spark or in barrier mode. Flare logs at debug and emits the span without them rather than failing the task, so all five are absent together when that happens.