Noise¶

The OpenTelemetry agent instruments everything in the JVM, not only Spark. Every platform makes its own calls from the driver and executors, and each would become a separate one-span trace next to Flare's:

What Flare does about it¶

By default Flare drops any span that has no parent and is not one of Flare's spark.* spans, on the driver and on executors. After this the verified platforms show one or two driver traces per run.

What is kept:

• Calls made inside Spark work, when you have turned that instrumentation back on. A JDBC query or an S3 read inside a task has the task span as parent, so it is kept and nests under the task.
• Work a task hands to another thread, through an ExecutorService or CompletableFuture: the agent carries the task's context there, so the child keeps its parent. A raw new Thread gets no context; its spans have no parent and are dropped. Without this filter they were exported as separate traces, never under the task.
• Requests that continue another trace. A request arriving with a traceparent header has a parent, so it is kept. This is why Databricks runs still show one small extra trace: Databricks sends its own traceparent on some calls into the driver.

Turning it off¶

Set FLARE_DROP_NON_SPARK_ROOTS=false when a Spark JVM does its own traced work that should start traces, such as a driver that also serves HTTP.

Since 1.3.0 Flare also turns off the agent's own instrumentation by default, so most of the sources above are not traced at all: see Agent instrumentation. The filter still matters once you turn an instrumentation back on. It then drops that library's calls with no parent and keeps those made inside Spark work, which nest under their task.