Skip to content

Kubernetes

Verified on k3s with Spark 4.0.4, 4.1.3 and 4.2.0 through spark-submit in cluster deploy mode, and with the Kubeflow Spark Operator 2.5.2 on Spark 4.2.0. Both the catalog and format() were run. Executors ran on all three nodes, and streaming was exact.

spark-submit

spark-submit --master k8s://https://kubernetes-api:6443 --deploy-mode cluster \
  --conf spark.kubernetes.container.image=your-spark-image \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --packages io.github.neutrinic:apilytics_2.13:1.0.1 \
  --conf spark.jars.ivy=/tmp/ivy \
  --conf spark.kubernetes.driver.node.selector.kubernetes.io/hostname=node-with-config \
  --conf spark.kubernetes.driver.volumes.hostPath.apilytics.mount.path=/etc/apilytics \
  --conf spark.kubernetes.driver.volumes.hostPath.apilytics.mount.readOnly=true \
  --conf spark.kubernetes.driver.volumes.hostPath.apilytics.options.path=/srv/apilytics \
  --conf spark.sql.catalog.api=com.apilytics.spark.RESTCatalog \
  --conf spark.sql.catalog.api.config=/etc/apilytics/api.conf \
  local:///opt/jobs/your-job.py

The YARN recipe doesn't carry over: --files and local Maven repositories are local to the machine you submit from. Kubernetes refuses local files unless spark.kubernetes.file.upload.path points at shared storage. What works:

  • Jar: --packages is resolved by the driver pod, from a Maven repository it can reach over HTTP: Maven Central, or your mirror. Point spark.jars.ivy at a writable path such as /tmp/ivy. Executors get the jar from the driver, and nothing needs uploading.
  • Config and spec: they must be inside the driver pod, and the catalog's config is the path inside the container. The verified runs mounted them with a hostPath volume, as above, which needs the files on the node the driver is placed on. In production, put the same files at the same path by baking them into your image, or by mounting a ConfigMap through a driver pod template. If the mount renames the files, set openapi to the spec's absolute path inside the pod, or a URL, because a relative spec is looked up beside the config under its original name.
  • Service account: the driver creates and deletes its executor pods, so it needs a service account allowed to do that in its namespace. Set it with spark.kubernetes.authenticate.driver.serviceAccountName; without it, Spark uses the namespace's default account, which usually can't. The verified runs used an account with a namespace-scoped Role over pods, services, configmaps and persistent volume claims.
  • Streaming checkpoints: a checkpoint on the driver pod's disk lasts only as long as that pod. A stream that has to survive a restart needs object storage or a persistent volume.

Spark Operator

The same job as a SparkApplication, for the Kubeflow Spark Operator:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: apilytics-job
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: your-spark-image
  mainApplicationFile: local:///opt/jobs/your-job.py
  sparkVersion: "4.2.0"
  deps:
    packages: ["io.github.neutrinic:apilytics_2.13:1.0.1"]
  sparkConf:
    spark.jars.ivy: /tmp/ivy
    spark.sql.catalog.api: com.apilytics.spark.RESTCatalog
    spark.sql.catalog.api.config: /etc/apilytics/api.conf
    # Placement and mounts declared in sparkConf work whether or not the operator's
    # webhook is enabled; the equivalent fields under spec.driver need the webhook.
    spark.kubernetes.driver.node.selector.kubernetes.io/hostname: node-with-config
    spark.kubernetes.driver.volumes.hostPath.apilytics.mount.path: /etc/apilytics
    spark.kubernetes.driver.volumes.hostPath.apilytics.mount.readOnly: "true"
    spark.kubernetes.driver.volumes.hostPath.apilytics.options.path: /srv/apilytics
  driver:
    cores: 1
    memory: 1g
    serviceAccount: spark
  executor:
    instances: 3
    cores: 2
    memory: 1g
  restartPolicy:
    type: Never
  • Declare placement and volumes in sparkConf. The operator's own fields for them, such as spec.driver.volumeMounts and nodeSelector, only take effect when its mutating webhook is enabled. A run that relied on them under an operator without the webhook failed: the driver couldn't find its files. sparkConf entries are applied by Spark itself, so they work either way.
  • Use a service account that can create executor pods in the namespace, set with spec.driver.serviceAccount. It needs the same permissions as for spark-submit above.
  • deps.packages and deps.repositories are --packages and --repositories. The driver pod resolves them, as above.