Skip to content

Development

How the repository is built and tested. To contribute, see CONTRIBUTING.md.

Building

Requires Java 17 or 21, and sbt.

sbt test       # unit and integration tests
sbt assembly   # target/scala-2.13/apilytics.jar, with its dependencies shaded

The build targets Spark 4.2 by default. CI builds and tests the same sources against every supported Spark release; to try another locally:

sbt -DsparkVersion=4.0.4 test

A local cluster

docker/spark has a standalone cluster: a master, two workers, a history server, a Thrift server and Jupyter. The assembled jar and examples/ are mounted into every container.

sbt assembly
docker compose -f docker/spark/compose.spark.yaml up -d

With the cluster up, the helper script opens a spark-shell with an example API's catalog, building the jar first if it's missing. It doesn't start the cluster. The examples are pokeapi (the default), github, slack, and multi, which has GitHub and PokeAPI together:

./scripts/spark-shell.sh github

The Thrift server's catalogs are set in compose.spark.yaml:

environment:
  - CATALOGS=github,pokemon
  - CATALOG_GITHUB_CONFIG=/opt/spark/examples/github/github-config.conf
  - CATALOG_POKEMON_CONFIG=/opt/spark/examples/pokeapi/pokeapi-config.conf

The Docker image

The published image is built from docker/dist/Dockerfile, from the repository root, after sbt assembly:

sbt assembly
docker build -t apilytics:latest -f docker/dist/Dockerfile .

Releases are tagged X.Y.Z, X.Y and latest, so latest is always the newest release.

Development builds

The latest commit on main is always published too, for trying out unreleased changes without building anything locally:

  • the jar, on the dev pre-release: https://github.com/Neutrinic/apilytics/releases/download/dev/apilytics_2.13-dev.jar, which works with --jars or on a Databricks volume
  • the image, as ghcr.io/neutrinic/apilytics:dev, and under the commit's short SHA

The dev release notes say which commit they're from. Development builds are not releases: nothing goes to Maven Central, and they can change with every push. Running the Dev builds workflow by hand on a branch publishes dev-<branch> the same way, to try a change before it merges.

Benchmarks

apilytics-bench holds a synthetic paginated API whose answers are known: fixtures of 100 MB to 10 GB, with clean and deliberately messy variants, and profiles for latency, rate limits, server errors and TCP resets. It runs on any Debian or Ubuntu host. Its GCP scripts measure apilytics against DuckDB and a plain JVM client at capped link speeds, and run against the dev jar by default, so a change on main can be measured before it's released. The Performance page summarises the results.

This site

The site is built with Zensical from docs/ and zensical.toml:

pip install zensical==0.0.67
zensical serve

It's published per release, one version per minor release, with a version selector. Pull requests that touch the docs build them in strict mode.

Architecture

Config (HOCON) → OpenAPI parser → Schema mapper → Arrow schema
                                                        ↓
Spark catalog ← Tables ← ScanBuilder (pushdown) → HTTP client
                                                        ↓
                                              Paginator (fs2 Stream)
                                                        ↓
                                              Arrow converter → ColumnarBatch

Stack

  • Scala 2.13, built against Spark 4.0 to 4.2
  • http4s Ember client, on cats-effect and fs2
  • circe, with circe-pointer for JSON pointers (RFC 6901)
  • swagger-parser, for OpenAPI
  • Typesafe Config, for HOCON
  • Apache Arrow, the one Spark ships, for columnar batches