Skip to content

Installation

Prerequisites

Requirement Version Notes
Python 3.12+ Required
Java 17 For Spark JVM; JAVA_HOME must point to JDK 17
Ray 2.53+ Python package; Java JARs pinned to 2.47.1
Apache Spark 4.1.1 Bundled via Maven build

Install from PyPI

pip install drls

Extras

Install optional dependencies for specific features:

Extra Dependencies Use Case
drls[cli] click CLI commands
drls[agent] litellm, openai Agentic layer, LLM integration
drls[grpc] grpcio, grpcio-tools, protobuf gRPC server (optional, for external consumers)
drls[all] All of the above Everything
drls[dev] pytest, ruff, all extras Development and testing
# Install everything
pip install "drls[all]"

# Development install
pip install "drls[dev]"

Build the JVM Modules

The Spark runtime requires JVM JARs built from the core/ directory:

cd core
mvn clean package -DskipTests

Java Version

The build requires Java 17. If you have multiple JDK versions, set JAVA_HOME explicitly:

JAVA_HOME=/path/to/jdk-17 mvn clean package -DskipTests

This produces three JARs:

  • drls-shim-0.1.0-SNAPSHOT.jar — Spark 4.1 compatibility shim
  • drls-0.1.0-SNAPSHOT.jar — Core runtime (AppMaster, Executor, ObjectStore, Streaming)
  • drls-agent-0.1.0-SNAPSHOT.jar — Java agent for module access

The Python setup.py copies these JARs into the package during pip install -e ..

Verify Installation

python -c "import drls; print(drls.__version__)"

Next Steps

  • Quickstart — Run your first Spark-on-Ray query
  • Dev Setup — Set up the full development environment