Installation¶
Prerequisites¶
| Requirement | Version | Notes |
|---|---|---|
| Python | 3.12+ | Required |
| Java | 17 | For Spark JVM; JAVA_HOME must point to JDK 17 |
| Ray | 2.53+ | Python package; Java JARs pinned to 2.47.1 |
| Apache Spark | 4.1.1 | Bundled via Maven build |
Install from PyPI¶
Extras¶
Install optional dependencies for specific features:
| Extra | Dependencies | Use Case |
|---|---|---|
drls[cli] |
click | CLI commands |
drls[agent] |
litellm, openai | Agentic layer, LLM integration |
drls[grpc] |
grpcio, grpcio-tools, protobuf | gRPC server (optional, for external consumers) |
drls[all] |
All of the above | Everything |
drls[dev] |
pytest, ruff, all extras | Development and testing |
Build the JVM Modules¶
The Spark runtime requires JVM JARs built from the core/ directory:
Java Version
The build requires Java 17. If you have multiple JDK versions, set JAVA_HOME explicitly:
This produces three JARs:
drls-shim-0.1.0-SNAPSHOT.jar— Spark 4.1 compatibility shimdrls-0.1.0-SNAPSHOT.jar— Core runtime (AppMaster, Executor, ObjectStore, Streaming)drls-agent-0.1.0-SNAPSHOT.jar— Java agent for module access
The Python setup.py copies these JARs into the package during pip install -e ..
Verify Installation¶
Next Steps¶
- Quickstart — Run your first Spark-on-Ray query
- Dev Setup — Set up the full development environment