JVM Modules¶

The JVM core is a Maven multi-module project under core/ that provides the Spark runtime components.

Module Structure¶

core/
├── pom.xml                    # Parent POM (groupId: io.drls)
├── drls-shim/         # Spark 4.1 compatibility shim
├── drls-main/         # Core runtime
└── drls-agent/        # Java agent for module access

Build Properties¶

Property	Value
`groupId`	`io.drls`
`version`	`0.1.0-SNAPSHOT`
Java	17
Spark	4.1.1
Scala	2.13.17
Ray	2.47.1
Jackson	2.17.0

drls-shim¶

Spark version compatibility layer. Contains SparkShims411 — a Scala singleton object.

Key design decision: The shim uses a direct singleton pattern (SparkShims411.method()), not SPI/ServiceLoader. All call sites reference the singleton directly.

object SparkShims411 {
  // Spark 4.1.1 specific implementations
}

drls-main¶

Core runtime providing:

AppMaster (DrlsSparkMaster) — Spark application master running as a Ray actor
Executor (DrlsExecutorBackendFactory) — Spark executor backend backed by Ray actors
ObjectStore — Integration between Spark's storage and Ray's object store
Streaming — JVM-side streaming bridge components

drls-agent¶

Java agent (-javaagent:drls-agent.jar) that opens required JVM module access for Spark internals.

Build¶

cd core
mvn clean package -DskipTests

Note

Requires Java 17. On systems with multiple JDKs:

JAVA_HOME=/path/to/jdk-17 mvn clean package -DskipTests

Produces three JARs:

JAR	Location
`drls-shim-0.1.0-SNAPSHOT.jar`	`core/drls-shim/target/`
`drls-0.1.0-SNAPSHOT.jar`	`core/drls-main/target/`
`drls-agent-0.1.0-SNAPSHOT.jar`	`core/drls-agent/target/`

The Python setup.py copies these JARs into the package during pip install.

Package Renames (from RayDP)¶

This project is derived from RayDP. Key renames:

Old (RayDP)	New (DRLS)
`com.intel.raydp.shims`	`io.drls.shims`
`org.apache.spark.deploy.raydp`	`org.apache.spark.deploy.drls`
`org.apache.spark.sql.raydp`	`org.apache.spark.sql.drls`
`RayDPSparkMaster`	`DrlsSparkMaster`
`RayDPExecutorBackendFactory`	`DrlsExecutorBackendFactory`
`RAYDP_*` env vars	`DRLS_*`
`SparkShimLoader.getSparkShims`	`SparkShims411` (direct singleton)

Intentionally preserved (JVM runtime references):

JVM actor names: raydp-executor-{id}
Prometheus metrics: raydp_stream_*