Skip to content

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

[Unreleased]

Added

  • Multi-user RBAC system with 4 hierarchical roles: Admin, Lead Data Engineer, Data Engineer, Auditor
  • DB-backed user management with Argon2 password hashing and JWT token issuance (24h expiry)
  • Auth endpoints: register (bootstrap first user as Admin), login, refresh, me
  • Role-checking middleware on grouped routes — public, auditor, data engineer, lead, admin
  • Frontend auth flow: login/register pages, AuthProvider context, ProtectedRoute wrapper, role-filtered sidebar navigation
  • User-attributed audit logging across all mutating handlers (operations, streaming, config, user management)
  • Audit log viewer page with user context display and filtering
  • Pipeline versioning, environments, and approval workflow — definitions separated from running instances, immutable version snapshots, format version tracking for compatibility
  • Environment lifecycle: development → qa → uat → production → retired, with read-only enforcement on uat/prod/retired
  • Approval workflow for qa→uat and uat→prod promotions — Lead Data Engineer reviews with comment threads
  • Pipeline import/export as JSON with format version compatibility checking
  • Approval notification badges for Lead Data Engineers with pending request count
  • Pipeline list page with environment filter tabs and version history panel
  • Marimo notebook service — per-user isolated pods managed via Kubernetes API (kube-rs), HTTP + WebSocket reverse proxy, React page with iframe embed
  • Notebook API endpoints: start, stop, status with Data Engineer+ role gate
  • K8s manifests for notebook infrastructure: namespace, RBAC, network policy, PVC template
  • PostgreSQL support in Rust backend — runtime detection from DATABASE_URL scheme (sqlite:// or postgres://) via SQLx AnyPool
  • JWT validation in auth middleware — verifies HS256 signature, checks expiry, extracts claims (sub, exp, iat, role)
  • Project documentation site (MkDocs Material)
  • Rust backend with Axum REST server and PyO3 embedded Python bridge
  • Python gRPC server (optional, for external consumers)
  • React management console with pipeline builder, streaming monitor, and agent chat
  • CI pipeline with 8 parallel jobs

Changed

  • UI frontend moved from ui/ to ui/frontend/
  • Replaced tonic gRPC bridge with PyO3 embedded Python — 2-process architecture (Rust + React) instead of 3 (Rust + gRPC + React)
  • Config and streaming pipeline handlers rewritten as pure Rust (no Python dependency)
  • Removed FastAPI legacy backend (python/drls/api/)
  • Removed drls serve CLI command
  • Removed drls[api] and drls[ui] optional dependency groups

Fixed

  • R package distribution now works on Standalone, Mesos, and Kubernetes clusters (previously YARN-only)
  • Ray logging file prefix is now dynamic via ray.logging.file-prefix system property (previously hardcoded to java-worker)
  • Monitor page blank render — WebSocket proxy missing ws: true and pipeline list response envelope not unwrapped
  • RayDP to DRLS JVM references