An embedded storage engine for Rust and Python. Keys sort byte-wise, batches commit atomically and iterators seek, on a local disk or straight on S3, GCS or Azure. Every commit is a Lance version, so several processes can write at once and any version can be opened again.
The same store from Rust and Python
A database is a path or a URI. Writes are visible at once, logged locally before they return, and committed to storage in batches.
import kv0
db = kv0.DB("s3://bucket/prefix/db")
batch = kv0.WriteBatch()
batch.put(b"run/7/loss/0003", b"0.25")
batch.delete(b"run/7/tmp")
db.write(batch)
it = db.iteritems()
it.seek(b"run/7/")
for key, value in it:
...use kv0::kv::{Db, Options, WriteBatch};
let db = Db::open("/data/db", Options::default())?;
let mut batch = WriteBatch::new();
batch.put(b"run/7/loss/0003", b"0.25");
batch.delete(b"run/7/tmp");
db.write(batch)?;
let mut it = db.iter();
it.seek(b"run/7/");
for (key, value) in it { /* ... */ }Measured, with the method
Medians of five runs on one machine (i5-13500, NVMe), local disk. The full tables, including what got worse, are one link away.
| Reopen 10M keys | 705 ms, 1.0 GB | from a view snapshot; 5.4 s without one |
|---|---|---|
| 100k values of 16 KiB, reopen | 20 ms, 90 MB | large values stay in Lance until read |
| Seek + 100 × next | 11 µs | p50, 1M keys |
| Tracker run, single puts | 386k puts/s | 520k puts into two databases, each logged before it returns |
| Crash recovery | 218 ms | 200k unflushed puts, all recovered |
Guarantees, written down
- A write survives a killed process: it is in a local write-ahead log before the call returns.
- A
WriteBatchis atomic, for readers and in storage. - Writers on several machines append without coordination; conflicting keys resolve by sequence.
- Every commit is a version; tagged versions survive cleanup.
Tracking tables
- Runs, parameters and metric series in columnar tables on the same storage.
- Step-range reads across many runs; automatic compaction while logging.
- Async Rust API for servers, a small Python API for training code.