Dataweaver is a privacy-first AI data analyst I lead at Techcora: an autonomous agent that cleans, reconciles, and transforms data from plain-English instructions. It shipped 40+ releases in 5 months and runs live inside a European payments company.
“Raw data never leaves the building. The AI plans against a redacted profile, and generated code runs in a local sandbox.”
— Gagan Raj
CONSTRAINTS
- Regulated industry: customer data cannot touch a third-party server, ever
- The agent must recover from its own bad code without a human in the loop
- One engineer owning product, architecture, and releases end-to-end
- Live production workflows at the launch customer, iterated forward-deployed
THE BUILD
Enterprise data teams in regulated industries have a contradiction to live with: the tools that could automate their cleanup work are cloud AI tools, and their data is the one thing that cannot go to the cloud. When Techcora made Dataweaver its flagship, that contradiction became my problem to solve.
The answer is a two-layer privacy architecture. A local Profiler reads the raw data and builds a PII-redacted statistical profile. The Weaver agent plans transformations against that profile, never the data itself. The code it generates runs in a local sandbox with self-healing execution: when a step fails, the agent reads the error, patches its own code, and retries. The customer watches an agent work; the data never moves.
Under the product sits a Rust streaming layer I built for it: SeqStream, a custom primitive crate with a ring buffer and a supersede protocol, plus a hybrid Channel-IPC data plane for live log tailing and PII egress gating on everything that leaves the process. The connector layer speaks PostgreSQL, MySQL, SQLite, and GCP logs, and imports HTTP API definitions straight from Postman.
DECISIONS — INCLUDING THE REJECTED ONES
Runtime
CHOSE Tauri desktop app: Rust core, React UI, Python execution sandbox
REJECTED Cloud SaaS with a browser frontend
The entire value proposition is that raw data stays on the machine. A server-side product cannot make that promise, whatever the marketing says.
What the AI sees
CHOSE A PII-redacted profile built by a local Profiler
REJECTED Sending sampled rows to the model for context
Sampling leaks. A statistical profile gives the agent enough to plan against and gives compliance teams an answer they can sign.
Failure handling
CHOSE Self-healing sandbox: the agent reads its own stack traces and patches the code
REJECTED Surfacing every failure to the user for manual retry
An analyst who has to debug generated Python is doing the job the product promised to remove.
Streaming internals
CHOSE A custom SeqStream crate: ring buffer plus supersede protocol
REJECTED Stitching together existing channel and buffer crates
Live log tailing needs newer data to replace stale data mid-flight. That supersede semantic did not exist off the shelf, so I built the primitive and load-tested it.
OUTCOME
- 40+ releases shipped in 5 months
- Live with the launch enterprise customer, a European payments company
- Full connector layer: PostgreSQL, MySQL, SQLite, GCP logs, HTTP APIs
- Zero raw-data egress by architecture, not by policy
CRAFT LEDGER
- Two-layer privacy architecture
- Self-healing code execution
- SeqStream ring buffer
- Channel-IPC data plane
- PII egress gating
- Load-test harness