01 · Production Tools

Tools we run ourselves, free to audit

These run inside our own research operations every day. Same code, same versions.

v0.2.1 · LiveAgent Runtime · Python 3.11+ · stdlib only311 tests

fullstop

A zero-dependency runtime for always-on agents. Identity manifests, a default-deny policy gate with human approval, a fenced tool-call protocol, and a hash-chained activity log that shows tampering. Crash mid-run and it resumes without executing anything twice; that behavior is test-proven.

Built the week OpenAI announced Dots, as the self-hosted counterpart. Inspired by, not affiliated with, OpenAI.

In plain English

AI agents can take actions on their own, which is exactly what makes people nervous. fullstop is the safety gate: every action needs permission, nothing runs that we did not allow, and there is a tamper-proof diary of everything the agent did.

GitHub ↗PyPI (soon)MIT

v0.2.0 · Live on PyPIBYOK Gateway · OpenAI-compatiblepip install keysmith

keysmith

A local gateway for your own API keys. Multi-provider routing, key rotation with cooldown on failing keys, per-request spend logging, and one redaction point so key material never reaches a response, a log, or a traceback.

In plain English

Companies pay for AI by the request, and the passwords to those accounts are valuable. keysmith keeps those passwords in one guarded place, watches the spending, and makes sure a password never leaks into an email, a log file, or a screen.

GitHub ↗PyPI ↗MIT

02 · Research Artifacts

Every published number has a repo behind it

Datasets, harnesses, and raw data, frozen and versioned with a DOI. Null results included.

v1.0.0 · PublishedTR#8 · LLM-as-Judge BiasDOI 10.5281/zenodo.22949194

judge-bias-battery

A preregistered 741-call battery that measured five bias classes in LLM judging panels. All three judges flipped verdicts on answer order and preferred padded incorrect answers. Two over-graded their own output; one under-graded it. Ships with the raw data and a significance-gated correction model. 44 tests, MIT.

In plain English

When one AI grades another AI's work, can you trust the grade? We ran 741 grading calls, found the graders play favorites in five different ways, and built a fix. The full experiment is open for anyone to check.

GitHub ↗Study Page ↗DOI ↗

Shipping NextTR#5 · Dataset

Failure taxonomy of an agent-run lab

105 documented agent failures from real research operations, classified with κ 0.965 independent-classifier agreement, with a detection layer per failure class. The open dataset behind the census.

In plain English

We wrote down 105 times our own AI systems made mistakes during real research work, sorted by what went wrong and who caught it. It is an honest, checkable record of how AI actually fails in the wild.

Shipping NextP2 · ProbeBench

A drift score for edited models

120 rubric-graded prompts, three independent judges. Whether an edited model still behaves like itself becomes a number, not a vibe.

In plain English

When engineers tweak an AI's brain, how do you prove it still acts the same? ProbeBench turns that question into a score anyone can measure.

Shipping NextChem Slice A · Benchmark

Blind chemistry protocol extraction

Two extraction pipelines, one sealed answer key, and a practitioner auditor who never saw either pipeline's reasoning. Structured scientific extraction, scored blind.

In plain English

We tested whether AI can correctly pull lab instructions out of real chemistry papers. A human expert scored the answers without knowing which AI produced them, and the whole test is being published so anyone can reuse it.

03 · What Stays Private

What we keep private, and why

Our fleet orchestration, our release pipeline, and our calibration data stay private. That machinery is the engine; what you see on this page are the receipts.

If a release would need that machinery to be useful, it does not ship.

04 · Shipped

The changelog, newest first

2026-10-15

fullstop v0.2.1 — publish workflow wired, 311 tests gate every release. PyPI listing in flight.

Repo

2026-10-14

TR#8 preprint live — judge-bias-battery v1.0.0, raw data included, Zenodo DOI minted.

Study

2026-10-02

keysmith v0.2.0 — live on PyPI. pip install keysmith

Tool

Next up

TR#5 failure taxonomy, ProbeBench, Chem Slice A. One release at a time, each with a repo and a DOI.

Queued

Open source from 321 AI

Tools we run ourselves, free to audit

fullstop

keysmith

Every published number has a repo behind it

judge-bias-battery

Failure taxonomy of an agent-run lab

A drift score for edited models

Blind chemistry protocol extraction

What we keep private, and why

The changelog, newest first

A claim you cannot click is a claim you cannot check. Ours are clickable.