lizhiyao/oh-my-knowledge

lizhiyao

TypeScript13updated 3mo ago

Evaluation framework for LLM knowledge inputs — prompts, RAG corpora, skills, agent workflows. Fix the model, vary the artifact. Built-in statistical rigor: bootstrap CI, Krippendorff α, length-debias, saturation curves.

Claude

agent-evaluationaibenchmarkbootstrap-ciclaudeclaude-codeevaluation-as-codeevaluation-frameworkknowledge-engineeringkrippendorff-alphallmllm-evaluation

View on GitHub

README

oh-my-knowledge

English | 简体中文

Did your prompt actually get better? A/B test your prompts and skills with statistical rigor — bootstrap CI and length-debias on by default, Krippendorff α the moment you add a gold set.

📖 Full documentation: oh-my-knowledge.pages.dev (searchable, English / 简体中文)

Quick start

npm i -g oh-my-knowledge
omk init demo && cd demo
omk eval --control code-review-v1 --treatment code-review-v2

That's it — no editing required. omk init scaffolds two skill variants and three sample cases; omk eval runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.

The CLI notifies you when a newer version is available (at most once per 20h); set OMK_SKIP_UPDATE_CHECK=1 to silence it permanently.

Walkthrough: 5-minute quickstart guide (recommended for first-time users).

Deeper: who omk is for · CLI reference · how it works · eval sample format · executors · artifact layout

Use inside AI Coding Agents

Ins