An independent research studio working on how generative video and AI systems are measured.
Open the live studio — the platform runs in your browser: 14 evaluation corpora, 1.4 million rated assets, reproducibility appendices generated from real run artifacts.
Benchmarks report a number. We work on what that number rests on: who rated what, how many times, on which scale, and whether anyone can recompute it from the data that was released.
We recover the individual votes behind a published reliability figure and recompute it, then report what the release does and does not support.
Rating protocols built so the coefficient can be estimated at all: panel size set from the target, shared items across raters, identifiers that persist.
Open software that turns a vote matrix into the declarations a reader needs, and flags the cases where no coefficient should be reported.
Sole-authored work on the reliability of evaluation, currently under review.
Sole author, under review, 2026. Index terms: inter-rater reliability; variance components and identifiability; human reference panels; LLM-as-a-judge evaluation; annotation budget allocation; calibrated diagnostics; reporting protocols.
Sole author, under review, 2026. Index terms: inter-rater reliability; construction choices in reliability estimation; rater screening and aggregation; panel size; multiverse analysis; reproducibility of published coefficients. Reproduction package: doi.org/10.5281/zenodo.22698801.
Seven peer-reviewed papers on cloud systems, network optimization and information-systems process analysis — IEEE Transactions on Cloud Computing, International Journal of Information Systems and Project Management, Procedia Computer Science, NBiS, Asia Modelling Symposium. Full list on DBLP.