Abstract

Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.

Method

Prior statistically-rigorous methods for ranking policies typically satisfy at most one of the following three properties at a time. AnyRank is designed to hold all three simultaneously.

Experiments

AnyRank is evaluated on real-world policy evaluation benchmarks, including RoboArena and the TRI-LBM dataset, totaling over 7000 hardware rollouts across 20+ policies. We organize our findings around four research questions.

• Statistical Reliability
• Efficiency
• Impact of Priors
• Scalability

Does AnyRank maintain family-wise error rate (FWER) control, even under incorrect priors — and does this guard against the ranking failures seen in Bradley–Terry baselines? On RoboArena evaluation data, Bradley–Terry methods can be both miscalibrated (e.g., confidently ranking one policy above another when the opposite is more likely true) and temporally inconsistent (flipping the ranking as more trials are collected). AnyRank avoids both failure modes: its rankings at any stopping time remain correct with probability at least \(1-\alpha\) and never contradict a ranking produced with more data.

	Method	RoboArena4	RoboArena7
Ours	AnyRank	A > E > F > G	A > E > F > G, {A,B,C,D}, {B,C,D,E}
	AnyRank0	A > E > F > G	A > E > F > G, {A,B,C,D}, {B,C,D,E}
	AnyRank∞	A > E > F > G	A > E > F > G, {A,B,C,D}, {B,C,D,E}, {B,C,D,F}
Baselines	Bonferroni	A > E > F > G	A > E > F > G, {A,B,C,D}, {B,C,D,E}
	Weighted Bonferroni	{A,E} > F > G	{A,B,C,D,E} > F > G

Table 1. Ranking DROID policies from RoboArena with all available data. Policies: π₀-FAST (A), PG-FAST (B), PG-FAST-Specialist (C), PG-FSQ (D), PG-Diff (E), π₀ (F), PG-Bin (G). Groups in curly braces are not statistically distinguishable by that method.

Does AnyRank improve statistical power — both hypotheses rejected and total sample complexity — over other FWER-controlling methods? We measure efficiency via Rejection Count (how many pairwise comparisons are resolved) and Sample Complexity (how many hardware rollouts it takes to resolve them). Across RoboArena and TRI-LBM, AnyRank and its variants reject more hypotheses using fewer samples than standard and weighted Bonferroni corrections.

Does the sample complexity of AnyRank improve with more accurate priors? With weak priors (e.g., world-model rollouts with low correlation to real-world performance), AnyRank performs comparably to its no-prior ablation AnyRank₀. But with a strong prior — such as a world model fine-tuned on the specific task, achieving r = 0.89 correlation with real-world performance — AnyRank requires substantially fewer hardware trials to reach a full ranking.

How does the sample complexity of AnyRank scale with the number of policies? Ranking more policies increases the number of pairwise comparisons quadratically and pushes policy performance gaps closer together on average, making the underlying ranking problem fundamentally harder for any statistically valid method. On synthetic data ranking up to 20 policies, and on all 15 policies from RoboArena (including several retracted for having too few trials), AnyRank uniformly reduces sample complexity relative to standard Bonferroni corrections at every scale we test, while remaining robust to policies with limited evaluation evidence.

BibTeX