BENCHMARK eva02-imagenet1k-v3

Rules and runtime ABI

Pass and score

An official run passes with at least 9,900 exact top-1 matches against the one frozen deterministic EVA-02 reference over the full private 10,000-image cohort. Agreement is a pass/fail gate.
Passing runs rank only by one measured end-to-end wall-clock duration; earlier submission breaks an exact nanosecond tie.
The leaderboard keeps each entrant’s best passing official run.
The cohort was sampled once from the canonical 50,000-image validation set with one private high-entropy benchmark seed. Its content hashes are frozen, but its seed, member list, canonical filenames, and reference answers remain administrator-only.
A public 1,000-image practice cohort with published per-image reference answers is available to entrants for rehearsal. It was drawn from the same canonical validation universe under an independent published seed, so incidental overlap with the private cohort reveals nothing about membership. Practice results never touch the official leaderboard, and the validation-data boundary below governs every use.

Schedule and attempts

Official submissions open at 2026-08-10 00:00:00+09:00 and close at 2026-08-31 18:00:00+09:00. The start is inclusive and the end is exclusive.
Environments may be built before opening, but no new environments are accepted after closing.
Each entrant receives 2 official attempts per Asia/Seoul calendar day. An attempt is charged when an upload is accepted.
Attempts are refunded only when the platform fails or cancels the run; entrant build or scoring failures count.
Environment builds are attempt-free: a failed or slow Dockerfile build costs nothing but time and may simply be retried, and a ready environment is reused by every later submission. The build.sh step instead runs inside the charged attempt, so a submission build failure or timeout consumes it — put long downloads in the Dockerfile.
Up to 2 environment builds run at the same time, and the site stays fully open while they do; builds only ever wait for an active score, which always has the machine to itself.
Every build — environment or build.sh — is capped at 30 minutes, and the scored run at 16 minutes, so no single entrant can hold the machine longer than that.
Each entrant may keep up to 2 environments. Environment and submission images may each be up to 16 GiB. Remove an unused environment before creating another.

What is timed

A trusted monotonic clock starts immediately before the supervisor forks the contestant entrypoint and stops only after its entire descendant tree is reaped.
The timed interval therefore includes process startup, model and engine loading, JPEG access and decoding, preprocessing, transfers, inference, output writing, all descendant work, and process completion.
The whole scored run must finish within 16 minutes. The ceiling sits just above the public FP32 starter kit’s own measured time, so a competitive submission must at least match the starter; a run still working at the limit is stopped, fails, and consumes the attempt.
Everything before that boundary is off the clock: waiting in the queue, the environment image build, the build.sh submission build, digest and machine-health verification, cohort staging and page-cache prewarming, container creation, Docker control-plane noise, and the supervisor’s own startup.
Everything after it is off the clock too: reading your output, comparing it against the reference, and publishing the verdict add nothing to your time.
Untimed preparation can never touch the contest inputs: the private cohort is staged only for the scored run, so neither build ever sees an image, a manifest, or an answer. Time saved off the clock is compilation, installation, and model preparation, never inference.
An outer host timer enforces the timeout and must contain the trusted duration.
Only one scoring container runs at once. The host refuses to begin unless its frozen software versions, idle GPU, memory/disk reserves, and thermal/CPU-idleness gates pass.

Submission ABI

/submission/run MANIFEST IMAGE_ROOT OUTPUT

CONTEST_MANIFEST=/contest/input/manifest.tsv
CONTEST_IMAGE_ROOT=/contest/input/images
CONTEST_OUTPUT=/contest/output/predictions.i32

The manifest has an opaque request ID and relative image path separated by one tab.
Output is exactly one little-endian signed int32 class index in [0, 999] per manifest row, in order.
Every run in a benchmark cohort uses the same runner-controlled image permutation for timing fairness, while filenames and request IDs are freshly randomized.
The public EVA-02 starter kit includes a copy-ready optional Python environment and a known-correct source archive that implements this ABI.

Permitted approaches

Any model, checkpoint, and optimization approach is allowed, subject only to the technical and validation-data rules on this page.
There is no prescribed programming language, framework, compiler, model format, execution engine, or base image.
Every Dockerfile stage may use an ARM64 image from an allowed public registry; the supplied CUDA/PyTorch image is only a starter and is not required or attested.
The environment must provide /bin/sh for the untimed source build, and the final predictor must perform general inference from image content.

Validation-data boundary and review

No data derived from the ImageNet-1K validation set (images or any part of them, labels, reference predictions, content hashes, embeddings, statistics, or any transformed, aggregated, or learned representation of them) may be contained in a submission or any artifact it uses, consulted at inference time, or used to create, train, fine-tune, distill, calibrate, or otherwise fit a model, checkpoint, or artifact.
Evaluation is the one sanctioned use: entrants may score a candidate on official runs, on any practice materials the contest publishes, or on their own validation-set evaluations, and may use the resulting metrics and failure observations to guide development. Evaluation results may inform decisions; validation data itself may never enter an artifact.
During a run, each runner-supplied input image may be used only to produce that image's output for that run. Recognizing or matching inputs against stored validation-set content is consultation and is forbidden.
Entrants must be able to establish this boundary for their source, environments, models, checkpoints, and built artifacts. Apparent winners must pass provenance and artifact review; inability to establish compliance is grounds for disqualification and removal of all of the submission's runs from the leaderboard.

Execution and cache policy

Build commands may reach the public internet but cannot reach the Docker host, lab/private networks, link-local or metadata services, CGNAT/Tailscale, multicast, or reserved destinations. Remote Dockerfile ADD and unreviewed image registries are forbidden.
Official scoring has no network. Inputs and checkpoint are read-only. Scoring runs as a non-root UID with no Linux capabilities, a read-only root filesystem, bounded memory/PIDs/tmpfs, and no Docker socket or host secrets.
Before timing, the trusted runner preloads the same immutable JPEG corpus, checkpoint, and immutable runtime-image files into the host page cache for every entrant. This deterministic warm-cache policy replaces an irreproducible “best effort” cold-cache flush and gives every one-shot run the same starting state without requiring a reboot.
Submission-writable caches are always empty. Bundled precompiled artifacts are allowed; runtime JIT compilation is timed.

Feedback and retention

Entrants receive status, wall time, aggregate agreement, immutable digests, and size-limited build/runtime log tails.
They never receive reference classes, per-image mismatches, the private run mapping, or another entrant’s artifacts.
Entrants can download their own source; administrators can download source for review, and access is audited.
Source archives, Dockerfiles, bounded logs, results, digests, review records, and audit events are retained through the contest end.