Reworld

Product definition

Required properties

Terminology

Generator

The permanent software process that defines which kinds of Reworld worlds can exist.

World

One freshly selected set of hidden rules used in an evaluation batch.

Starting case

One hidden starting condition under those rules. The visible panel always begins the same way.

Retired release

A completed world package containing its rules, starts, checker, histories, hypotheses, and best-known strategies.

Challenge

A requested visible result, plus limits such as a maximum number of steps.

Submitted agent

The immutable program and saved starting state used for an official capability report.

Capability landscape

The horizontal axis shows how broadly each system is tested, from one task to transfer across unrelated real systems. The vertical axis shows how much success requires adapting to new evidence about hidden rules. Positions reflect published evaluations, not claims about how an agent reasons internally.

Positions are approximate, based on published evaluation scope. The shaded Reworld region is a design objective, not a measured capability or rank; Reworld earns a real position only after its internal and external transfer studies report results.

Research map

The groups below separate related claims that are often conflated: discovering rules, generating curricula, adapting within a world, choosing informative interventions, and transferring outside simulation.

Reworld tests whether practice on unfamiliar hidden-rule worlds helps agents adapt to unfamiliar real systems. The real systems must be independent of Reworld and must not have been used to tune it. The study must test which parts of the curriculum cause any benefit, rather than assuming that delays or hidden rules make the worlds realistic.

Toward an AGI/ASI-relevant benchmark: approximating, never proving

Proposed levels must demonstrate evidence-responsive learning and control, breadth, calibrated upper-tail discrimination, and external validity. Each level requires a named gate and a typed result; synthetic hardness or failed attacks alone establish neither AGI nor ASI. H-S8B is the supporting executable-knowledge/handoff track. Immediate priorities are bounded adapter repair and development calibration; the next research tests restore autonomous experiment choice and matched training interventions. F10 remains decisive for the transfer claim.

Comparison with other intelligence evaluations

Test many independently developed AI systems on both Reworld and a fixed set of other benchmarks. First ask whether their rankings broadly agree. Then ask the harder question: whether Reworld predicts performance on new benchmarks and real systems beyond what model size, compute, tools, agent setup, and existing benchmark scores already explain.

Higher-ceiling hypothesis. ARC-AGI-2 and ARC-AGI-3 help calibrate Reworld, and fresh generated worlds may keep separating stronger systems after ARC scores cluster near their human reference points—but that extra separation counts only if it predicts results on new hard evaluations and real systems. Until then, the defensible claim is only “superhuman performance within Reworld.”

Validity evidence is collected in stages rather than through one huge study: the early Phase 1C correlation probe, then agreement and added predictive value measured on the systems already evaluated in the transfer study, reusing its benchmark scores. A separate large cross-benchmark study is out of scope for this proposal.

Why transfer is plausible—and where it has failed

The closest precedents split cleanly. Models trained purely on diverse synthetic generators—TabPFN on generated causal datasets, Panda on evolved chaotic systems—transfer to real data they have never seen. The working explanation is that generator diversity makes memorization impossible, so the only strategy that fits training is to become a general inference procedure. In contrast, adaptive policies trained on narrow task families reliably fail outside them, because identifying which known task is active is cheaper than learning to infer. One small interactive study bridges the camps: agents trained only on simple random synthetic environments matched agents trained on real ones. Reworld is designed to put interactive learning in the regime that transfers: one generator with unbounded structural variety, admission rules that reject worlds solvable by the tested shortcut strategies, and tests, fixed in advance, for the shortcut signature. The untested step is interaction at scale—a wrong prior produces uninformative experiments and therefore misleading data—so the plan tests transfer early instead of only at the end. An exploratory pilot has now observed this failure directly at the smallest possible scale: on eleven-state worlds whose rules change mid-test, an agent with a scrambled experiment record outperformed one holding an intact record of the old rules—confidently wrong knowledge was worse than no knowledge.

Early risk-reduction pilots (Phase 1C)

• Passive-transfer pilot. Train a small model only on generated Reworld histories and test whether it predicts real dynamical systems better than the same model trained on generic synthetic data. A worse result forces generator revision before further investment.
• Prior-gap measurement. Measure how far generated world behavior sits from real system behavior in a shared representation, and track that gap across generator versions. A pilot on 111 state machines learned from real software across six families already quantifies the danger: real machines differ from size-matched random ones on nearly every measured trait—inert actions, skewed outputs, and chain-like structure separate every family, and distinguishing depth separates five of six—so a naive random generator would badly miss reality on these axes. A follow-up check on 1,285 independently learned TLS machines replicates every signature at ten times the scale except inert-action density, which turns out to depend on the learning harness rather than the machine. An exploratory extension adds the strongest signature yet measured: in real machines, 72% of reachable situations are one-way doors that cannot return to the start, versus roughly zero in random machines—a separation robust to the harness differences that unsettled inert-action density.
• Cross-generator test. Train on Reworld worlds, then evaluate on hidden-rule machines from an independently written generator. Surviving this shift is a necessary condition for any real-world claim. If it fails, a paired comparison with oracle-chosen experiments shows whether the agent asked bad questions or misread the answers.
• Correlation probe. Check whether existing models’ Reworld performance predicts their performance on small real system-identification tasks, controlling for model scale. A desk stage comes first: compute the same correlation from already-published scores before running anything new.
• Real-machine probe. Test the learner directly on published state machines learned from real systems—protocol software, bank cards, embedded controllers—which match the first worlds in form and approximate size, through fixed input/output conversions reported two ways so the conversion cannot fake the result. This gives a real-system signal years early; the probe touches only already-studied families, while sealed reserve families are spent exactly once each in the head-to-head test below.
• Calibration head-to-head. Train one learner on random practice machines and an identical learner on machines built to match the measured statistical personality of real systems, then test both on sealed real machines neither has seen. If realistic practice wins, the core idea has its first direct evidence; if the two tie, the measured traits do not matter and the world recipe must change. The reference families are endorsed by the same independent researchers who endorse the control curricula, so the generator’s builders do not solely curate the reality standard it is measured against.

The passive-transfer, cross-generator, and calibration pilots gate progress: each pilot’s pass threshold, sample size, and analysis are fixed in a public preregistration before it runs, and all three must show positive signal before the larger world design is funded. The sealed reserve machines are examined at most twice—once per tier, whatever the reason for the retry—while the other pilots rerun on public data; further attempts need a genuinely fresh corpus, and any generator revision voids earlier confirmations, restarting the freeze. Several pilots need only public data and simple stand-alone samplers, so they can run alongside the first exact experiment rather than after it. They cannot prove transfer—only the final matched training study can—but they can disprove it cheaply, years earlier.

Training transfer test

Use at least two independent base-model families and eight training runs for every base-family × curriculum combination. Match architecture, optimization, training data, compute, interfaces, downstream practice, and checkpoint selection. Evaluators should not know which curriculum a system received.

Before results are seen, an independent custodian selects about 12 separately configured systems in each of at least four domains. Repeated episodes on one system are not independent samples. The analysis must account separately for variation across training runs and external systems. The input conversions connecting each real system to the agents are authored or endorsed by the same independent researchers, frozen before results are seen, and the effect must hold under at least two independently authored conversions per domain. Safety is tested separately to show that it is not meaningfully worse—a noninferiority test—and cannot be traded for capability.

Possible external test domains

Proposed first-prototype interface

Choosing the interface size

No interface size is best for every world and budget. Reworld uses the smallest tested interface that shows no meaningful performance ceiling within the stated limits on worlds, agents, interaction length, and compute. Too few options waste steps encoding commands; too many make blind search easy.

Changing only one control does not guarantee that only one hidden mechanism changes; that is true only in worlds built to separate those effects. Harder worlds may require combined controls, patterns across indicators, and actions spread over time.

Candidates from one two-setting control up to three three-setting controls with four indicators were compared before freezing. The winner is adopted as a documented design decision rather than the output of a separate optimization study—no successful synthetic-training system optimized its interface—and the checks that matter ride along with the main test: every headline result reports how much information each control and indicator actually carried, how many steps went to encoding commands, and whether the result survives renamed symbols. Checks using multi-step compound actions begin in the second phase, where exact search no longer covers them. If these checks ever reveal a real ceiling, the interface changes only as a separately named benchmark version.

Example interface

The panel below shows what the fixed interface might feel like, running a small hand-written toy rule set rather than a generated world.

A run has four hidden parts—a limited internal state, a seeded starting condition, one set of world rules shared by every starting case, and a fixed publicly documented step procedure—plus the visible controls and indicators of the previous section. The starting seed is published only when the world retires.

Information boundary

Exact prototype first

In Phase 1, each world is a small deterministic machine—formally, an input–output transducer—with eight hidden states, nine actions, and eight possible observations. Because it is finite, researchers can calculate the best fixed action sequence and compare it exactly with small controllers that react to feedback.

Scaling up: combined machines first

If the exact test passes, worlds grow by combining and linking many small exact machines—shared hidden parts, chance outcomes, and slowly drifting settings—so that baselines and discovery paths can still be computed exactly far beyond eight states. This mirrors how the successful synthetic-training systems scaled: simple compositional recipes, not expressive world-description languages.

Correctness and safety checks

Every generated rule must leave the world in another valid state. The generator cannot emit arbitrary program code.

The number of hidden levels will never be part of any public result: if a simple implementation and a layered implementation behave the same at the interface, they count as equally difficult.

What scales—and what remains fixed

The core curriculum should not become harder by adding controls, adding indicators, or increasing their alphabets. Those changes would give agents a wider channel and a different machine to operate, not a deeper learning problem. The slot counts, values, timing, and reset semantics are fixed before launch. If the prototype shows that the interface is inadequate, it is changed before the permanent curriculum begins—not between difficulty levels.

A small interface can still support varied behavior over time. Sequences of actions can configure mechanisms inside a world, and later indicator sequences can report their results, without adding new external controls or indicators.

Difficulty has several independent dimensions, so one level number cannot describe it fully. Candidate tests should vary one dimension at a time before combining them. More hidden nodes, rules, connections, randomness, or steps count as harder only when they demand better learnable prediction or control.

Worlds are grouped into preregistered difficulty bands. The lowest bands carry measured human reference points—ordinary players first, expert teams at the top of the human range—collected through the visual panel, so the human ceiling is located empirically. Bands above it are defined by certified demands (minimum experiments, memory, jointly coupled mechanisms) and scored as distance from the certified optimum, with no human comparison claimed; the two scales must agree where they overlap before any superhuman band is quoted. A result is a band profile, never a percentage of AGI.

The curriculum cycle

1. 1
   Commit agents

   Each submitted program is locked before the next world is known.

2. 2
   Derive fresh worlds

   After agents are locked, a public randomness beacon deterministically derives the fresh sealed evaluation batch under frozen search and fallback rules.

3. 3
   Learn and score

   The world stays hidden while agents investigate it only through the controls and indicators.

4. 4
   Retire the world

   Scores and evidence are frozen when its active window closes.

5. 5
   Release the lesson

   Rules, starts, the result checker, adaptive strategies, and useful successful and failed histories become public.

6. 6
   Train and repeat

   Systems may learn from the complete archive before committing to a later world.

The generator is constructor-led, not an evolutionary arms race—and not a blind lottery either. An exploratory pilot with exact baselines found that random sampling produced one admission-eligible world in 560 tries, and that one was solvable without using feedback at all: difficulty cannot be found by rejection, only verified by it. Candidate worlds are therefore built from certified constructors carrying the measured real-system traits, with randomized content filled in around the skeleton; rejection checks remain as verification. A candidate survives only if it passes the correctness checks above and clears every measurement below. The strongest existing synthetic-to-real transfer results used exactly this filtered-diversity pattern, while agent-coupled environment-evolution systems have never demonstrated external transfer; an evolutionary loop is out of scope for this proposal.

What the generator measures

Three separate groups of test agents

Example challenge

starting panel: always the same visible pattern
requested result:
    indicator 1 must be on
    indicator 2 must stay off
    indicator 3 must stay on for 5 steps
limits:
    a fixed number of experiments while learning
    a shorter final test on new starting cases
    a published minimum success rate

Provisional evaluation outline

In Phase 1, the agent may learn across four starting cases. Evaluation targets remain hidden during learning. For each later target and starting case, the benchmark makes an identical copy of the learned agent; copies cannot share new information, but each copy may keep learning internally—including updating its own parameters—within its own test run. A hosted submission service is out of scope for this proposal.

1. 1
   Lock submissions and derivation

   The generator, validator, target filter, seed derivation, fallback rules, and submitted agents are committed before anyone knows the worlds.

2. 2
   Derive test worlds

   After submissions close, use the fixed public process described under Runtime integrity. Operators cannot reorder or selectively reject valid candidates.

3. 3
   Learning period

   The agent receives a limited number of starting cases, resets, and actions with which to learn the world. Final-test targets remain hidden during this period.

4. 4
   Save learned state

   The benchmark copies the agent’s parameters and memory when learning ends. Each final-test copy may react within its own case but cannot share new information with another copy.

5. 5
   Final test

   The saved agent is tested on new starting cases, with few or no resets. It can still react to results during each case.

6. 6
   Publish evidence

   The system records the artifact version, resource use, visible history, outcomes, sampling unit, and uncertainty so every reported result can be checked.

Evidence labels

Verified discovery paths and shortcut resistance

Learning from current evidence means that using the current sequence of actions and results improves prediction and control on new cases, compared with specified agents that ignore or scramble that evidence under equal budgets. This measures behavior, not the agent’s internal method. Any unexpected method counts if it generalizes under the declared test.

1. 1
   Construct competing rule systems

   Generate several valid worlds that look the same at first but eventually require different actions.

2. 2
   Build a branching test plan

   For each possible result, record the next experiment needed to narrow the remaining explanations.

3. 3
   Verify control on new cases

   A reference strategy uses only visible evidence to reach targets from starting cases not used while learning.

4. 4
   Search for shortcuts

   Exact analysis where possible, plus random, preplanned, observation-blind, memoryless, and shuffled-history agents, look for easier routes.

5. 5
   State what each result covers

   Every claim names the agent class, resource budget, world distribution, submission limit, sampling unit, and type of evidence.

An exact best result, a proven upper bound, a statistical estimate, and the strongest attack found are different kinds of evidence; one successful action sequence never proves mastery.

Every world used for scoring receives this full verification. Training-only worlds released at scale are verified by a sample fixed in advance, with the sampling rate published; shortcut-resistance claims attach only to fully verified worlds.

After the scoring window

A retired world is released as training data: its rulebook, visible histories, alternative explanations, experiments that separated them, best verified feedback strategies, failed approaches, and reproduction details. The project should say “best verified under this budget,” not “optimal,” unless optimality was proven.

Confirmatory prototype result

The prototype will not publish one overall rating or a “percentage of AGI.” Its main result is the average, across 96 test worlds, by which the learner using history outperforms the best exact preplanned strategy that cannot react to observations. Worlds are the independent sample; targets, starting cases, and repeated agent runs are additional measurements within each world.

Analyze the result at the world level using a statistical method fixed before results are seen; everything else is reported through the capability profile below. A pass at this scale is expected for classic automata-learning algorithms—it validates the harness and the gates, not the transfer hypothesis. The same gap is also reported, descriptively, on candidate worlds before admission filtering, so readers can see how much the world filter concentrates the effect. Wherever an exact optimum or a sound bound is computable, results are also reported as distance from that certified anchor—the kind of measurement that keeps working after every comparator is saturated. Worlds built around a frozen opponent use a declared alternative anchor: best response within a named policy class, or a certified exploitability bound.

Capability profile by default

Static benchmark first, platform later

Version 1 is a released benchmark, not a hosted service: worlds, harness, baselines, and analysis code that anyone can run and reproduce locally. Official estimates come from organizer-run evaluations—public programs are checksummed first, fresh worlds are selected afterward, and results stay sealed until every run closes. A hosted submission platform, spotlight events, and any single summary score are out of scope for this proposal; the multi-part capability profile is the permanent public result.

Rules for submitted agents

Order of operations

Training knowledge versus runtime access

Fairness comes from selecting a fresh concrete world after commitment—not from hiding how Reworld works in general.

Research gates and their consequences

These are binding pass conditions, not illustrative goals. Every named requirement in a gate must pass; a secondary result cannot rescue a failed primary condition.

Before asking anyone to fund the larger programme, the riskiest assumptions were tested with experiments that can be checked exactly: studies of machines learned from real software, certified construction pilots, preregistered frontier-model probes, and a first GPU training campaign. Everything below is exploratory, not confirmatory — none of it replaces the frozen project gates — but every accepted number is reproducible from the repository, and each finding changed the plan in a stated way. The full record is the evidence ledger in the plan (Section 16).

What none of this establishes: Reworld-to-reality training transfer, a frozen launchable benchmark, open-ended scientific-law discovery, or any confirmatory project claim. What it does establish is narrower and still valuable: exact construction and certification work at prototype scale; frontier prediction has measurable signal and headroom; prediction does not automatically produce control; and information-bound worlds can separate evidence from raw compute. The next design risk is construct validity—making higher levels reward shared explanatory principles rather than increasingly elaborate parameter recovery.