SKY-001 Benchmarks: CAD & BIM Results Explained

Start with the correct comparator.

Skyfall reports the following task time, task cost, and intersection-over-union scores on its SKY-001 product page, linked in the primary sources. CAD and BIM remain separate: neither their time nor their overlap scores should be pooled into one overall ranking.

BIM comparator check: GPT-6 Astra’s IoU is 40.6%, not 42.3%. The 42.3% result labeled as the best LLM belongs to Gemini 3.8 Flash. Substituting that result would silently change which model is being compared.

What methodology does Skyfall publish?

Skyfall’s introduction, cited in the source list, describes a CAD construction benchmark of 1,000 components drawn from complex objects, with about 100 drawing steps and final 3D IoU evaluation. Its BIM construction benchmark contains 1,000 floor plans: approximately 350 whole floors from IFC files and 650 apartment plans. Sequences range from 20 to more than 200 steps, averaging 70, with final floor-plan IoU evaluation.

This is a partial published protocol, not an absence of methodology. It establishes task scope and the reported evaluation target. It does not establish that we can reproduce the exact runs. Skyfall’s separate Composer evaluation uses qualitative blind judging by two LLMs; it should not be blended with these construction IoU results.

Arithmetic audit: ratios are derived, not new tests.

Divide Astra’s reported task time by SKY-001’s to obtain the time ratio. Apply the same operation to cost. A ratio of 51.0× means Astra’s reported time is approximately 51 times SKY-001’s; it is not a measured 51-fold improvement in an entire engineering workflow.

Before rounding, CAD’s time ratio is 51.0256… and its cost ratio is exactly 150 from the displayed inputs. BIM’s ratios are 118.5246… for time and 185.8333… for cost. These inputs may already be rounded by the publisher; extra decimal places would imply unwarranted measurement precision.

IoU is overlap, not engineering approval.

Intersection over union divides the overlap between a predicted result and its reference by their combined extent. The reported CAD gap is 0.6 percentage points; the BIM gap is 3.0 percentage points. Those are absolute differences between percentage scores, not relative increases of 0.6% and 3%.

Aggregate scores alone do not establish statistical significance. We have not verified per-example results, repeated-run variation, or confidence intervals. Nor does IoU establish dimensional tolerances, constraint validity, editable design intent, structural compliance, or manufacturing safety. Engineering acceptance requires checks appropriate to the artifact, beyond geometric overlap.

Evidence checklist for a reproducible comparison.

Reported: source scores and the task descriptions above. Derived: our ratios and percentage-point subtraction. Unverified here: independent reproduction and the following controls. These are evidence requests, not claims that Skyfall omitted every item:

• Dataset: obtain the exact examples, reference geometry, split, exclusions, and any training-set overlap disclosure.
• Model setup: record model versions, prompts, reasoning settings, tool access, context, and allowed construction operations.
• Execution: specify hardware, concurrency, caching, timing boundaries, timeouts, retry policy, and treatment of failures.
• Scoring: publish the IoU implementation, alignment and discretization choices, aggregation rule, and per-example outputs.
• Repeatability: retain run logs, repeat stochastic runs, and report dispersion alongside the headline averages.

Use identical held-out tasks and acceptance criteria for both models. A nominally matching dataset is insufficient if one system gets additional retries, different tools, or a different success denominator.

Account for cost per accepted artifact.

The source’s dollar figures are benchmark task costs, not verified SKY-001 commercial prices. OpenAI publishes Astra’s token pricing; our cost and pricing comparison sets out the reviewed rates and their units. A price per million tokens cannot be substituted for cost per construction task without token usage and execution details.

• Record input, cached-input, and output usage; the applicable rate and billing date; and any context-length pricing tier.
• Include failed attempts, retries, external tools, compute, and paid dependencies rather than counting only successful generations.
• Track engineer review, repair, and validation time separately, then include those costs in the operational total.

A useful accounting definition is total evaluation spend ÷ number of artifacts that pass your acceptance checks. If none pass, no finite cost per accepted artifact can be reported. This denominator prevents cheap but unusable outputs from looking economical.

Common interpretation questions.

Primary sources and next reading.

• Skyfall: SKY-001 benchmark results — reported CAD and BIM numbers.
• Skyfall: Introducing SKY-001 — vendor-published task and evaluation descriptions.
• OpenAI: GPT-6 Astra model documentation — API capabilities and token pricing.

Compare SKY-001 and GPT-6 Astra for your workflow · Return to the SKY-001 guide

Translate benchmark questions into an acceptance brief with the CAD evaluation guide or BIM evaluation guide. The architecture explainer separates visual planning from executable output.