colony.technology/record
record / exact models, upstream fixes, audits
each result once, with the way to run it again
epfl entries are on benchexact models / measuredsmollm2-135m (apache-2.0, revision 93efa2f) in integer arithmetic
64 prompts / sha-256 of the logits at 471 positions by 49,152
machine exact logits float32 logits onnxruntime logits
linux, x86-64 30ed613b…cee773 e1f142df…2b018f b0fd10d5…90e05b
linux, aarch64 30ed613b…cee773 b2f3e029…f99637 bd9dd159…fb6374
macos 26.7, arm64 30ed613b…cee773 32b192d5…fa2db1 bd9dd159…fb6374
macos 15.7, x86-64 30ed613b…cee773 83ccea5a…40be9c 761b6fb2…7e5d89
windows server 2025, x86-64 30ed613b…cee773 da18301e…71040d b0fd10d5…90e05b
windows 11, arm64 30ed613b…cee773 b2f3e029…f99637 bd9dd159…fb6374
apple m1, macos 27.2 30ed613b…cee773 aafc4308…f738c2 bd9dd159…fb6374
distinct, 7 machines 1 6 3
top token as float32 at 471 of 471 prompt positions
run it again
pip install numpy onnxruntime
python exact_llm.py check
method, bounds and the seven-machine recordsmollm2-360m, 361,821,120 weights / one pair of hashes on 7 machines, three routes
python exact_llm.py check --model SmolLM2-360M
smollm2-1.7b, 1,711,376,384 weights / one pair of hashes on 6 machines, 13 runs
python exact_llm.py check --model SmolLM2-1.7B --routes f64 --no-float
top token as float32 at 471 of 471 prompt positions where float32 finished
all-minilm-l6-v2 in integer arithmetic
2,758 sentences of the sts-b test set / one sha-256 on 7 machines
float32 and onnxruntime / 6 each
sts-b spearman 0.8203, exact and float32
method, bounds and the record / run it in colabupstream fixes / mergedneorv32 #1692, merged 14 november, our pull request
fpu: fcvt sticky bit and sign of an exact zero sum in rdn mode
pull request / report #1689black-parrot #1380, merged 15 november, our pull request
fcvt.s.l and fcvt.s.lu rounded once
pull requestbasejump_stl #765, merged 19 november, our pull request
bsg_fpu_add_sub: zero operands and the sign of an exact zero sum
pull requestcva5 #37, merged 15 november, the maintainer's fix to our reports #34, #35, #36
its tree checked at 372dcdf: 403,402,400 vectors, 0 wrong values, 0 missing answers
pull requestxls #5437, #5439, #5440, merged 17 november, the maintainers' fixes to our reports #5419, #5418, #5417
apfloat: round-to-zero cast on overflow, the sign of zero results in fma, nan results in fma
#5437 / #5439 / #5440esp-miner #2045, #2046, merged 5 and 18 november, our pull requests, firmware
batched share counts kept across rejections / elapsed time kept across short counter polls
#2045 / #2046audits / open arithmetic cores against exact arithmetic
every core run as its authors published it; the vectors, the counts and a one-line reproducer per defect are in the audit record
audit record and reproducersdawsonjon/fpu, double divide 2^29 times the quotient, subnormal band / fix filed
dawsonjon/fpu, double to float 1 - 2^-53 converts to 0.5 / fix filed
or1200 fpu, multiply twice the product, subnormal operand
opencores fpu inf * -inf returns nan
basejump stl fpu, add 0 + 0 returns the smallest normal number / fix merged upstream, #765
lowrisc muntjac fpu, fma the 106-bit product loses its last bit / reported
neorv32 fpu, float to integer 0.5 + 2^-24 converts to 0 / fix merged upstream
mor1kx, marocchino fpu overflow returns inf in the directed rounding modes / reported
marocchino fpu 1/0, then 1+1 returns inf / reported
openhw cvfpu, bfloat16 divide a quotient below 2^-134 returns zero, no flags / cause and fix filed
openhw cvfpu, narrowing add 0 + 2^-129 returns 2^127 / reported
openhw cva5 fpu, divide 0.0 / 2.0 right after another divide never returns / fix merged upstream
openhw cva5 fpu, square root inexact roots returned as exact, 29-bit roots / fix merged upstream
openhw cva5 fpu, maintainer's fix merged as 125fb2b; its tree checked at 372dcdf, 403,402,400 vectors, 0 wrong values, 0 missing answers / merged upstream
gaisler noel-v fpu, multiply 3.0 * 5.0 - 15.0 returns 1.65e-24 after a mispredicted branch / cause and fix sent
gaisler noel-v fpu, fma 16 - (1 + 2^-52)(1 - 2^-52) rounds up to exactly 15 / cause and fix sent
ieee vhdl float_pkg, sqrt sqrt(2.0) in double precision is one unit off / cause and patch sent
ieee vhdl float_pkg, mac (-6)(-1) + (-5) returns -15 / cause and patch sent
ieee vhdl float_pkg, resize the double 2^-512 converts to 1.0 in single / cause and patch sent
blackparrot, integer to float a 64-bit integer converted to single is rounded twice / fix merged upstream
microwatt fpu, enabled underflow a result that rounds up is classed as denormalized / cause and fix filedsiemens ac_std_float, fma with ties to odd, a tie under a power of two returns twice the result / cause and fix filed
siemens ac_std_float, add 1.0 minus a far smaller number is one unit low at four fraction widths / cause and fix filed
siemens ac_std_float, ceil ceil(-0.5) returns +0 where the c library returns -0 / cause and fix filed
siemens ac_std_float, convert the largest subnormal of binary32 becomes zero in bfloat16's layout / cause and fix filed
siemens ac_std_float, from int 4294967295 with ties to odd converts to zero / cause and fix filed
siemens ac_math, fixed divide 0.5 / 3 into an integer type leaves the output unwritten / cause and fix filed
llvm apfloat, multiply (1 - 2^-53) * 2^-1022 rounds up to the smallest normal number with no underflow status / reported, patch published
google xls apfloat, fma a nan operand returns infinity on the e4m3 and e5m2 layouts / fix merged upstream
google xls apfloat, fma (1.0 * -0.0) + -0.0 returns +0 / fix merged upstream
google xls apfloat, from int 100000 to half precision, rounding toward zero, returns infinity / fix merged upstream
pulp common_cells, lfsr 16-bit repeats after 64,897 states where a maximal register runs 65,535 / cause and fix filed
opentitan prim_fifo_sync, pass with zero-if-empty write data shows while rvalid is 0 after reset, 20 of 92 configurations / cause and fix filed
taxi axis fifo, depth under two bus cycles zeros through the ram in 66 of 192 configurations / cause and fix sent
adi util_axis_fifo, remove_null_beat_en a stale word in place of each null beat, 180 of 180 configurations / cause and fix filed
adi util_axis_fifo_asym, width ratio 3, 5, 6, 7 342 of 896 configurations wrong / cause and fix filed
rust rustc_apfloat, multiply llvm's missing underflow status in rust's port / reported
dendreth, circom-pairing ec add a first selected key of -g aggregates to g; upstream fix e1b6209 not in the pinned fork / reported
chisel and opentitan lfsr taps 2,568 entries, every one of maximal period
t-head e906, c906, c910 0 wrong values in 1.1 million vectors per operation
benchengineeringcolony
samples / compiled r1cs / analysed in this browser
circom source of the sample / paste your own for the local command