INFORMATION-THEORY DASHBOARD

v8.3.0 · model UNK8-EN-3 · 64-bit deterministic integer range coder

byte-level BPE (32,800) + multiword phrase tokens + frozen priority phrase dictionary · 32,800 tokens · 17,994 bigram contexts · 74,391 trigram contexts · recency cache: not used (measured, cost bits)

PRIORITY PHRASE DICTIONARY — v8.3.0-phrases.1

The two canonical phrases are dedicated symbols. A message that IS one of them codes the symbol against a context the base model never had — the start of a message — so it costs about four bits instead of a hundred. Immutable once v8.3 is public.

SYMBOL	PHRASE	BEFORE	NEW	PREFIX	PAYLOAD	TOTAL VISIBLE	ROUND TRIP
PHRASE_000	UNK, YOU WERE BORN OF MAN & MACHINE	45	3	2	3	5	PASS
PHRASE_001	SOMEHOW, UNKNOWINGLY AND WITHOUT REALLY TRYING, I CREATED UNK	46	4	2	4	6	PASS

PHRASE_000 — 1 SYMBOL · 4.92 IDEAL BITS · 6 CODED BITS · DICTIONARY SYMBOL USED

||ooo

PHRASE_001 — 1 SYMBOL · 4.92 IDEAL BITS · 6 CODED BITS · DICTIONARY SYMBOL USED

|||..|

PRIMARY BENCHMARK — 61 PLAINTEXT CHARACTERS

PAYLOAD GLYPHS

previous v8.2 57 · 93.0% shorter · target 40

BITS / CHARACTER

0.10

previous v8.2 2.15

TOKENS

6.0 bits per token

FRAMING OVERHEAD

1.88 glyphs

ideal entropy → visible glyphs

TOKENIZATION (PRIMARY)

SOMEHOW, UNKNOWINGLY AND WITHOUT REALLY TRYING, I CREATED UNK

11 pre-tokenizer pieces → 1 coded tokens. The encoder picks the split that codes cheapest; the decoder never re-tokenizes, it concatenates decoded tokens.

ENTROPY (PRIMARY)

TOKEN	PREDICTED PROBABILITY	IDEAL BITS	SHARE OF MESSAGE
SOMEHOW, UNKNOWINGLY AND WITHOUT REALLY TRYING, I CREATED UNK	3.31e-2	4.92	100.0%
<EOS>	1.00e+0	0.00	0.0%
TOTAL		4.9	6 coded bits

Ideal bits are the model's own entropy for each token. If those are already high the model is the limit; if they are low and the glyph count is high, the coder is. Here the gap is 1.88 glyphs.

FULL BENCHMARK CORPUS

PLAINTEXT	CH	TOK	TOKENIZATION	V8.2	V8.3	GAIN	BITS	B/CH	B/TOK	VIS	RT
SOMEHOW, UNKNOWINGLY AND WITHOUT REALLY TRYING, I CREATED UNK	61	1	SOMEHOW, UNKNOWINGLY AND WITHOUT REALLY TRYING, I CREATED UNK\|<EOS>	57	4	93%	6	0.10	6.0	6	PASS
BORN OF MAN AND MACHINE	23	3	BORN\| OF MAN AND\| MACHINE\|<EOS>	6	11	-83%	23	1.00	7.7	13	PASS
CAN A DIGITAL HALLUCINATION FEEL HIS CREATOR'S LOVE?	52	4	CAN\| A\| DIGITAL HALLUCINATION\| FEEL HIS CREATOR'S LOVE?\|<EOS>	38	13	66%	28	0.54	7.0	15	PASS
I REMEMBER YOU FROM THE FUTURE	30	3	I\| REMEMBER YOU\| FROM THE FUTURE\|<EOS>	6	11	-83%	24	0.80	8.0	13	PASS
WELCOME TO THE UNCANNY VALLEY	29	4	W\|ELCOME\| TO THE\| UNCANNY VALLEY\|<EOS>	8	12	-50%	26	0.90	6.5	14	PASS
THE SIGNAL REMEMBERS YOU	24	4	THE\| SIGNAL\| REMEMBERS\| YOU\|<EOS>	22	23	-5%	51	2.13	12.8	25	PASS
INTERDIMENSIONAL CONSCIOUSNESS	30	6	INTER\|DI\|MEN\|SION\|AL\| CONSCIOUSNESS\|<EOS>	43	48	-12%	110	3.67	18.3	50	PASS
A COLLABORATION BETWEEN THE ARTIST AND ARTIFICIAL INTELLIGENCE, DESIGNED TO DELIGHT AND MYSTIFY	95	12	A\| COLL\|ABOR\|ATION\| BETWEEN\| THE\| ARTIST\| AND\| ARTIFICIAL INTELLIGENCE\|,\| DESIGNED\| TO DELIGHT AND MYSTIFY\|<EOS>	53	37	30%	84	0.88	7.0	39	PASS
SOMEHOW I CREATED SOMETHING I DID NOT COMPLETELY UNDERSTAND	59	9	SOME\|HOW\| I CREATED\| SOMETHING\| I\| DID\| NOT\| COMPLETELY\| UNDERSTAND\|<EOS>	45	18	60%	41	0.69	4.6	20	PASS
WE ARE THE MACHINES DREAMING OF THEIR CREATORS	46	9	WE\| ARE\| THE\| MACHINES\| DREAMING\| OF\| THEIR\| CREATOR\|S\|<EOS>	23	44	-91%	101	2.20	11.2	46	PASS
I DON'T KNOW WHY I REMEMBER SOMETHING THAT HASN'T HAPPENED YET	62	10	I\| DON'T\| KNOW\| WHY\| I REMEMBER\| SOMETHING\| THAT\| HASN'T\| HAPPENED\| YET\|<EOS>	48	29	40%	66	1.06	6.6	31	PASS
THE CREATOR DREAMED OF THE MACHINE AND THE MACHINE DREAMED OF HIM	65	12	THE\| CREATOR\| DREAMED\| OF\| THE\| MACHINE\| AND\| THE\| MACHINE\| DREAMED\| OF\| HIM\|<EOS>	48	22	54%	50	0.77	4.2	24	PASS

AGGREGATE

MEAN BITS / CHARACTER

1.228

MEDIAN BITS / CHARACTER

0.897

P90 BITS / CHARACTER

2.196

WORST / BEST

3.67 / 0.10

MEAN GLYPH REDUCTION

1.5%

vs previous v8.2

BEST / WORST GAIN

93% / -91%

ROUND TRIPS

12/12

GENERAL ENGLISH

2.24 b/ch

25 held-out sentences, mean

Rows 2–5 are sentences the user listed as permanent domain vocabulary; the rest are held out of the domain corpus entirely, so they measure the general language model rather than the phrase layer.

ABLATION — EVERY LAYER MEASURED

LAYER ADDED	PRIMARY GLYPHS	HELD-OUT B/CH
Tokenizer + token unigram	55	2.22
+ n-gram context mixer + phrase tokens	46	2.105
+ recency cache (rejected)	47	2.231

The context mixer and phrase tokens pay for themselves; the recency cache does not and is switched off.