en-or-not: English vs non-English text detection

en-or-not is a small byte-level convolutional neural network that predicts whether an input is English (EN) or not English (NOT-EN). It is intended for short-text filtering, not for identifying which non-English language is present.

The implementation and training/provenance companion for v1 is on GitHub.

Model

• Architecture: TinyByteCNN_EN, six convolutional blocks
• Parameters: 156,642
• Input: raw UTF-8 bytes, truncated to 256 bytes and zero-padded
• Output: two logits in the order NOT-EN, EN
• Files: model.onnx for ONNX Runtime and model.safetensors for PyTorch

Published OOD evaluation

The primary evaluation set is evaluation/ood_set.jsonl, with its provenance and selection rules in evaluation/ood_manifest.json. It contains 17,544 real Tatoeba sentences:

• 8,772 English examples
• 8,772 non-English examples
• 52 non-English language strata, covering multiple scripts and language families
• No synthetic fallback rows
• No duplicate normalized-text hashes
• Every row is valid for the model's 256-byte input limit

The set was built from the pinned Tatoeba sentence export dated 2026-07-18. Tatoeba data is released under CC-BY 2.0. Each row retains its Tatoeba source ID and attribution URL. The benchmark is source-level OOD relative to the training sources documented in the linked project: Project Gutenberg, papluca/language-identification, and optional OSCAR. This is not a guarantee of zero lexical overlap with undisclosed training artifacts.

ONNX Runtime result:

accuracy: 17007/17544 = 96.9391%
95% Wald CI: 96.6842%-97.1940%
EN: 8465/8772 = 96.5002%
NOT-EN: 8542/8772 = 97.3780%
language macro accuracy: 97.3802% across 52 non-English languages

The benchmark is balanced by class, so the overall accuracy is also balanced accuracy. Short examples are harder than longer examples in this set:

short:  6931/7327 = 94.5953%
medium: 9390/9531 = 98.5206%
long:    686/686 = 100.0000%

Rebuild and evaluate it with:

python3 scripts/build_ood_set.py \
  --source /path/to/sentences.tar.bz2 \
  --output evaluation/ood_set.jsonl \
  --manifest evaluation/ood_manifest.json

python3 scripts/evaluate_onnx.py evaluation/ood_set.jsonl

The old hand-authored evaluation/hostile_ood.jsonl file is retained as an adversarial diagnostic set. It is not included in the published OOD figure.

Quick start

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("model.onnx")

def english_probability(text: str) -> float:
    data = text.encode("utf-8", errors="ignore")[:256]
    inputs = np.zeros((1, 256), dtype=np.int64)
    inputs[0, :len(data)] = list(data)
    logits = session.run(["logits"], {"input_bytes": inputs})[0][0]
    probabilities = np.exp(logits - np.max(logits))
    probabilities /= probabilities.sum()
    return float(probabilities[1])

print(english_probability("Hello world!"))

Training result and limitations

The original model metadata reports 99.94% validation accuracy on 17,543 training-pipeline samples. The source data, split, and training evaluator are not included here, so that number is a reported training result, not an independently reproducible general-accuracy claim.

This is binary classification, not language identification. Very short text, capitalization, transliteration, mixed language, code, URLs, numbers, and non-linguistic strings can be out of distribution. Confidence scores are model probabilities, not calibrated guarantees. Evaluate on data representative of the deployment domain before using the model as a gate or safety-critical decision.

License

The model artifacts and code are released under the MIT License. The published evaluation text is sourced from Tatoeba and remains subject to CC-BY 2.0 attribution requirements.