CorX1.5
A ~158M parameter language model, built entirely from scratch by a single developer. Architecture, tokenizer, pretraining pipeline and fine-tuning — all written and trained solo, on a single GPU, from the ground up.
Repository: Sigmandndnns/CorX1.5-160M · Developer: Nathan · Origin: Jamaica
CorX1.5 is an open-source language model with roughly 158 million parameters, built from scratch by CorX Labs in Jamaica. It is a custom decoder-only transformer using RoPE positional encoding, SwiGLU feed-forward layers and RMSNorm, with 10 layers, a 1,024-token context window and a 32,768-token custom BPE vocabulary. It is not a fine-tune of an existing base model — it was trained from random weights.
- Parameters
- 157.8M
- Layers
- 10
- Context length
- 1,024 tokens
- Vocabulary
- 32,768 custom BPE
- Pretraining
- ~3B tokens, 25 datasets
- Licence
- Apache 2.0
- Task
- Text generation
- Status
- Actively trained
Architecture
A custom decoder-only transformer.
Not copied from an existing model family — designed and written here.
Core stack
RoPE positional encoding, SwiGLU feed-forward layers and RMSNorm throughout — the combination that gets the most quality out of a small parameter budget.
Role embeddings
A learned signal marking every token as user, assistant or raw text, so the model always knows whose turn it is. Most models infer this only from special tokens.
Assistant-only loss masking
Trained only to predict the tokens it should generate, never the user's side of the conversation — cleaner learning signal per token spent.
Custom tokenizer
A 32,768-token BPE vocabulary trained from the ground up for this model rather than borrowed from another model family.
Training
Trained in stages, from zero.
Pretraining
~3 billion tokens across 25 open datasets covering code, math, reasoning, conversation and general web text, with a held-out validation split and early stopping to catch overfitting.
Distillation
Fine-tuned on responses generated by a larger teacher model (Qwen2.5-Instruct), with automated quality filtering keeping only well-formatted, non-repetitive, correctly-verified answers.
Reasoning training
Trained to show step-by-step work in
<think>...</think>blocks before answering, including verified worked arithmetic and debugging exercises with a known, injected bug to check against.
Honest scope
What it does, and what it doesn't.
Good at
- Holding a consistent identity and conversational style
- Short, direct answers to greetings and common everyday questions
- Declining what it genuinely can't do — no internet, no memory between chats, no tools — instead of fabricating
- Small worked arithmetic, shown step by step
- Recognising and explaining common Python bugs in a structured format
Not built for
- Broad factual knowledge or general reasoning outside its training patterns
- Complex or multi-step maths beyond basic arithmetic
- Production or safety-critical use of any kind
Models of a similar size from major labs are typically trained on trillions of tokens. CorX1.5 was trained on billions — that gap explains most of the above, and it is documented rather than glossed over.
Safety layer
CorX1.5 includes a hardcoded response layer for messages indicating self-harm or crisis, which returns a fixed set of crisis resources rather than a generated response. This is implemented in code, not left to the model's judgment.
Usage
Running CorX1.5.
CorX1.5 uses a custom architecture and is not compatible with
AutoModel.from_pretrained() out of the box.
# Clone the repository — weights, tokenizer and model definition
git lfs install
git clone https://huggingface.co/Sigmandndnns/CorX1.5-160M
cd CorX1.5-160M
# Load with the bundled model definition (modeling_corx.py), not AutoModel
python inference.pyOpen source
Download it, read it, take it apart.
Weights, tokenizer and model definition are published under Apache 2.0 on Hugging Face.