ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
AI & ML interests
I like encoders because they don't talk back.
Recent Activity
liked a model about 22 hours ago
mistralai/Mistral-Large-4-1T-A52B liked a dataset 1 day ago
ReliableAI/irish_blimp updated a model 2 days ago
crawka/Crawka-1Organizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 765 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.94k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 877 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 366
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 765 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.94k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 877 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 366
models 42
DanielGallagherIRE/ModernBERT-O3-Shuffling-SENT
0.2B • Updated • 10
DanielGallagherIRE/ModernBERT-O3-Shuffling-DOC
0.2B • Updated • 12
DanielGallagherIRE/ModernBERT-O3-Baseline
0.2B • Updated • 62
DanielGallagherIRE/ModernBERT-O3-PMI-5
0.2B • Updated • 4
DanielGallagherIRE/ModernBERT-O3-PMI-2.5
0.2B • Updated • 5
DanielGallagherIRE/ModernBERT-O3-PMI-10
0.2B • Updated • 10
DanielGallagherIRE/ModernBERT-O3-Nouns-POS
0.2B • Updated • 15
DanielGallagherIRE/ModernBERT-O3-PMI-7.5
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-PMI-12.5
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-Nouns-DEL
0.2B • Updated • 9
datasets 23
DanielGallagherIRE/FineWeb-Edu-Irish
Viewer • Updated • 700k • 59
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-DOC-Truncated
Viewer • Updated • 11.3M • 218
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-SENT-Truncated
Viewer • Updated • 11.3M • 515
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation-Truncated
Viewer • Updated • 11.3M • 408
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered-Delete
Viewer • Updated • 9.71M • 303
DanielGallagherIRE/FineWeb-Edu-20B-E2-OBF-FULL
Viewer • Updated • 9.69M • 167
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3-Obfuscated
Viewer • Updated • 9.69M • 163
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3
Viewer • Updated • 9.69M • 402
DanielGallagherIRE/FineWeb-Edu-20B-E2-v2
Updated • 3
DanielGallagherIRE/FineWeb-Edu-20B-E2
Updated • 4