The nanos.
AbstractPhila PRO
AbstractPhil
AI & ML interests
datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.
Recent Activity
updated a dataset about 4 hours ago
AbstractPhil/thought-symbolic-image-captions repliedto their post about 5 hours ago
The rotary test cases are in and a solution was found.
https://huggingface.co/blog/AbstractPowered/rotary-addressed-selection
I've coined the term Rotary Addressed Selection, at least until I discover if someone else has made it first.
This rotary can handle an absurd length of context window, though the technical documentation is quite complex. Targeting context window after 1 million was how the addressing corruption was solved.
Yes this is all compacted into a 9m param model. A universe in an atom.
The rotary system involves including a full address for every 4096 context window lookup state. This allows the model to differentiate between a multitude of modular model extensions.
How does that work? Well if there's nothing at the address, then there's no return. Addresses are essentially selectors promoting the cooperative state and it does have some limitations, but the full global rotary supports DIRECT dynamic extension to the model at runtime.
Need only a simple classifier? The selectors know which experts classify and which state they exist in.
Need a full VLM? Load up the experts.
You get the idea.
Query with the address, get the correct addressing. That's how it works. The model learns to handle everything internally, so there is no need for a large structural addressing screening beforehand.
With this development, full mixture of experts with uncertain (absurdly long) context depth can have a consistent context window guarantee. There is bound to be some direct breakdowns and numerics that can't simply reach beyond certain points likely based on hardware or rounding differences, but the system essentially allows full MOE where experts are selectable and the selected experts are a guaranteed addressing.
The upcoming experimentation will fill in the gaps for the test cases, confirm or refute clauses that are established on the rotary, and then training a functional Alhena at the size of the mini-beatrix scale with some tweaks for the changes. posted an update about 6 hours ago
The rotary test cases are in and a solution was found.
https://huggingface.co/blog/AbstractPowered/rotary-addressed-selection
I've coined the term Rotary Addressed Selection, at least until I discover if someone else has made it first.
This rotary can handle an absurd length of context window, though the technical documentation is quite complex. Targeting context window after 1 million was how the addressing corruption was solved.
Yes this is all compacted into a 9m param model. A universe in an atom.
The rotary system involves including a full address for every 4096 context window lookup state. This allows the model to differentiate between a multitude of modular model extensions.
How does that work? Well if there's nothing at the address, then there's no return. Addresses are essentially selectors promoting the cooperative state and it does have some limitations, but the full global rotary supports DIRECT dynamic extension to the model at runtime.
Need only a simple classifier? The selectors know which experts classify and which state they exist in.
Need a full VLM? Load up the experts.
You get the idea.
Query with the address, get the correct addressing. That's how it works. The model learns to handle everything internally, so there is no need for a large structural addressing screening beforehand.
With this development, full mixture of experts with uncertain (absurdly long) context depth can have a consistent context window guarantee. There is bound to be some direct breakdowns and numerics that can't simply reach beyond certain points likely based on hardware or rounding differences, but the system essentially allows full MOE where experts are selectable and the selected experts are a guaranteed addressing.
The upcoming experimentation will fill in the gaps for the test cases, confirm or refute clauses that are established on the rotary, and then training a functional Alhena at the size of the mini-beatrix scale with some tweaks for the changes.Organizations
A series of useful models expert-trained using the GEOLIP distillation and constellation process.
My flagship models that actually work or are the best I have capable from a category currently.
-
AbstractPhil/mini-beatrix-2s
Text Generation • 0.2B • Updated • 292 • 1 -
AbstractPhil/mini-beatrix-1
Text Generation • 0.1B • Updated • 310 • 2 -
AbstractPhil/captionbert-8192-v2-b
Feature Extraction • 58.3M • Updated • 29 • 2 -
AbstractPhil/captionbert-8192-v2
Feature Extraction • 58.3M • Updated • 24 • 1
The ones that didn't quite work, but are still very interesting and have their use cases nonetheless.
Distilled models.
-
AbstractPhil/mini-beatrix-3
Text Generation • 0.4B • Updated • 1.5k • 1 -
AbstractPhil/mini-beatrix-2s
Text Generation • 0.2B • Updated • 292 • 1 -
AbstractPhil/mini-beatrix-1
Text Generation • 0.1B • Updated • 310 • 2 -
AbstractPhil/clip-vitb-mini-distilled
Image Feature Extraction • 8.93M • Updated • 35
A series of repos dedicated to geolip research and results, some with stored weights, some without. All entirely based on the progress of geolip.
The good stuff.
These are Beeper-based models.
A collection of experimental shunts that work very well with the comfyui suite for tuning diffusion and removing noise.
The nanos.
Distilled models.
-
AbstractPhil/mini-beatrix-3
Text Generation • 0.4B • Updated • 1.5k • 1 -
AbstractPhil/mini-beatrix-2s
Text Generation • 0.2B • Updated • 292 • 1 -
AbstractPhil/mini-beatrix-1
Text Generation • 0.1B • Updated • 310 • 2 -
AbstractPhil/clip-vitb-mini-distilled
Image Feature Extraction • 8.93M • Updated • 35
A series of useful models expert-trained using the GEOLIP distillation and constellation process.
A series of repos dedicated to geolip research and results, some with stored weights, some without. All entirely based on the progress of geolip.
My flagship models that actually work or are the best I have capable from a category currently.
-
AbstractPhil/mini-beatrix-2s
Text Generation • 0.2B • Updated • 292 • 1 -
AbstractPhil/mini-beatrix-1
Text Generation • 0.1B • Updated • 310 • 2 -
AbstractPhil/captionbert-8192-v2-b
Feature Extraction • 58.3M • Updated • 29 • 2 -
AbstractPhil/captionbert-8192-v2
Feature Extraction • 58.3M • Updated • 24 • 1
The good stuff.
These are Beeper-based models.
The ones that didn't quite work, but are still very interesting and have their use cases nonetheless.
A collection of experimental shunts that work very well with the comfyui suite for tuning diffusion and removing noise.