AbstractPhila PRO

0.8B and I are going to be good friends.

I've managed to condense a prototype to substantially smaller size but it's not as accurate as the original due to the generic topology being more challenging. I'm working it out though.

I've figured many new formulas based on the results of the last, which enable more deterministic projection rather than requiring the learning process to be so dispersed among many different subsystems.

I've also managed to form a 5d deterministic projection scaffold that should enable the entire structure to be even smaller, assuming I can work out the edge cases.

It's considerably cheaper than expected to keep volume valid. This seems like a partial regression for now but I can improve it a bit before heading back in the original direction. Hopefully it's worth the time spent on the potentially improved more sleek structure.

The smaller one can handle more shapes, considerably more shapes per scene, at a much higher complexity than voxel association. This has drawbacks though, namely these are essentially a gate set for now and the gates aren't perfect. These CAN find the correct potential, however the subprocessing isn't enabled yet, meaning our little 400k param set here is powerful but in a different kind of way.

I've started making pushes to include the missing pieces, so the colab will start to comply to the training regime and the geovocab2 will no longer be required.

The majority of the geovocab2 specific formulas and factories used will be directly represented in the vocabulary directory, which will be optimized to a better state than the originals. They will include both numpy and torch synthesis, as well as numpy and torch optimizations for worker creation and transforms.

With this I will include the more robust shape factory from the original, and expand it to include deformation perturbation. This will be a learned behavior of the model, which will allow the deformation of shapes to be directly aligned and trained in bulk along with multiple overlapping shapes, multiple sectorized shapes, sub-shapes, deviant shapes, and everything related directly to shape pooling rather than using hard-set spectra of shapes projected into space.

These patches will essentially be alignment sectorization in their first states for the first 8piece prototype of the chunk, as I can train that on the currently available G4 issued by COLAB.

This is a required element for increasing the learner to full definition capacity, and is a required hurdle before the patchwork can be expanded to a full chunk. The experiments are promising leading to this point, and as I snap pieces together from the successful experiments the system will begin to converge exactly where the expectation rests.

After that, it's just a matter of expanding upward to the necessary architecture and introducing the weights in sequential linear interpolative sequencing, which is something transformers are uniquely capable at handling with minimal calculations after the pre-calculations.

So far so good.

I'll be running multiple alucard fusion ablations on the patchwork before defaulting to the dual-stream slit-light superposition crystal topology architecture that I've proven works for the smaller patchmaker. My hope is that I can approximate the behavior in a more concise way without requiring the full spread of geometric globalization, but there's no guarantees yet. This could save a huge chunk of training time if it works, and alucard's scheduling internal step system will have a place. This may cut a huge percentage of the overall followup training, potentially allowing for the training on less machines. The topology architecture may be fully required, so hopefully I can just avoid all through some clever math and be done with it.

Avoiding the full multi-tower Beatrix oscillation system would be absolutely fantastic, but I think the predictions afforded by the system may be fully required, and the oscillation system will likely need to be tuned into a new form for this use case as well.

I do apologize for the nasty code, but Claude tends to be very difficult to make cooperate if you drive the code too far from Claude's context window. Much of my organization has helped but not enough, but Claude DOES afford rapid prototype capacity. The current repo itself houses a mostly incomplete representation of the outcome, but I want to make sure at least SOME of the formulas align before I start pushing further iterations.

Fair organization can be found in the router section of the geofractal router, the hierarchy spectrum of the geovocab, and the entire system of the pytorch-wide-compiler. They are ugly though and evolved in their own way, I just let Claude work sometimes because otherwise it would take 4x as long to organize in a reusable fashion.

MOST of the code compiles, but I believe there's some .item() edge cases in the current code that causes graph breaks. I'm working on it.

I'll HOPEFULLY be pushing a fairly organized update to the geolip repo this afternoon with a more complete interpretation of the subsystems, but the formulas aren't perfect yet. I have a couple prototype patchmakers in training but they have some bugs. I'll try to keep them organized.

I need to clean up this sewer honestly, the code got nasty. It's more often fast than not. Might be worth porting all classes directly to the geolip repo, which will centralize for AI development rather than have everything out in divergent systems.

In the gaming industry we call this "YOUR PRODUCER IS CONFUSED AND MAD BECAUSE TECH DEBT"

It's coming together, but the repo is pretty outdated.

Is it ensemble or hierarchical?

They aren't releasing their weights, so other studios have to do it the slow way. This seems like a huge waste of computation, and a response to that in any way other than a utilitarian sense is just going to make the problem worse.

The reasonable solution would be to simply distribute curated distillations to prevent this sort of problem and save global power consumption.

Distillations with expert expectations are very difficult to finetune in a reasonable fashion. They often take more compute than the original took to even reach a similar state.

Distill, snap the experts off, boom you have yourself a distilled computation that can be utilized by companies on their own hardware, and then people will stop trying to reverse engineer and bulk extract information from your hardware. They'll be using their own internal hardware in a different and more cost effective fashion.

Make them good, reusable, expandable within reason, and this problem will evolve to distillation research. By that point the next generation of the big models will be out and the next series of distillations can be made, obsoleting the others.

50k test completed using synthetic data extracted from flux for another project;
https://huggingface.co/datasets/AbstractPhil/synthetic-characters

This is more than enough inference information to get a fair measure as to which features are the most helpful and which aren't so useful.

The results are here as well as the runner;
https://huggingface.co/AbstractPhil/grid-geometric-multishape/tree/main/50k_results

It requires the cell 1 model code and then it'll run.

So what we do here, is snap off the classifier and utilize the various features in cosine similarity conjunction. The accuracy of the tested model is roughly 93% 3-4 shape shared space in the patches, so this can be greatly expanded but it requires additional computational power.

The 3-4 shape shared space should be more than enough pretraining for this hypothesis; which seems to be building more and more potency as something beyond a possibility. This is most definitely a measurable phenomena. Geometric structure most definitely can be analyzed and compacted into useful discriminative features in order to apply a learned bias. How USEFUL those features are? Well, they're pretty discriminative, so there needs to be more tests.

This leaves many questions. Predominantly, the singular one that will be required; can the patches be made smaller if the mathematics are condensed and the shared attention is expanded, and how many patches can this actually support within a nearly-instant computation window?

Does this require the geometric transformers to train or can it learn useful features independently?

Can this benefit from captured embeds in differential conjunction sharing space with a powerful text encoder such as Qwen 2.5 instruct?

Will these patches actually provide attention use down the chain to a diffusion model, or will the mechanism simply get lost in the noise?

So far I've found the most meaningful and reusable representations can be formatted through a gated geometric hierarchy. I'm currently running roughly 50k images through the VAE in order to assess the capacity of the model's components before refactor or reassessment. So far the results are promising for synthetic supervised local patch geometric contribution bias being a very real potential. The model learns to predict the classification elements and then the model no longer requires the transformer blocks, so the gates can be snapped off and the model turned into a fragment of it's larger self. A form of hardened crystalline.

The gates are nearly deterministic between trains, however the classification elements are non-determinant - which means the model is learning to bias in specific routes beyond the current stage in order to justify classification goals. The gates themselves are producing utilizable feature information however, so the outcomes are promising on the refactor.

So far the patch features are showing the most robust reusability potential, but that's only about 120 images or so total, the 50k 15 category test will be the real measure.

Surprisingly the gate statistics are essentially useless, nearly identical through all stages.

The various mechanisms aren't named EXACTLY what I described, and their purposes may have been tweaked a bit. However, the trainer_v4 is running now.
https://huggingface.co/AbstractPhil/tiny-flux-deep/resolve/main/scripts/trainer_v4_testing.py

After converting the model, I've reinitialized the EMA due to the last EMA being essentially completely garbage noise.
https://huggingface.co/AbstractPhil/tiny-flux-deep/tree/main/checkpoint_runs/v4_init

This EMA will be more closely monitored to ensure it doesn't collapse or implode.
Adjacently, the old EMA will be updated to keep hope alive that it will learn eventually as well.

I'll format a safetensors variant for the sol unet today, and ensure the experts exist in a model repo for ease-of-utility.

Talk to Lune here; should be absolutely stunning.
https://huggingface.co/AbstractPhil/tinyflux-experts/blob/main/inference_sd15_flow_lune.py

Talk to Sol here; should encapsulate the entirety of flat output geometric structure in a shape.
https://huggingface.co/AbstractPhil/tinyflux-experts/blob/main/inference_sd15_flow_sol.py

Seems the SOL training NEVER advanced far enough to become full flow-matching, but it definitely aligns to velocity prediction. This may provide a more useful representation than lune in many avenues. Lune is most definitely a full rectified flow model.

Alright I've decided, I'll be training experimentally for some epochs the expertise afforded by sd15-flow-lune's timestep and trajectory knowledge as the guidance distillation mechanism for training. How accurate to the interpolation requirement of tinyflux is to be determined.

Flow-Lune is an acceptable distillation that converted sd15 into a useful representation of an image synthesizer with entirely synthetic data based on sd15 and schnell data.

The pretraining has hit an impasse.
Currently it's a linear timestep based on shift and a random number between 1 and 5 for guidance. I have narrowed the possibilities down to two that can be implemented today to solve this problem; CFG or TIMESTEP, which expert is required and which is the best candidate?

1. The model WILL require a timestep expert manifold. This will allow the expertise for the timestep manifold to be managed by something much more trained and more intelligent during training, which will require CFG guidance training controlled by learning or complete random chance. E.G. standard dropout to encourage CFG.
2. OR the model WILL require a cfg expert to distill the guidance embeds. This model is simply too small. The embeds CAN learn useful information yes, if they are distilled from an expert to cake the CFG into the model by default. This will likely require a third expert that can be modularly snapped off for inference; this expert will likely need to be present during training otherwise the model will heavily drift due to the model's small size.

I have trained a multitude of v-pred sdxl models and a flow-matching shift sd15 model that can represent this necessary implication. This begs the question now; which expert should be used and should I just make a very specific tinyflux expert distilled from ALL SD15 and SDXL timestep variants using David?

This leads to one CORE and IMPORTANT question; CAN THIS BE REPRESENTED WITHOUT AN EXPERT!? I think this is possible, I've ran VIT experiments that used raw sinusoidal for encodings with a surprisingly fair representation of encoding capacity.

The model is ALREADY responsive to CFG but only in part. The current cfg guidance is only getting in the way in many points and I assume is just jiggering in noise, so I'll need to either disable it or use it correctly. The further in training the model gets the more retraining will be required for such a component, so the decisions need to happen sooner rather than later.