Mixtral-8x22B with a bit of tensor surgery/lobotomy to chop out all but one of the MoE FFNs and configured to act like a dense model. Is it useful? Not in this state, without further training of the FFN or merging a few models together. When testing it, the only tokens it wanted to output was space and carriage return. Enjoy.
There are 8 variations as you might expect.
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-1-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-2-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-3-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-4-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-5-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-6-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-7-22B-v0.1]
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-8-22B-v0.1]