A playful abduction: Naronics, or basing neural networks on the boolean base not-and, for which I seem to have a never-ending infatuation.
Just an idea which popped in my head and I am toying with: can I arrive at something, say, n-inputs, n-outputs, log(n) depth, roughly quadratic size and which learns? Hmm, guess not.