Geometry as Logic
Mathematics is an extension of logic. Every domain in mathematics can be reduced to logic. Geometry is the logic of space.
The most foundational, granular unit of any quantitative system is one bit. This is true by definition--whatever the fundamentum is, it's the bit of that system. The granular, base unit.
In geometry, the foundational unit is therefore one bit of space--the geometric atom. We will call these atoms points.
All geometric structure can be reduced to sets of points. "Shapes," therefore, refer to specific sets of points.
Position and State
We can build geometries out of two fundamental concepts: position and state.
Position: where a point is located relative to other points.
State: the state the point is in.
These two concepts give us the most granular way of talking about any geometric structure. In other words, we can describe all the information by referring to "That point in that state."
It's a Matrix
I don't claim to be a mathematician, but from what I can tell, this concept is best captured by matrix theory. In the simplest model (allowing only two possible states for each point) space is essentially an array of bits.
In more complex models, the points can be in a range of possible states, not just 0 or 1.
I know some physicists and computationally-biased mathematicians will appreciate this perspective. So let me make it more controversial.
My claim is this is the a priori geometry. I believe this is the way minds actually think about space, the way minds must think about space, and the only way for space to be. Since logic and existence are inseparable, in order for space to exist, it must be logical, and this is the logic of space: position and state.
Every geometric model can be put into this framework. In fact, it's the litmus test for geometry--if it cannot be put into this framework in principle, it's logically incoherent.
Computers all over the world, rejoice!
Physics is Geometry + Time
Physics is simply keeping track of geometry. It's observing how geometric states change over time--and therefore figuring out how this part of the matrix connects to that part of the matrix. We discover patterns of state changes over time and infer the laws of physics from them.
Not a Reduction of Everything
The physical world can be reduced down to quantitative, logical analysis. That does not imply the entirety of existence can be reduced in such a way. Metaphysical pluralism allows for things like consciousness, information, and abstract stuff to exist outside of the matrix. They are in separate but related domains.
Give it a few decades, and I expect this framework will conquer everything.
It is a bit of a leap to start with a description (“we can describe all the information by referring to "That point in that state."”) and end up with a claim about the essential nature of the thing being described (“space is essentially an array of bits”). Shouldn’t there be a bit more of a bridge between those?
And as you note, the description is incomplete without some additional something that tells us how the description will change over time. One convenient hypothesis would be that space and that something are distinct and can be dealt with separately. But there are other hypotheses that could be considered. Have they been rejected?