Nord Studio
Micro

Refactor entire data layer

When I started building Micro, I didn't scope far enough to realise the underlying issues that we're facing at the moment. Right now we have the following layers:

  1. REST API (NextJS App Router) - Primary CRUD

  2. Server Actions (NextJS) - Used for the app

  3. WebSocket event bus - structural-change broadcasts

  4. Yjs over WebSocket - collaborative rich-text only

  5. Asset transport - via S3 or local FS vs s3mini

In theory, this layer shouldn't change much anymore, but for the amount of different connections that need to be made can be hugely reduced by replacing most of the stack with a sync engine.

We're already paying most of the cost of a sync engine without getting most of the benefits that it provides. We have three parallel transports in place doing the work that a single sync engine can do. Yjs is narrow purposed and could honestly be removed, the likelyhood that two people the same task description is extremely low and even if there are two version that conflicts, we can easily handle this.

As of right now, every mutation goes: client -> REST -> DB -> broadcast -> other clients refetch on event alert

That's the exact loop sync engines collapse into one primitive.

Benefits:

  • Kill the dual REST + WS-invalidation dance: Right now WS events exist mostly to tell clients "refetch X." A sync engine is that, but with structured diffs instead of "go look again."

  • Optimistic updates, everywhere: Most of the UI appears like it has optimistic updates, but in reality, if you're moving too fast, those changes could get lost as the UI waits for the server roundtrip to complete.

  • Offline + reconnect resilience

  • One transport instead of 4 (CRUD / Actions / WS / Yjs)

Costs:

  • Serious Migraton: ts is gonna take 10 years to complete bro :fear:

  • Permissions need moving to the sync layer

  • Logout revocation based on what sync engine we go with


Activity

t
tygrdev created the task and added the Refactor label·5 months ago
t
tygrdev5 months ago
t
tygrdev5 months ago
t
tygrdev4 months ago