non-linear-docs/09-TECH-STACK-AND-ARCHITECTURE.md

11 KiB

Non-Linear: Tech Stack & Architecture

Stack Overview

┌─────────────────────────────────────────────────────────┐
│                        FRONTEND                         │
│  Vue 3 + Tailwind + Headless UI + ECharts               │
│  Graph Viz: TBD (D3 vs Cytoscape — eval pending)        │
│  Command Palette: vue-command-palette / custom           │
│  Keybindings: VueUse useMagicKeys                       │
│  Icons: Lucide  │  Font: Inter  │  Motion: @vueuse/motion│
│  State: Pinia   │  HTTP: ofetch │  WS: native/socket.io │
├─────────────────────────────────────────────────────────┤
│                     CROSS-PLATFORM                      │
│  Desktop: Tauri (wraps Vue app)     — v0.1              │
│  Mobile: Capacitor (responsive web first) — v0.2+       │
├─────────────────────────────────────────────────────────┤
│                        BACKEND                          │
│  FastAPI (Python)                                       │
│  Taskiq (async task queue — webhooks, imports, agents)   │
├─────────────────────────────────────────────────────────┤
│                      DATA LAYER                         │
│  Neo4j — issue graph (nodes, edges, status, labels)     │
│  Postgres — content & metadata (rich text, comments,    │
│             attachments meta, audit logs, project cfg)   │
│  Redis — caching, WebSocket pub/sub, rate limiting      │
│  Meilisearch — full-text search (issues, comments)      │
│  MinIO — S3-compatible file storage (attachments)       │
├─────────────────────────────────────────────────────────┤
│                         AUTH                            │
│  Authentik — OIDC, API tokens, role mgmt, SSO-ready     │
├─────────────────────────────────────────────────────────┤
│                     DEPLOYMENT                          │
│  Docker Compose (dev + self-hosted)                     │
└─────────────────────────────────────────────────────────┘

Data Boundary

Neo4j — Graph Topology

Owns the decomposition tree and lateral links:

  • Node identity (UUID), short ID
  • Lightweight properties: status, labels, assignee_id, created_at, updated_at
  • Parent → child edges (decomposition tree)
  • Lateral link edges: blocks, blocked_by, relates_to, duplicates
  • Project root references, cycle membership

Why Neo4j over Postgres recursive CTEs: Queries like "find all unblocked leaves in this subtree," "critical path through blocks links," "everything 3 hops from this node" are what Cypher is built for. CTEs get painful with lateral links and variable-depth queries. The gap widens in v0.2+ with cross-project edges.

Postgres — Content & Metadata

  • Rich text content: issue descriptions (markdown)
  • Comment threads: body, author, parent_comment_id (threading), timestamps
  • Attachment metadata: filename, size, mime_type, s3_key, uploader_id, uploaded_at
  • User/agent accounts: profile data, preferences, notification settings
  • Project settings: configuration, member lists, default policies
  • Audit logs: who changed what, when, with before/after snapshots
  • Policy definitions: role templates, custom permission rules

Linked to Neo4j by UUID. Neo4j node stores id: "abc-123". Postgres stores full content keyed by same UUID. FastAPI joins them as needed.

Redis — Caching & Real-Time

  • Subtree query cache (TTL, invalidated on graph mutations)
  • WebSocket pub/sub for real-time updates
  • Rate limiting for agent API
  • Authentik token validation cache

Meilisearch — Search Index

  • Indexes issue titles, descriptions, comments, labels
  • Fed from both Neo4j and Postgres
  • Powers command palette search (issues + commands in one result set)
  • Typo-tolerant, prefix search, filtering by label/status/assignee

MinIO — File Storage

  • S3-compatible API, self-hosted
  • Stores attachment files (images, docs)
  • Postgres stores metadata and S3 key; MinIO stores bytes
  • Migration path to AWS S3: zero code changes

Backend Architecture

FastAPI Application Structure

non-linear-api/
├── app/
│   ├── main.py                 # App, middleware, startup/shutdown
│   ├── config.py               # Settings from env vars
│   ├── dependencies.py         # Shared deps (db sessions, auth, current_user)
│   ├── auth/                   # Authentik integration
│   │   ├── oidc.py             # Token validation, OIDC discovery
│   │   ├── permissions.py      # Policy engine evaluation
│   │   └── agent_tokens.py     # API token management for agents
│   ├── graph/                  # Neo4j layer
│   │   ├── connection.py       # Neo4j driver management
│   │   ├── queries.py          # Cypher query templates
│   │   ├── mutations.py        # Graph write operations
│   │   └── traversal.py        # Subtree, path, neighbor queries
│   ├── content/                # Postgres layer
│   │   ├── models.py           # SQLAlchemy/SQLModel models
│   │   ├── descriptions.py     # Rich text CRUD
│   │   ├── comments.py         # Comment thread CRUD
│   │   └── attachments.py      # Metadata + MinIO upload/download
│   ├── search/                 # Meilisearch integration
│   │   ├── indexer.py          # Index updates on mutations
│   │   └── search.py           # Query interface
│   ├── realtime/               # WebSocket layer
│   │   ├── manager.py          # Connection management
│   │   └── events.py           # Event types and broadcasting
│   ├── tasks/                  # Taskiq background jobs
│   │   ├── webhooks.py         # Deliver webhooks to agent endpoints
│   │   ├── indexing.py         # Async search index updates
│   │   └── notifications.py   # Notification delivery
│   └── api/v1/                 # Route handlers
│       ├── nodes.py            # CRUD + tree operations
│       ├── links.py            # Lateral link management
│       ├── projects.py         # Project CRUD
│       ├── comments.py         # Comment endpoints
│       ├── attachments.py      # Upload/download
│       ├── search.py           # Search endpoint
│       └── agent.py            # Agent-specific API surface
├── tests/
├── alembic/                    # Postgres migrations
├── docker-compose.yml
└── pyproject.toml

Request Flows

Typical read ("get node with full context"):

Client → FastAPI → Auth middleware (validate token via Authentik)
  → Policy engine (check permissions)
  → Neo4j: fetch node + parent + children + links
  → Postgres: fetch description, comments, attachment meta
  → Merge response → Client

Typical write ("change node status"):

Client → FastAPI → Auth → Policy engine
  → Neo4j: update node status
  → Redis: invalidate cache, publish event
  → Taskiq: queue webhook delivery, search index update
  → WebSocket: broadcast to connected clients
  → Response → Client

Sync Strategy (Neo4j ↔ Postgres)

Not replicated — they own different data. Linked by UUID. Both operations happen in same API request. Compensating transaction pattern for consistency. Eventual consistency acceptable for search index and cache.

Auth Architecture

┌──────────┐     OIDC token      ┌───────────┐
│  Vue App ├─────────────────────►│ Authentik │
└────┬─────┘     (login flow)    └─────┬─────┘
     │                                 │
     │ Bearer token                    │ Token introspection
     ▼                                 ▼
┌──────────┐◄────────────────────┌───────────┐
│ FastAPI  │  validate token     │ Authentik │
│ (resource│  check claims       │  (OIDC    │
│  server) │                     │  provider)│
└──────────┘                     └───────────┘
  • Human users: OIDC login flow. JWT access tokens.
  • AI agents: API tokens issued through Authentik, tied to agent actor accounts.
  • FastAPI: pure resource server. Validates tokens, reads claims, enforces policies.

Design Language

Targets Linear's aesthetic: minimal, fast, slightly dark-IDE feel.

  • Spacing: tight, no wasted space
  • Colors: muted base palette, high-contrast accents only for status/priority
  • Borders: almost none — separation via spacing and subtle background shifts
  • Dark mode: default, light mode secondary
  • Typography: Inter, small-but-readable sizes
  • Animations: subtle slides and fades, 100-150ms, nothing bouncy
  • Optimistic updates: every interaction feels instant, syncs in background

Docker Compose (Dev)

services:
  api:          # FastAPI
  frontend:     # Vue 3 (vite dev / nginx prod)
  worker:       # Taskiq worker (same codebase as api)
  neo4j:        # Graph database
  postgres:     # Relational database
  redis:        # Cache + pub/sub
  meilisearch:  # Search engine
  minio:        # Object storage
  authentik:    # Identity provider (server + worker)
  authentik-db: # Authentik's own Postgres

~10 containers. Runs comfortably on 16GB RAM.

Open Technical Questions

  1. Graph viz library: D3 vs Cytoscape — prototype comparison pending
  2. Real-time strategy: WebSocket via FastAPI or dedicated service (Centrifugo)?
  3. Neo4j driver: official neo4j Python driver vs neomodel OGM
  4. Rich text format: Markdown (simple) vs ProseMirror JSON (collaborative editing)
  5. Gantt implementation: custom or frappe-gantt as starting point