corpus

self-hosted listenbrainz and last.fm proxy
Log | Files | Refs | README | LICENSE

architecture.md (12064B)

      1 # Corpus Architecture
      2 
      3 Corpus is a self-hosted music listening history dashboard and analytics service. It supports multiple users, synchronizing scrobbles from ListenBrainz and Last.fm and providing a performant web interface for data exploration and statistics.
      4 
      5 ## System Components
      6 
      7 ### Web Server
      8 The server is built with PureScript running on Node.js. It handles several core responsibilities:
      9 - **HTTP API**: Serves the frontend, scrobble data (with filtering/pagination), statistics, and similar tracks (via [cosine.club](https://cosine.club)). It also provides a **ListenBrainz-compatible scrobble submission endpoint**.
     10 - **ListenBrainz Sync**: A background process that polls the ListenBrainz API every 60 seconds to fetch new scrobbles.
     11 - **Last.fm Sync**: A background process that polls the Last.fm API every 60 seconds to fetch new scrobbles. Both syncs write to the same `scrobbles` table; duplicate timestamps are silently ignored.
     12 - **Metadata Enrichment**: A background process that identifies scrobbles with missing metadata (genres, labels, release years) and fetches information from MusicBrainz, Last.fm, and Discogs.
     13 - **Cover Art Proxy**: A specialized endpoint that fetches, caches, and serves cover art, utilizing a multi-source fallback strategy (CAA → Last.fm → Discogs).
     14 - **Observability**: Prometheus metrics exposed at `/metrics`.
     15 
     16 ### Frontend
     17 A Single Page Application (SPA) built with [Elm](https://elm-lang.org).
     18 - **Real-time Updates**: Periodically refreshes the scrobble list.
     19 - **Filtering & Search**: Supports deep filtering by genre, label, or release year.
     20 - **Search Functionality**: Global search across tracks, artists, albums, and labels with real-time results.
     21 - **About Page**: Provides system information, feature list, and links to related resources.
     22 - **Clickable Metadata**: Track name, artist, album, and label in listen entries are all clickable for quick filtering.
     23 - **Responsive UI**: Designed for both desktop and mobile viewing with a "retro-modern" aesthetic.
     24 
     25 ### Database
     26 Corpus uses **DuckDB** for its primary data storage. Each user has their own database file.
     27 - **Schema**:
     28     - `scrobbles`: Stores the core listening history (timestamp, track, artist, album, MBIDs). The `listened_at` Unix timestamp is the primary key — scrobbles from ListenBrainz and Last.fm deduplicate naturally.
     29     - `release_metadata`: Stores enriched metadata indexed by MusicBrainz Release ID (MBID), including genre, label, and release year.
     30     - `api_tokens`: Stores hashed user API tokens for scrobble submission. Tokens are hashed with SHA-256 before storage.
     31 - **Performance**: DuckDB's columnar storage allows for extremely fast analytical queries across large listening histories.
     32 
     33 ### Storage
     34 Uses an S3-compatible bucket to cache cover art images.
     35 - **Caching Strategy**: Images are fetched once from external APIs and stored in S3 to reduce latency and avoid rate-limiting on external services.
     36 
     37 ## Multi-User Support
     38 
     39 Corpus runs as a single server process serving multiple users. User configuration is defined in `users.json`, managed via the built-in CLI commands.
     40 
     41 ### Routing
     42 - `/` and `/u/<slug>` — serve the Elm SPA for the root user and named users respectively
     43 - `/proxy?user=<slug>`, `/stats?user=<slug>`, `/cover?user=<slug>` — shared API endpoints, user selected via query parameter
     44 - `/healthz?user=<slug>` — liveness check; pings the user's DuckDB connection
     45 - `/1/submit-listens` — ListenBrainz-compatible scrobble submission endpoint (requires `Authorization: Token <token>` header)
     46 - `/metrics` — Prometheus metrics (no user parameter; covers all users; only available when `METRICS_ENABLED=true`)
     47 
     48 ### User Management
     49 
     50 Users are managed via built-in CLI commands. The server must not be running when modifying `users.json`.
     51 
     52 ```sh
     53 # Add a new user (creates the DB, prints the API token once)
     54 node server.js add-user --slug filip --name "Filip" --db filip.db
     55 
     56 # Optional: link a ListenBrainz or Last.fm account at creation time
     57 node server.js add-user --slug filip --name "Filip" --db filip.db \
     58   --listenbrainz-user filip --lastfm-user filiplfm
     59 
     60 # List all users
     61 node server.js list-users
     62 
     63 # Regenerate the API token for a user
     64 node server.js reset-token --slug filip
     65 ```
     66 
     67 The API token is only printed once on creation or reset — store it securely. It is used for the `/1/submit-listens` endpoint via `Authorization: Token <token>`.
     68 
     69 ### Configuration
     70 User configuration is split into two layers:
     71 
     72 1. **`users.json`** (non-sensitive): defines user slugs, source usernames, database filenames, and feature flags. Managed via CLI (`add-user`, `reset-token`, `list-users`). The server reads this file at startup from the path in `CORPUS_USERS_FILE` (defaults to `users.json`).
     73 
     74 2. **Environment variables** (runtime, sensitive): shared API keys and S3 credentials are read from the environment at startup and applied to all users.
     75 
     76 Each user gets their own `UserContext` with an independent DuckDB connection, sync loop, and write lock (`AVar Unit`). The write lock serializes all sync transactions — if a user has both ListenBrainz and Last.fm configured, their transactions are queued rather than run concurrently. HTTP reads do not acquire the lock; DuckDB's MVCC provides consistent snapshots.
     77 
     78 ### Graceful Shutdown
     79 
     80 The server implements graceful shutdown handling to ensure clean termination of all background processes. When the server receives a shutdown signal, it:
     81 
     82 1. **Kills background fibers**: Terminates all running background tasks (metadata enrichment, database backups) for each user
     83 2. **Closes database connections**: Properly closes all DuckDB connections
     84 3. **Logs cleanup progress**: Provides detailed logging during the shutdown process
     85 
     86 This prevents data corruption and ensures that any in-progress operations are completed or safely aborted before the server exits.
     87 
     88 ## Configuration Reference
     89 
     90 ### Environment Variables
     91 
     92 | Variable | Default | Purpose |
     93 |---|---|---|
     94 | `CORPUS_USERS_FILE` | `users.json` | Path to the compiled users config |
     95 | `DATABASE_PATH` | _(cwd)_ | Root directory for all user database files |
     96 | `LASTFM_API_KEY` | — | Last.fm API key (required if any user has `lastfmUser`) |
     97 | `DISCOGS_TOKEN` | — | Discogs token for cover/genre fallback |
     98 | `S3_BUCKET` | — | S3 bucket for cover art cache |
     99 | `S3_REGION` | `us-east-1` | S3 region |
    100 | `AWS_ACCESS_KEY_ID` | — | S3 credentials |
    101 | `AWS_SECRET_ACCESS_KEY` | — | S3 credentials |
    102 | `AWS_ENDPOINT_URL` | — | S3 endpoint (for S3-compatible storage) |
    103 | `AWS_S3_ADDRESSING_STYLE` | — | `virtual` or `path` |
    104 | `COSINE_API_KEY` | — | [cosine.club](https://cosine.club) API key for similar tracks |
    105 | `PORT` | `8000` | HTTP listen port |
    106 | `METRICS_ENABLED` | `false` | Set to `true` to enable the Prometheus `/metrics` endpoint |
    107 
    108 ### users.json Fields
    109 
    110 | Field | Type | Purpose |
    111 |---|---|---|
    112 | `slug` | `Text` | URL slug (`""` for root user, `"filip"` for `/u/filip`) |
    113 | `name` | `Optional Text` | Display name for the user (defaults to slug if not provided) |
    114 | `listenbrainzUser` | `Optional Text` | ListenBrainz username |
    115 | `lastfmUser` | `Optional Text` | Last.fm username |
    116 | `databaseFile` | `Text` | DuckDB filename (relative to `DATABASE_PATH`) |
    117 | `coverCacheEnabled` | `Bool` | Enable S3 cover art caching |
    118 | `backupEnabled` | `Bool` | Enable periodic S3 database backups |
    119 | `backupIntervalHours` | `Natural` | Backup frequency |
    120 
    121 ## Data Flow
    122 
    123 ### Scrobble Synchronization
    124 
    125 Both sync processes follow the same pattern: fetch the most recent page, insert any new scrobbles, and paginate backwards through history until an already-known timestamp is encountered.
    126 
    127 **ListenBrainz** (timestamp-based pagination):
    128 1. Fetch latest 100 scrobbles from the ListenBrainz API.
    129 2. Insert new scrobbles; stop if an existing timestamp is found.
    130 3. Paginate backwards using `max_ts` until fully caught up.
    131 
    132 **Last.fm** (page-based pagination):
    133 1. Fetch page 1 (most recent 200 scrobbles) from the Last.fm API.
    134 2. Insert new scrobbles; stop if an existing timestamp is found.
    135 3. Paginate through subsequent pages using `totalPages` from the API response until fully caught up.
    136 
    137 Both processes run every 60 seconds per user. On subsequent syncs they stop at the first known timestamp, making incremental updates efficient.
    138 
    139 ### Metadata Enrichment
    140 1. Background task identifies MBIDs in `scrobbles` that are not in `release_metadata`.
    141 2. Queries MusicBrainz API for release details.
    142 3. If MusicBrainz lacks genre information, falls back to Last.fm and Discogs APIs.
    143 4. Updates `release_metadata` with found information.
    144 
    145 ### Cover Art Retrieval
    146 When a cover is requested:
    147 1. Check S3 cache.
    148 2. If not found:
    149     - Try **Cover Art Archive (CAA)** using the Release MBID.
    150     - Fallback to **Last.fm** using Artist/Album name.
    151     - Final fallback to **Discogs** search API.
    152 3. If found in any source, the image is proxied to the client and uploaded to S3 in the background.
    153 
    154 ## Observability
    155 
    156 ### Prometheus Metrics
    157 
    158 Prometheus metrics are **disabled by default**. Set `METRICS_ENABLED=true` to enable them. When enabled, all HTTP requests are instrumented via `Metrics.wrapRequest` and background work is tracked with dedicated counters and gauges. When disabled, the `/metrics` endpoint returns 404 and all metric-increment calls are no-ops with no runtime overhead.
    159 
    160 | Metric | Type | Labels | Description |
    161 |---|---|---|---|
    162 | `corpus_http_requests_total` | Counter | `method`, `path`, `status` | HTTP requests |
    163 | `corpus_http_request_duration_seconds` | Histogram | `method`, `path` | HTTP request latency |
    164 | `corpus_sync_runs_total` | Counter | `user`, `source`, `result` | Sync loop iterations |
    165 | `corpus_sync_scrobbles_added_total` | Counter | `user`, `source` | Scrobbles inserted per sync |
    166 | `corpus_sync_last_success_seconds` | Gauge | `user`, `source` | Timestamp of last successful sync |
    167 | `corpus_enrichment_fetches_total` | Counter | `user`, `source`, `result` | Metadata enrichment API calls |
    168 | `corpus_enrichment_queue_size` | Gauge | `user`, `type` | Releases pending enrichment |
    169 | `corpus_cover_requests_total` | Counter | `user`, `source`, `result` | Cover art requests |
    170 | `corpus_db_backup_runs_total` | Counter | `user`, `result` | Database backup runs |
    171 | `corpus_db_backup_last_success_seconds` | Gauge | `user` | Timestamp of last successful backup |
    172 
    173 Node.js default metrics (GC, event loop, memory) are also collected via `prom-client`'s `collectDefaultMetrics`.
    174 
    175 ## Tech Stack
    176 
    177 - **Language**: [PureScript](https://purescript.org) (server), [Elm](https://elm-lang.org) (frontend)
    178 - **Runtime**: [Node.js](https://nodejs.org)
    179 - **Database**: [DuckDB](https://duckdb.org) (one file per user)
    180 - **Config**: JSON (`users.json`, managed via CLI)
    181 - **Bundling**: [spago](https://github.com/purescript/spago) + [esbuild](https://esbuild.github.io/) (server), [elm make](https://guide.elm-lang.org/install/elm.html) (frontend)
    182 - **Environment**: [Nix](https://nixos.org) for reproducible development shells and container builds
    183 
    184 ## Foreign Function Interface (FFI)
    185 
    186 Corpus relies on FFI to interact with the Node.js ecosystem where native PureScript wrappers are unavailable. Key FFI integrations:
    187 
    188 - **Database (`Db.js`)**: Interface to the `@duckdb/node-api` library. Includes BigInt → Number conversion for JSON compatibility.
    189 - **Cloud Storage (`S3.js`)**: AWS SDK (`@aws-sdk/client-s3`) for cover art caching. Takes explicit config structs rather than reading `process.env`.
    190 - **System Utilities (`Main.js`)**: Bridges PureScript with Node.js — `dotenv` loading and request helpers.
    191 - **Config (`Config.js`)**: Reads and parses `users.json` from the path given by `CORPUS_USERS_FILE`.
    192 - **Observability (`Metrics.js`)**: Initialises `prom-client` (Prometheus). Exports metric-increment helpers called from PureScript and the `wrapRequest` function that records metrics and logs each HTTP request.
    193 
    194 ## System Flow
    195 
    196 See [`architecture.dot`](architecture.dot) (render with `just docs`).
    197 
    198 ![System Flow](architecture.svg)