commit 2cc4891e3fef66a340ef4634c5a0c42b6d33d71f
parent 978298d616a5c72ef42e0ca57522ca83a23aab7c
Author: mtmn <miro@haravara.org>
Date: Sun, 12 Apr 2026 21:25:30 +0200
docs: add and reference docs, just usage
Diffstat:
| M | README.md | | | 48 | +++++++++++++++++++++++++++++++++++++++++++++++- |
| A | docs/architecture.md | | | 125 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
| A | docs/duckdb.md | | | 78 | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
| M | justfile | | | 23 | ++++++++++++++++++++++- |
4 files changed, 272 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
@@ -2,4 +2,50 @@
| | |
| :--- | ---: |
-| **Scorpus** is an alternative [ListenBrainz](https://listenbrainz.org) dashboard written in [PureScript](https://www.purescript.org). It stores listens metadata in [DuckDB](https://duckdb.org) and caches cover art in a blob storage.
+| **Scorpus** is a personal [ListenBrainz](https://listenbrainz.org) dashboard. It syncs scrobbles, stores metadata in [DuckDB](https://duckdb.org), and caches cover art in S3 to provide a fast, searchable interface for your music history.<br><br>Features include automated syncing, rich metadata enrichment from MusicBrainz, and an interactive [PureScript](https://purescript.org)/Halogen frontend for deep exploration of your listening habits. | <img src="docs/korpus.webp" width="400" alt="Korpus"> |
+
+## Documentation
+
+- [Architecture](docs/architecture.md) — Deep dive into the system components, data flow, and FFI usage.
+- [DuckDB](docs/duckdb.md) — Schema details, analytical queries, and tools for data exploration.
+
+## Usage
+
+This project uses [just](https://github.com/casey/just) and [Nix](https://nixos.org) for development and deployment.
+
+### Development
+
+```bash
+# Enter the development shell (includes PureScript, DuckDB, etc.)
+just shell
+
+# Build the project (server and client)
+just build
+
+# Run the binary built by Nix
+./result/bin/scorpus-server
+```
+
+### Spago Build (PureScript)
+
+If you prefer to run PureScript commands manually:
+
+```bash
+# Install dependencies
+npx spago install
+
+# Build the project
+npx spago build
+
+# Run tests
+npx spago test
+
+# Bundle the client for the browser
+npx spago bundle --module Client --outfile client.js --platform browser
+```
+
+
+Required environment variables:
+- `LISTENBRAINZ_USER`: Your ListenBrainz username.
+- `DATABASE_FILE`: Path to the DuckDB file (defaults to `scorpus.db`).
+- `S3_BUCKET`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, etc. (for cover art caching).
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -0,0 +1,125 @@
+# Scorpus Architecture
+
+Scorpus is a personal music listening history dashboard and analytics service. It synchronizes scrobbles from ListenBrainz and provides a performant web interface for data exploration and statistics.
+
+## System Components
+
+### Web Server
+The server is built with PureScript running on Node.js. It handles several core responsibilities:
+- **HTTP API**: Serves the frontend, scrobble data (with filtering/pagination), and statistics.
+- **ListenBrainz Sync**: A background process that polls the ListenBrainz API every 60 seconds to fetch new scrobbles.
+- **Metadata Enrichment**: A background process that identifies scrobbles with missing metadata (genres, labels, release years) and fetches information from MusicBrainz, Last.fm, and Discogs.
+- **Cover Art Proxy**: A specialized endpoint that fetches, caches, and serves cover art, utilizing a multi-source fallback strategy (CAA → Last.fm → Discogs).
+
+### Frontend
+A Single Page Application (SPA) built with PureScript and the [Halogen](https://github.com/purescript-halogen/purescript-halogen) framework.
+- **Real-time Updates**: Periodically refreshes the scrobble list.
+- **Filtering & Search**: Supports deep filtering by genre, label, or release year.
+- **Responsive UI**: Designed for both desktop and mobile viewing with a "retro-modern" aesthetic.
+
+### Database
+Scorpus uses **DuckDB** for its primary data storage.
+- **Schema**:
+ - `scrobbles`: Stores the core listening history (timestamp, track, artist, album, MBIDs).
+ - `release_metadata`: Stores enriched metadata indexed by MusicBrainz Release ID (MBID).
+- **Performance**: DuckDB's columnar storage allows for extremely fast analytical queries across large listening histories.
+
+### Storage
+Uses an S3-compatible bucket to cache cover art images.
+- **Caching Strategy**: Images are fetched once from external APIs and stored in S3 to reduce latency and avoid rate-limiting on external services.
+
+## Data Flow
+
+### Scrobble Synchronization
+1. Server triggers sync process.
+2. Fetches latest 100 scrobbles from ListenBrainz.
+3. Performs a "gap-fill" by paginating backwards if the local database is significantly behind.
+4. Stores new scrobbles in the DuckDB `scrobbles` table.
+
+### Metadata Enrichment
+1. Background task identifies MBIDs in `scrobbles` that are not in `release_metadata`.
+2. Queries MusicBrainz API for release details.
+3. If MusicBrainz lacks genre information, falls back to Last.fm and Discogs APIs.
+4. Updates `release_metadata` with found information.
+
+### Cover Art Retrieval
+When a cover is requested:
+1. Check S3 cache.
+2. If not found:
+ - Try **Cover Art Archive (CAA)** using the Release MBID.
+ - Fallback to **Last.fm** using Artist/Album name.
+ - Final fallback to **Discogs** search API.
+3. If found in any source, the image is proxied to the client and uploaded to S3 in the background.
+
+## Tech Stack
+
+- **Language**: [PureScript](https://purescript.org) (strongly typed functional programming).
+- **Frontend Framework**: [Halogen](https://github.com/purescript-halogen/purescript-halogen).
+- **Runtime**: [Node.js](https://nodejs.org).
+- **Database**: [DuckDB](https://duckdb.org).
+- **Bundling**: [spago](https://github.com/purescript/spago) and [esbuild](https://esbuild.github.io/).
+- **Environment**: [Nix](https://nixos.org) for reproducible development shells and container builds.
+
+## Foreign Function Interface (FFI)
+
+Scorpus relies on FFI to interact with the Node.js and browser ecosystems. Key FFI integrations include:
+
+- **Database (`Db.js`)**: Provides a high-performance interface to the native `duckdb` library. It includes custom logic to handle BigInt conversions, ensuring database results are compatible with standard JSON serialization.
+- **Cloud Storage (`S3.js`)**: Leverages the official AWS SDK (`@aws-sdk/client-s3`) to manage cover art caching in S3-compatible storage.
+- **System Utilities (`Main.js`)**: Bridges PureScript with essential Node.js functionality, including environment variable management (`dotenv`), buffer operations, and URL parsing.
+- **Browser Integration (`Client.js`)**: Manages client-side concerns such as URL parameter extraction and history state manipulation.
+
+## System Flow
+
+```mermaid
+graph TD
+ subgraph External APIs
+ LB[ListenBrainz API]
+ MB[MusicBrainz API]
+ LF[Last.fm API]
+ DC[Discogs API]
+ CAA[Cover Art Archive]
+ end
+
+ subgraph Scorpus Server
+ Sync[Sync Process]
+ Enrich[Enrichment Task]
+ Proxy[Cover Proxy]
+ API[Web API]
+ end
+
+ subgraph Storage
+ DB[(DuckDB)]
+ S3[[S3 Bucket]]
+ end
+
+ subgraph Frontend
+ UI[Halogen SPA]
+ end
+
+ %% Scrobble Sync Flow
+ LB -- "Fetch scrobbles" --> Sync
+ Sync -- "Store scrobbles" --> DB
+
+ %% Enrichment Flow
+ DB -- "Get MBIDs" --> Enrich
+ Enrich -- "Metadata" --> MB
+ Enrich -- "Fallback Genre" --> LF
+ Enrich -- "Fallback Genre" --> DC
+ Enrich -- "Store Metadata" --> DB
+
+ %% Web UI Flow
+ UI -- "Request Data" --> API
+ API -- "Query" --> DB
+ DB -- "Results" --> API
+ API -- "JSON" --> UI
+
+ %% Cover Art Flow
+ UI -- "Request Cover" --> Proxy
+ Proxy -- "1. Check Cache" --> S3
+ Proxy -- "2. Fallback CAA" --> CAA
+ Proxy -- "3. Fallback Last.fm" --> LF
+ Proxy -- "4. Fallback Discogs" --> DC
+ Proxy -- "Cache Result" --> S3
+ Proxy -- "Serve Image" --> UI
+```
diff --git a/docs/duckdb.md b/docs/duckdb.md
@@ -0,0 +1,78 @@
+# DuckDB in Scorpus
+
+Scorpus uses [DuckDB](https://duckdb.org/) as its primary analytical database. DuckDB's columnar storage and efficient query engine allow Scorpus to provide fast filtering, pagination, and statistics over large sets of listening history data.
+
+## Database Schema
+
+The database consists of two main tables:
+
+### `scrobbles`
+Stores the raw listening history synced from ListenBrainz.
+
+| Column | Type | Description |
+| :--- | :--- | :--- |
+| `listened_at` | BIGINT | Unix timestamp (Primary Key) |
+| `track_name` | VARCHAR | Name of the track |
+| `artist_name` | VARCHAR | Name of the artist |
+| `release_name` | VARCHAR | Name of the album/release |
+| `release_mbid` | VARCHAR | MusicBrainz Release ID |
+| `caa_release_mbid` | VARCHAR | Cover Art Archive Release ID |
+
+### `release_metadata`
+Stores enriched metadata fetched from MusicBrainz, Last.fm, and Discogs.
+
+| Column | Type | Description |
+| :--- | :--- | :--- |
+| `release_mbid` | VARCHAR | MusicBrainz Release ID (Primary Key) |
+| `genre` | VARCHAR | Primary genre |
+| `label` | VARCHAR | Record label |
+| `release_year` | INTEGER | Year of release |
+| `genre_checked_at` | INTEGER | Timestamp of last enrichment attempt |
+
+## Application Usage
+
+The application interacts with DuckDB via a PureScript FFI layer (`src/Db.js` and `src/Db.purs`).
+- **BigInt Handling**: Since DuckDB returns `BIGINT` as JavaScript `BigInt`, the FFI layer converts these to `Number` to ensure compatibility with standard JSON serialization.
+- **Background Enrichment**: The server identifies "unenriched" scrobbles (those with an MBID but no metadata) and performs background updates to the `release_metadata` table.
+
+## Common Analytical Queries
+
+You can run these queries directly against your `scorpus.db` file using the DuckDB CLI or any compatible tool.
+
+### Top 10 Artists of All Time
+```sql
+SELECT artist_name, count(*) as play_count
+FROM scrobbles
+GROUP BY artist_name
+ORDER BY play_count DESC
+LIMIT 10;
+```
+
+### Listening Activity by Hour
+```sql
+SELECT
+ extract('hour' from to_timestamp(listened_at)) as hour,
+ count(*) as count
+FROM scrobbles
+GROUP BY hour
+ORDER BY hour;
+```
+
+### Genre Distribution
+```sql
+SELECT rm.genre, count(*) as count
+FROM scrobbles s
+JOIN release_metadata rm ON s.release_mbid = rm.release_mbid
+WHERE rm.genre IS NOT NULL
+GROUP BY rm.genre
+ORDER BY count DESC;
+```
+
+### MBID Enrichment Coverage
+```sql
+SELECT
+ count(*) as total,
+ count(release_mbid) FILTER (WHERE release_mbid != '') as with_mbid,
+ (count(release_mbid) FILTER (WHERE release_mbid != '')::FLOAT / count(*)) * 100 as percentage
+FROM scrobbles;
+```
diff --git a/justfile b/justfile
@@ -1,17 +1,38 @@
help:
@just --list
+# Enter the Nix development shell
shell:
nix develop
-build:
+# Build the project using Nix
+nix-build:
nix build .
+# Build the project locally (using npm)
+build:
+ npm run build
+
+# Run the project locally
+run:
+ npm start
+
+# Format PureScript source code
+tidy:
+ npm run tidy
+
+# Run tests
+test:
+ npm test
+
+# Build the container image using Nix
container:
nix build .#container
+# Load the built container image into podman
load: container
podman load < result
+# Push the container image to the registry
push: container
skopeo copy --dest-precompute-digests docker-archive:result docker://ghcr.io/mtmn/scorpus:latest
