Self-hosted deduplication for large image, animated-image, and video libraries. One Docker container, a browser UI, and a review workflow built for collections of around a million files.
VisualStore is the successor to ImageStore, a Windows PowerShell module that found byte-identical files with SHA-1 and visually similar images with pHash. VisualStore keeps ImageStore's domain rules and can import its database, but is a rewrite: cross-platform, browser-operated, and with a similarity engine that treats stills, animated images, and video frames in one vector space.
Status: all six milestones shipped (2026-11-07). Scanning with move detection, SHA-1 and visual-similarity groups (stills, animated images and video frames in one vector space), the three-keystroke review with image comparison, bulk rules, trash with restore, archive move, v1 import with a model evaluation report, index memory options, maintenance, a non-root image with a health check, and a user guide. What remains is measuring on real data; see the roadmap and
docs/user/.
- Finds duplicates two ways in one pipeline. Byte-identical files (SHA-1) and visually similar media (deep copy-detection features with nearest-neighbour search over a vector index) land in the same groups and the same review UI.
- Compares across media types. Still images, animated images, and video all take part. A screenshot can match the frame of a video it came from.
- Scores you can act on in bulk. Similarity falls into stable bands: identical, near-duplicate, variant, candidate. Byte-identical groups resolve automatically; near-duplicates are preselected for one-key confirmation; variants get a human look.
- Is built for CG libraries. A Folder can be told not to compare files within the same directory, so intentional variants are not flagged against each other. Folders have an archive or inbox role that drives which copy is kept by default.
- Keeps the fixed workflow short. Ingest → compute → find duplicates → resolve → move to archive, all in the browser. Resolving a group takes about three keystrokes.
- Does not lose work. Deletion goes to a trash directory with a retention period. Moving or renaming files on disk is detected on the next scan and keeps hashes and judgments. Scheduled SQLite backups keep the most recent N copies.
- Runs where the files are. A single container on a NAS or home server with media mounted as volumes. The first full computation can run on a faster machine and the data directory then moves to the NAS as-is.
- Hosts several libraries. Profiles isolate databases, settings, and Folders for unrelated collections inside one service instance.
browser ──HTTP──► server (FastAPI + static React frontend)
│ API, scan, file actions, grouping, backup, scheduling
└─► compute process pool
SHA-1, decode, frame sampling, ONNX inference, thumbnails, index queries
/data SQLite metadata and vectors per profile, usearch index, thumbnails
/backups timestamped SQLite backups
/media/... your libraries, mounted read-write
One image, one container, one process tree. The server is the only database writer; compute children receive work over in-process queues. No external broker, no database server.
Stack: Python 3.12, FastAPI, SQLite (WAL), ONNX Runtime, pyvips, ffmpeg, usearch, React + TypeScript + Vite. Multi-arch image for linux/amd64 and linux/arm64. Details in Architecture.
| Variable | Description |
|---|---|
VISUALSTORE_PASSWORD |
Login password. Required. |
VISUALSTORE_DATA_DIR |
Data directory, default /data. |
VISUALSTORE_BACKUP_DIR |
Backup directory, default /backups. |
VISUALSTORE_COMPUTE_PROCESSES |
Compute children, default CPU count. |
VISUALSTORE_PORT |
Listen port, default 8080. |
Everything else is a per-profile setting in the UI. A compose example is in Operations. Images are published to ghcr.io/secretnest/visualstore and to Alibaba Cloud Container Registry under the same path.
cd docker
cp ../.env.example .env # set VISUALSTORE_PASSWORD in docker/.env
docker compose up -d # data in docker/data, backups in docker/backupsKeep /data on a local disk, mount the media at container paths you will keep on every machine,
and put a TLS-terminating reverse proxy in front. The container runs as PUID:PGID (default
1000:1000) and ships both similarity models; nothing is downloaded at run time.
The user guide under docs/user/ covers the rest:
- Deployment: image, compose, environment, permissions, proxy, health.
- Initialise and migrate: first computation on a strong machine, stop-and-copy to the NAS.
- Backups and restore.
- Import from ImageStore, including the model evaluation report.
- Review workflow: queue, three-keystroke group detail, frame strip, candidates, bulk rules, trash.
- Settings: every key, index memory options, sizing the compute pool.
Development: uv sync, cd web && npm ci && npm run build, then
VISUALSTORE_PASSWORD=pw VISUALSTORE_DATA_DIR=./data VISUALSTORE_BACKUP_DIR=./backups uv run python -m visualstore.
Tests: uv run pytest, cd web && npm test, cd web && npm run e2e (media tests need libvips and
ffmpeg; without them locally they run in the visualstore:test image, see docker/Dockerfile).
Everything lives under docs/:
docs/design/: the specification, by chapter. Start with Overview.docs/decisions/: architecture decision records.docs/plans/: implementation plans, one per milestone.
- Skeleton: container, profiles, migrations, login, job system, backup, frontend shell.
- Byte-identical pipeline with ImageStore import. Replaces ImageStore's same-file workflow.
- Similarity engine: ONNX inference, vector index, model evaluation.
- Similarity review UI: image comparison, shortcuts, bulk rules.
- Video and animated images.
- Performance, maintenance, user documentation, migration rehearsal.
The ImageStore SQLite database imports in one step: Folder configuration, files, SHA-1 hashes, and byte-identical groups are kept, including their ids. pHash values are dropped and similarity is recomputed with the new model. ImageStore's manual similarity judgments are used to evaluate the new model and can optionally be imported. See v1 Data Import.