NicheDB

Sources in, feeds out. An open platform for databases that only ever grow: game releases, package registries, government filings, and any other public data with a big seed and daily growth. One Postgres, one Redis, and every row reachable by web, RSS, JSON, API, CLI and MCP.

The first deployment is nichedb.dev. Run your own on anything that has Postgres and Redis.

The model

Adapters are one file each in packages/adapters/src. Twenty-nine ship today across thirteen collections:

Enrichment

After ingest, every item is enriched by the enrichers that apply to it (packages/enrichers/src), and the results live on the item under enrichment.<name>:

A feed's enrichers list picks which of these it shows; absent means the collection's defaults. The feed builder exposes them as checkboxes. Items are enriched once, newest first with a fair share per collection, and a missing image, summary or tags are filled from whatever the enrichers found while the source's own words always win.

Run it

git clone https://github.com/profullstack/niche-db && cd niche-db
bun install
docker compose up -d db redis          # or point at your own Postgres 16+ and Redis
cp .env.example .env                    # DATABASE_URL, SITE_URL, CONTACT_EMAIL at minimum
bun run build:client
bun run dev                             # web + worker in one process on :3000

Migrations apply themselves on boot. The three collections, their default sources and a dozen feeds are seeded on first boot; sources whose adapter needs a credential the deployment lacks are created paused. The first account to sign in is an admin.

bun run ingest [slug ...] runs sources from a terminal without Redis. bun test runs the suite against an in-process Postgres (PGlite), so it needs no server.

Surfaces

• Web: a PWA. /sources manages sources (status, last run, next run, run now, pause, edit config). /feeds/new builds a feed with a live preview. /settings holds passkeys, API keys and notifications.
• RSS / JSON Feed: /f/<slug>.rss, /f/<slug>.json.
• API: /api/v1, documented at /docs/api. Reads need no key.
• CLI: npm i -g @profullstack/nichedb → nichedb. Documented at /docs/cli.
• MCP: /mcp (Streamable HTTP, stateless) or nichedb mcp over stdio. Documented at /docs/mcp.
• llms.txt: /llms.txt describes the whole deployment in one document.

Accounts and money

Magic link and passkey only; there is no password column. API keys are minted from settings and shown once.

Free pages and feeds carry one CrawlProof ad and a tracker (CRAWLPROOF_AD_SLOT, CRAWLPROOF_SITE_ID). Pro ($120 a month through CoinPay) is the tier without them: no ads, no tracking, the high API rate limit, unlimited feeds, own sources, and a crawl pass for the whole term (GET /api/v1/crawl-pass) so a member's own agents walk through the paywall. Referral links give the new customer 20% off and the referrer 60% of the first payment.

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent…) get a 402 with an x402 offer and buy a pass at /crawl: $1 a day for everything, and the more a buyer has paid here the less a day costs (CRAWL_LOYALTY, default 20% off after $10, 40% after $50, 60% after $100; every sale is a row in crawl_sales). A pass may switch ads or tracking off for its own requests with ?disable=ads,tracking. People, search engines and retrieval crawlers pass through untouched.

Stack

Bun, Hono (server-rendered JSX), Postgres via Bun's native driver, BullMQ on Redis, Biome. bun apps/worker/src/enrich-cli.js [n] runs enrichment from a terminal. Shared Profullstack modules: @profullstack/x402-gateway, @profullstack/emailer, @profullstack/coinpay, @profullstack/referrals, @profullstack/api-key-manager, @profullstack/autoblog (signed webhooks), @profullstack/favicon-generator.

Configuration

Everything is read once from the environment in packages/config/src/index.js; .env.example lists every variable. Production secrets live in the nichedb--prod logicsrc vault:

bun run secrets:pull     # logicsrc teams pull profullstack nichedb prod --env .env
bun run secrets:push     # after adding a key locally

ROLES=web,worker runs everything in one container; ROLES=worker on a second service splits ingestion out when one instance stops being enough.

Deploy

One Dockerfile, one Railway service, plus Railway's Postgres and Redis. railway.json sets the healthcheck at /healthz. Set SITE_URL to the public origin before anyone registers a passkey: the passkey relying-party id is derived from it.

Adding an adapter

import { defineAdapter } from '@nichedb/core/adapter';

export const example = defineAdapter({
  name: 'example',
  title: 'Example API',
  collection: 'packages',
  description: 'What it watches, in one paragraph.',
  kinds: ['thing'],
  cadenceMinutes: 30,
  configFields: [{ key: 'topic', label: 'Topic', required: true }],
  defaultSources: [{ slug: 'example-default', name: 'Example', config: { topic: 'all' } }],
  async pull({ config, cursor, http, log, budget, deadline }) {
    const page = await http.json(`https://example.com/api?topic=${config.topic}&since=${cursor.since ?? ''}`);
    return {
      items: page.results.map((r) => ({ externalId: r.id, kind: 'thing', title: r.name, url: r.url, publishedAt: r.at, tags: r.tags, data: r })),
      cursor: { since: page.next },
      note: `${page.results.length} things`,
    };
  },
});

Register it in packages/adapters/src/index.js. The core normalises items, hashes them so unchanged rows cost no write, writes in batches, records the run and reschedules.

License

MIT