Click to expand — full deployment runbook

\n\n> Runs on `/home/ubuntu/nse-public-mcp/` — completely isolated from `/home/ubuntu/swingtrader/`.\n\n### 1. Clone\n\n```bash\ncd /home/ubuntu\ngit clone https://github.com/alokbarnwal/nse-public-mcp.git\ncd nse-public-mcp\n```\n\n### 2. Create database and user\n\n```bash\nsudo mysql <<'SQL'\nCREATE DATABASE IF NOT EXISTS nse_public CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;\nCREATE USER IF NOT EXISTS 'nse_writer'@'localhost' IDENTIFIED BY 'CHANGE_ME_STRONG_PASSWORD';\nGRANT ALL PRIVILEGES ON nse_public.* TO 'nse_writer'@'localhost';\nFLUSH PRIVILEGES;\nSQL\n```\n\n### 3. Apply schema\n\n```bash\nmysql -u nse_writer -p nse_public < db/schema.sql\nmysql -u nse_writer -p nse_public < db/schema_indicators.sql\nmysql -u nse_writer -p nse_public < db/schema_price_action.sql\n```\n\n### 4. Install dependencies\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n### 5. Configure environment\n\n```bash\ncp .env.example .env\nchmod 600 .env\nnano .env # set DB_PASSWORD to match step 2\n```\n\n### 6. (Optional) Refresh full Nifty 500 list\n\nThe repo ships with the top ~100 stocks hardcoded. To expand to all 500:\n\n```bash\npython -m config.stocks --refresh # fetches NSE archives, rewrites config/stocks.py\n```\n\n### 7. Load stock universe into DB\n\n```bash\npython -m config.stocks --load-to-db\n```\n\n### 8. Backfill historical data (long-running, use screen/tmux)\n\n```bash\nscreen -S backfill\nsource .venv/bin/activate\npython -m data.backfill --all\n# Ctrl-A D to detach. Reattach: screen -r backfill\n```\n\nExpected runtime:\n- `daily 5y × 500`: ~30 min\n- `15min 2y × 500`: ~2 h\n- `5min 6mo × 500`: ~1 h\n\nIf anything fails, the failure list is written to `backfill_failures_.json`. Re-run:\n\n```bash\npython -m data.backfill --timeframe 15min --resume\n```\n\n### 8b. Bulk-compute indicators (long-running, use screen/tmux)\n\nAfter the candle backfill completes, populate the `indicators` table for all 500 stocks across all 3 timeframes:\n\n```bash\nscreen -S indicators\ncd /home/ubuntu/nse-public-mcp\npython3 -m indicators.runner --all\n# Ctrl-A D to detach\n```\n\nExpected runtime: ~30–45 min. Idempotent — safe to re-run if interrupted.\n\n### 8c. Bulk-compute price action (long-running, use screen/tmux)\n\nAfter indicators are populated, compute price-action features (zones, S/R,\npatterns, breakouts, fibs, gaps, volume profile) for all 500 stocks across\nall 3 timeframes:\n\n```bash\nscreen -S price_action\ncd /home/ubuntu/nse-public-mcp\npython3 -m price_action.runner --timeframe daily --all-features\npython3 -m price_action.runner --timeframe 15min --all-features\npython3 -m price_action.runner --timeframe 5min --all-features\n# Ctrl-A D to detach\n```\n\nExpected runtime: ~60–90 min total. Idempotent — safe to re-run if interrupted.\n\n### 9. Install crontab\n\n```bash\ncrontab -e\n```\n\nAdd:\n\n```cron\n# nse-public-mcp candle updaters\n*/5 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_5min >> logs/cron_5min.log 2>&1\n*/15 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_15min >> logs/cron_15min.log 2>&1\n0 18 * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_daily >> logs/cron_daily.log 2>&1\n\n# nse-public-mcp indicator updaters (each runs after the matching candle fetch)\n30 18 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_daily >> logs/indicators_daily.log 2>&1\n20,35,50 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_15min >> logs/indicators_15min.log 2>&1\n5 16 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_15min >> logs/indicators_15min.log 2>&1\n2-57/5 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_5min >> logs/indicators_5min.log 2>&1\n\n# nse-public-mcp price-action updaters (each runs after the matching indicator cron)\n35 18 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_daily >> logs/price_action_daily.log 2>&1\n25,55 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_15min >> logs/price_action_15min.log 2>&1\n9-54/15 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_5min >> logs/price_action_5min.log 2>&1\n```\n\n> Cron times are server-local. Verify the server is on IST (`timedatectl`) — if not, shift the hour ranges.\n> The 5-min indicator cron is offset to `2-57/5` so it runs ~2 minutes after `cron_5min` and never reads stale candles.\n> The price-action 5-min cron is offset to `9-54/15` (minutes 9, 24, 39, 54) so it runs ~2 minutes after each `cron_indicators_5min` slot.\n\n### 10. Verification\n\n```bash\nmysql -u nse_writer -p nse_public <<'SQL'\nSELECT 'daily' AS tf, COUNT(*) AS n FROM candles_daily\nUNION ALL SELECT '15min', COUNT(*) FROM candles_15min\nUNION ALL SELECT '5min', COUNT(*) FROM candles_5min;\n\n-- Per-stock coverage on daily — flag any with < 1000 rows (~4 trading years)\nSELECT symbol, COUNT(*) AS n\nFROM candles_daily\nGROUP BY symbol\nHAVING n < 1000\nORDER BY n;\n\n-- Latest data freshness\nSELECT MIN(candle_date) AS earliest, MAX(candle_date) AS latest\nFROM candles_daily;\n\n-- Recent cron health\nSELECT job_type, timeframe, symbols_success, symbols_failed, started_at, duration_seconds\nFROM fetch_log\nORDER BY started_at DESC\nLIMIT 20;\nSQL\n```\n\nDB size on disk:\n\n```bash\nsudo du -sh /var/lib/mysql/nse_public/\n```\n\n

\n\n---\n\n## Service Management\n\n```bash\n# Server status\nsudo systemctl status nse-public-mcp\n\n# Restart\nsudo systemctl restart nse-public-mcp\n\n# Logs\nsudo journalctl -u nse-public-mcp -n 100 -f\n\n# nginx\nsudo nginx -t && sudo systemctl reload nginx\n\n# fail2ban\nsudo fail2ban-client status nginx-mcp-ratelimit\n```\n\n---\n\n## Layout\n\nSee [PROJECT.md](https://github.com/alokbarnwal/nse-public-mcp/blob/main/PROJECT.md) for architecture and data-flow diagram.\n\n```\nconfig/ # settings, stock universe, holiday calendar\ndata/ # yfinance fetcher, backfill CLI, shared upsert helpers\ndb/ # schema and pooled connection\nindicators/ # technical indicator compute, persist, runner\nprice_action/ # zones, patterns, levels, trends\nmcp_server/ # FastMCP server + 12 tool modules\njobs/ # cron entrypoints (candles + indicators + price action)\ndeploy/ # systemd service, nginx config, fail2ban rules\ntests/ # mocked unit tests (82 tests)\n```\n\n---\n\n## Operational Notes\n\n- **Be respectful to Yahoo Finance.** The fetcher sleeps 1–3 s between requests and retries with backoff. Don't reduce these.\n- **Idempotent.** All inserts are `INSERT … ON DUPLICATE KEY UPDATE`. Replaying a cron tick or rerunning backfill is safe.\n- **Holiday calendar** lives in `config/holidays.py`. Update it once a year when NSE publishes the next year's list.\n- **Retention cleanup** runs at the end of `jobs/cron_daily.py`. Daily candles older than 5 y, 15min older than 2 y, 5min older than 6 mo are purged.\n\n---\n\n## License\n\n[MIT](https://github.com/alokbarnwal/nse-public-mcp/blob/main/LICENSE) — free to use, modify, and distribute.\n\n## Disclaimer\n\nThis project is for **educational and informational purposes only**. Not investment advice. See [DISCLAIMER.md](https://github.com/alokbarnwal/nse-public-mcp/blob/main/DISCLAIMER.md).\n","changelogMarkdown","videoKey","requiredSecretsJson","publishedAt","priceCents","trialDays",0,"iconType","image","upstream",{"_100":101,"_102":103,"_104":105,"_106":107,"_108":-5,"_109":110},"requiredSecrets",[],"tags",[],"assets",[],"stats",{"_98":85,"_99":85},"ratingAvg","reviewCount","topics",[],"license","MIT","language","Python","registryName","io.github.alokbarnwal/nse-public-mcp","homepage","packages",[],"ENV",{"_113":114,"_115":116,"_117":118},"ENVIRONMENT","production","TURNSTILE_SITE_KEY","0x4AAAAAAB8-rEI2gNDf_QhU","SENTRY_DSN","https://45fd1229f78089d5c3ee78be128030e1@o4510583261757440.ingest.de.sentry.io/4510584253448272"]

Indian NSE Stock Insights