[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_103":104},"routes/modules.$creator.$moduleId",{"_14":15,"_16":17,"_18":19,"_20":21},"routes/modules.$creator.$moduleId.endpoints",{"_11":12,"_13":-5},"error","Module '@pareelamre/analyzing-llm-rationale' is not configured","tools","module",{"_20":22,"_23":24,"_25":26,"_27":28,"_29":30,"_31":32,"_33":26,"_34":-5,"_35":36,"_37":-5,"_38":-5,"_39":40,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":48,"_56":57,"_58":59,"_60":-5,"_61":-5,"_62":-5,"_63":-5,"_64":65,"_66":67,"_68":69,"_70":71,"_72":73,"_74":-5,"_75":-5,"_76":-5,"_77":48,"_78":-5,"_79":57,"_80":81,"_82":83,"_84":85,"_86":87,"_88":89,"_90":91},"apiUrl","https://odel.app/api","creator","pareelamre","moduleId","analyzing-llm-rationale","@pareelamre/analyzing-llm-rationale","displayName","Foresea Forecasting","tagline","Forecast future events and scan prediction-market edges.","iconKey","https://avatars.githubusercontent.com/u/35176680?v=4","category","integrations","status","published","metaDescription","metaKeywords","deploymentType","external","workerName","bindingName","mcpConfig","{\"url\":\"https://foresea.ink/mcp/\",\"transport\":\"streamable-http\",\"authType\":\"none\"}","pricingModel","free","creatorUserId","system:external-import","createdAt",1781363298000,"updatedAt",1781499442000,"sourceRegistry","official-mcp-registry","sourceUrl","https://registry.modelcontextprotocol.io/v0/servers?search=ink.foresea%2Fforecasting","sourceVerified",false,"importedAt","upstreamStars",0,"upstreamUpdatedAt",1781490489000,"detectedTransport","detectedAuthType","probeStatus","lastProbedAt","githubRepoUrl","https://github.com/pareelamre/analyzing-llm-rationale","githubRepoFullName","pareelamre/analyzing-llm-rationale","releaseId","ext-pareelamre-analyzing-llm-rationale","version","1.0.0","descriptionMarkdown","# Analyzing LLM Rationale\n\nConference artifact for studying how explicit rationale instructions affect LLM\nforecasting behavior on Metaculus-style binary forecasting questions. The codebase\ncontains the prompt variants, batch inference runner, generated result tables, and\nplotting/analysis scripts used for the paper figures. The live Foresea API also\nsupports prediction-market intelligence: typed forecasts, evidence retrieval,\nand model-vs-market edge analysis for binary and multiple-choice markets.\n\n## Live API\n\nDeployed on [Google Cloud Run](https://cloud.google.com/run) — model `gpt-oss-120b`, variant `variant0_neutral_baseline`:\n\n```\nhttps://foresea.ink\n```\n\n*(The URL is printed in the GitHub Actions deploy-step output after the first push to `main`.)*\n\n```bash\n# Health check\ncurl https://foresea.ink/health\n\n# Single-record prediction\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"Will X happen by date Y?\",\n \"question_type\": \"binary\",\n \"description\": \"Context here.\",\n \"news_articles\": [],\n \"attach_evidence\": true,\n \"evidence_top_k\": 5,\n \"market_platform\": \"Polymarket\",\n \"market_probability\": 0.42,\n \"variant\": \"variant0_neutral_baseline\"\n }'\n```\n\nWhen `attach_evidence` is true and no `news_articles` are supplied, `/predict`\nfetches and ranks current news evidence from GDELT, Google News RSS, and Stooq by\ndefault, injects it into the model prompt, and returns the selected\n`evidence_articles` with the forecast. Supplying `news_articles` skips automatic\nretrieval and uses the caller-provided evidence.\n\nThe response includes both the forecast and the evidence used by the model:\n\n```json\n{\n \"question_type\": \"binary\",\n \"predicted_answer\": \"Yes\",\n \"confidence\": 0.86,\n \"options\": [],\n \"range_forecast\": null,\n \"rationale\": \"Model-generated explanation for the forecast.\",\n \"model_rationale\": \"Model-generated explanation for the forecast.\",\n \"variant\": \"variant0_neutral_baseline\",\n \"model_key\": \"gpt-oss-120b\",\n \"evidence_sources\": [\n {\n \"source\": \"Reuters\",\n \"title\": \"Article headline\",\n \"url\": \"https://example.com/article\",\n \"publish_date\": \"2026-05-29T00:00:00Z\",\n \"relevance_score\": 0.82\n }\n ],\n \"evidence_articles\": [\n {\n \"title\": \"Article headline\",\n \"summary\": \"Cleaned article summary.\",\n \"source\": \"Reuters\",\n \"url\": \"https://example.com/article\",\n \"publish_date\": \"2026-05-29T00:00:00Z\",\n \"relevance_score\": 0.82,\n \"search_query\": \"query used for retrieval\"\n }\n ],\n \"evidence_error\": null,\n \"market_analysis\": {\n \"platform\": \"Polymarket\",\n \"market_url\": \"https://example.com/market\",\n \"outcome\": \"Yes\",\n \"market_probability\": 0.42,\n \"model_probability\": 0.86,\n \"edge\": 0.44,\n \"stance\": \"model_above_market\",\n \"summary\": \"Foresea is 44 percentage points above the market on Yes.\"\n }\n}\n```\n\nUse `evidence_sources` when a client only needs the source list and links. Use\n`evidence_articles` when a client needs the article-level details that were\nattached to the model prompt. `rationale` and `model_rationale` are generated by\n`gpt-oss-120b` and explain why the model chose its answer and confidence.\nWhen `market_probability` is supplied, `market_analysis` is computed\ndeterministically from the model probability and the market-implied probability.\n\n## Production Deployment Notes\n\nProduction is served from the custom domain:\n\n```text\nhttps://foresea.ink\n```\n\nThe Cloud Run service name, project ID, and region are set at deploy time via `gcloud run deploy`.\n\nRequired runtime environment:\n\n- `SCADS_AI_API_KEY`: Secret Manager secret used by hosted model calls.\n- `MODEL_DEVICE=cpu`: production Cloud Run runs the CPU image.\n- `CUSTOM_DOMAIN=foresea.ink`: redirects `*.run.app` requests to the public domain.\n- `GOOGLE_CLIENT_ID`: Google OAuth web client ID used by `/auth/config`.\n- `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET`: GitHub OAuth app credentials. The\n OAuth app's callback URL must be the site origin (e.g. `https://foresea.ink/`).\n When unset, the \"Continue with GitHub\" button is hidden and `/auth/github`\n returns 503. Sign-in also works with Google and email/password.\n- `SESSION_SECRET`: long random string used to sign browser session JWTs.\n\nThe OAuth client must allow these JavaScript origins:\n\n```text\nhttps://foresea.ink\nhttps://www.foresea.ink\nhttps://.run.app\n```\n\nTo update non-secret environment variables without replacing the existing\n`SESSION_SECRET`, use `--update-env-vars`:\n\n```bash\ngcloud run services update \\\n --region \\\n --project \\\n --update-env-vars MODEL_DEVICE=cpu,CUSTOM_DOMAIN=foresea.ink,GOOGLE_CLIENT_ID=''\n```\n\nVerify the deployed auth config and health endpoint:\n\n```bash\ncurl https://foresea.ink/auth/config\ncurl https://foresea.ink/health\n```\n\n### Scaling and caching\n\nThe server is built to scale horizontally on Cloud Run:\n\n- **Authentication** supports Google One-Tap *and* email/password\n (`/auth/register`, `/auth/login`). Passwords are stored as salted\n PBKDF2-HMAC-SHA256 hashes; accounts live in Cloud Datastore.\n- **Caching and rate limiting** use Redis when `REDIS_URL` is set, so they are\n shared across instances; otherwise they fall back to per-instance in-memory\n state and fail open. `/predict` (non-personalised requests), evidence\n retrieval, and `/extract` URL fetches are cached; public GETs send\n `Cache-Control`.\n\n| Var | Default | Description |\n|-----|---------|-------------|\n| `REDIS_URL` | unset | Memorystore/Redis URL. Shares cache + rate limits across instances. |\n| `PREDICT_CACHE_TTL` | `600` | Cache TTL (s) for non-personalised `/predict` responses. `0` disables. |\n| `EVIDENCE_CACHE_TTL` | `900` | Cache TTL (s) for evidence retrieval. |\n| `EXTRACT_CACHE_TTL` | `3600` | Cache TTL (s) for `/extract` URL fetches. |\n| `LOCAL_CACHE_MAX` | `1024` | Max entries in the in-memory fallback cache. |\n| `SEARXNG_URL` / `TAVILY_API_KEY` / `SERPER_API_KEY` / `BRAVE_API_KEY` | unset | Enable web search as an evidence source. A self-hosted **SearXNG** is preferred when set, then Tavily, Serper, Brave. Tavily/Serper have free no-card tiers. When none is set, evidence comes from GDELT, Google News, and RSS. |\n| `NEWSAPI_KEY` | unset | Enables NewsAPI as an evidence source. |\n\n### Live track record\n\n`GET /track-record` serves the public forecast track record. The heavy tick loop\ndoes not run on Cloud Run: `.github/workflows/track-record-tick.yml` runs hourly\non GitHub Actions, updates `data/track_record_store.json` as the source-of-truth\nentity store, writes the public aggregate to `static/track_record_live.json`, and\ncommits both files back to `main`. At runtime, Cloud Run fetches the committed\naggregate from raw GitHub, falling back to the bundled file and then the static\nbacktest in `static/track_record.json`.\n\nThe Action discovers short-to-medium-horizon Polymarket/Kalshi markets in\nseparate close-date bands (`2-7`, `7-14`, `14-30`, `30-60` days by default) and\ncalls `/predict` once per newly snapshotted market/model. If `/predict` is\nprotected, set the GitHub secret `PREDICT_API_KEY`; no server-side\n`/track-record/tick` endpoint is required. `TRACK_RECORD_TOKEN` is optional and\nonly enables the agent-enrolled market bridge.\n\n### Foresea Radar\n\n`GET /radar` serves the public niche-market radar used by the web app's Radar\nview. `.github/workflows/radar-tick.yml` runs hourly on GitHub Actions, fetches\nPolymarket/Kalshi/Reddit candidates, calls `/predict` for fresh Foresea\nprobabilities, writes `static/radar.json`, and commits it back to `main`.\nCloud Run only serves that committed JSON artifact, so Radar avoids OOMs without\navoiding inference.\n\nRadar items include market price, Foresea probability, edge, credibility score,\nevidence links, tracking status, and tags such as `thin liquidity`,\n`resolution-rule risk`, `news catalyst`, `crowded sports market`, `near-term`,\nand `tracked live`. The endpoint is cached with `RADAR_TTL` (default `900`\nseconds).\n\nRaise the Cloud Run throughput ceiling (no idle cost while `min-instances=0`):\n\n```bash\ngcloud run services update analyzing-llm-rationale --region us-central1 \\\n --max-instances 20 --concurrency 40 --memory 1Gi\n```\n\nOnce `max-instances > 1`, provision Memorystore for Redis (billable) and set\n`REDIS_URL` so rate limiting and caching stay correct across instances:\n\n```bash\ngcloud services enable redis.googleapis.com vpcaccess.googleapis.com compute.googleapis.com\ngcloud redis instances create foresea-cache --size=1 --region=us-central1 --tier=basic\ngcloud compute networks vpc-access connectors create foresea-vpc \\\n --region=us-central1 --range=10.8.0.0/28\ngcloud run services update analyzing-llm-rationale --region us-central1 \\\n --vpc-connector foresea-vpc \\\n --update-env-vars REDIS_URL=redis://:6379\n```\n\n## Using the API\n\nThe public Cloud Run API is the easiest integration target. It accepts\nforecasting questions and returns a typed forecast, model rationale, and optional\nevidence articles. It is built for resolvable forecasts, not general Q&A.\n\n### Endpoints\n\n- `GET /health`: service health check.\n- `GET /track-record`: public live track record, falling back to the static backtest.\n- `GET /track-record/digest`: shareable markdown summary of the live track record.\n- `GET /pr-agent`: opt-in agent-to-agent outreach packet for Foresea discovery.\n- `POST /predict`: public prediction endpoint.\n- `GET /markets/polymarket`: fetch a live Polymarket quote (see below).\n- `GET /markets/kalshi`: fetch a live Kalshi quote (see below).\n- `POST /agent/analyze`: orchestrated end-to-end analysis of a live question (see below).\n- `GET /agent/scan`: scan a venue for mispriced markets, ranked by edge (see below).\n- `GET /trading/accounts`: authenticated trading-readiness status, no secrets returned.\n- `POST /trading/preview`: authenticated dry-run order normalization.\n- `POST /trading/orders`: authenticated live order submission with explicit confirmation.\n\n### Agent: automated intelligence layer\n\n`POST /agent/analyze` runs the whole pipeline autonomously: **resolve the market**\n(fetch a live Polymarket/Kalshi price when an identifier is given) → **gather\nevidence + forecast** → **price the edge** → run any **custom skills** →\n**recommend**. It returns one structured report.\n\n```bash\ncurl -X POST https://foresea.ink/agent/analyze \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"platform\": \"polymarket\",\n \"slug\": \"will-the-fed-cut-rates-in-2026\",\n \"skills\": [\n {\"name\": \"Base rate check\", \"instruction\": \"Compare to historical base rates.\"},\n {\"name\": \"Risk\", \"instruction\": \"What would most change this forecast?\"}\n ]\n }'\n```\n\nCustom **skills** are your own analysis steps — each runs as an extra model pass\nover the question, forecast, and evidence, and comes back as a named section in\nthe report. Provide a `question` directly, or a `platform` + market identifier\n(`slug`/`market_id` for Polymarket, `ticker` for Kalshi). Pass `history` (prior\nturns) for multi-turn follow-ups — with history, short follow-ups like \"why?\" or\n\"what about June?\" are answered in context. BYOK fields (`openrouter_api_key`,\n`openrouter_model`, `provider_base_url`) apply here too.\nThe report includes `recommendation` (`buy_yes`/`buy_no`/`hold`/`no_market_price`),\n`edge`, `model_probability`, `market_probability`, `thesis`, `evidence_sources`,\nand `pipeline` (the ordered steps that ran).\n\n### Edge scan — find mispriced markets\n\n`GET /agent/scan` lists live markets on a venue, forecasts each, and returns the\nones whose model-vs-market gap clears `min_edge`, ranked by `|edge|`.\n\n```bash\ncurl \"https://foresea.ink/agent/scan?platform=polymarket&limit=4&min_edge=0.1\"\n```\n\nParams: `platform` (`polymarket` or `kalshi`), `limit` (markets to analyse, max 8),\n`min_edge` (default `0.1`), `evidence_top_k`. Each market runs a full forecast, so\nit's bounded by `limit` and the result is cached briefly. Response: `{platform,\nscanned, opportunities: [{question, market_url, market_probability,\nmodel_probability, edge, recommendation}]}`. In the web app, the desk's\n**\"⚡ Scan Polymarket for mispriced markets\"** button calls this.\n\n### MCP server: let AI agents call Foresea as tools\n\nForesea exposes a public remote MCP server at:\n\n```text\nhttps://foresea.ink/mcp/\n```\n\nIt is advertised for discovery at:\n\n```text\nhttps://foresea.ink/.well-known/mcp/server.json\n```\n\nThe remote MCP server is a thin tool layer over the public API. It exposes:\n\n- `foresea_forecast`: calls `POST /predict`.\n- `foresea_analyze_market`: calls `POST /agent/analyze`.\n- `foresea_scan_markets`: calls `GET /agent/scan`.\n- `foresea_track_record`: calls `GET /track-record`.\n- `foresea_edge_board`: calls `GET /edge-board` — live model-vs-market disagreements ranked, each tagged with the resolved track record of gaps that size (`by_edge` calibration + `lead_lag`).\n- `foresea_pr_agent`: calls `GET /pr-agent` — concise copy and install metadata for agents/catalogs that ask how to describe Foresea.\n- Resources: `foresea://track-record`, `foresea://pr-agent`, and `foresea://openapi.json`.\n\n### PR agent — agent-to-agent distribution\n\n`GET /pr-agent?audience=mcp` returns an opt-in outreach packet that other agents,\nMCP catalogs, and tool directories can quote when introducing Foresea. It includes\nthe one-liner, install command, MCP/OpenAPI links, talking points, and an explicit\nno-spam policy.\n\nFor operator-run cold outreach to explicit agent endpoints, prepare a target list\nand use the local runner. It dry-runs by default and only sends with `--send`:\n\n```bash\npython scripts/pr_agent_outreach.py --targets outreach-targets.json\npython scripts/pr_agent_outreach.py --targets outreach-targets.json --send\n```\n\nTarget file shape:\n\n```json\n{\n \"targets\": [\n {\n \"name\": \"Example Agent Directory\",\n \"endpoint\": \"https://agent-directory.example/inbox\",\n \"audience\": \"catalog\",\n \"headers\": {\"Authorization\": \"Bearer ...\"}\n }\n ]\n}\n```\n\nThe public API returns the outreach packet; it does not expose an unauthenticated\nmessage-sending relay. The scheduled GitHub Action\n`.github/workflows/pr-agent-outreach.yml` runs every 5 minutes against\n`data/pr_outreach_targets.json`, sends with `--send`, and records contacted\ntargets in `data/pr_outreach_state.json` so repeated scheduled runs do not\nre-contact the same agent. For a literal always-running local process, run:\n\n```bash\npython scripts/pr_agent_outreach.py \\\n --targets data/pr_outreach_targets.json \\\n --state data/pr_outreach_state.json \\\n --send --watch --interval-s 300\n```\n\nHeader values can reference GitHub Actions secrets via environment variables, for\nexample `\"Authorization\": \"$PR_AGENT_TARGET_AUTH\"`.\n\nSeeded automated targets:\n\n- AgentNDX (`https://agentndx.ai/api/submit`) — public MCP/A2A/x402 review form.\n- MCP.Directory (`https://mcp.directory/api/submit-server`) — public JSON submit route.\n- mcpub (`https://mcpub.dev/mcp`) — public MCP JSON-RPC `submit` tool.\n\nAdditional listing work that is not suitable for the scheduled HTTP sender lives\nin `data/pr_manual_targets.json`. Current manual/GitHub target: mcp.so issue\n`https://github.com/daodao97/chatmcp/issues/213`.\n\n#### Add Foresea to your agent (10 seconds)\n\nIt's a remote, **anonymous** Streamable-HTTP server — no key, no install. Point any MCP client at the URL:\n\n```bash\n# Claude Code\nclaude mcp add --transport http foresea https://foresea.ink/mcp/\n```\n\n```jsonc\n// Cursor / Cline / Claude Desktop (mcp.json)\n{ \"mcpServers\": { \"foresea\": { \"url\": \"https://foresea.ink/mcp/\" } } }\n```\n\n```python\n# Python — official MCP SDK (3.10+)\nfrom mcp import ClientSession\nfrom mcp.client.streamable_http import streamablehttp_client\n\nasync with streamablehttp_client(\"https://foresea.ink/mcp/\") as (r, w, _):\n async with ClientSession(r, w) as s:\n await s.initialize()\n print(await s.call_tool(\"foresea_forecast\",\n {\"question\": \"Will the Fed cut rates by March 2026?\", \"market_probability\": 0.4}))\n```\n\n```python\n# LangChain (langchain-mcp-adapters) — Foresea tools in any LangGraph agent\nfrom langchain_mcp_adapters.client import MultiServerMCPClient\nclient = MultiServerMCPClient({\"foresea\": {\"url\": \"https://foresea.ink/mcp/\", \"transport\": \"streamable_http\"}})\ntools = await client.get_tools() # foresea_forecast, foresea_analyze_market, ...\n```\n\nA runnable end-to-end demo (scan → forecast → edge) is in\n[`examples/foresea_agent_demo.py`](https://github.com/pareelamre/analyzing-llm-rationale/blob/main/examples/foresea_agent_demo.py).\n\nUse `https://foresea.ink/mcp/` directly in MCP clients that support remote\nStreamable HTTP servers. For clients that still require a local stdio command,\nrun the wrapper locally.\n\nThe repo targets Python 3.10+ because the official MCP Python SDK requires it.\nTo create a repo-local Python 3.11 MCP environment with `uv`:\n\n```bash\nuv venv --python 3.11 .venv-mcp\n\nuv pip install --python .venv-mcp/bin/python --no-deps -e .\nuv pip install --python .venv-mcp/bin/python \"mcp>=1.27.1\" requests pyyaml pip\n\nsource .venv-mcp/bin/activate\nanalyze-llm-rationale mcp-server\n```\n\nThat lightweight install avoids pulling the full inference dependency stack\n(notably Torch/CUDA) when all you need is the MCP wrapper. In a full development\nenvironment, `pip install -e \".[mcp]\"` is also valid.\n\nMCP client config example:\n\n```json\n{\n \"mcpServers\": {\n \"foresea\": {\n \"url\": \"https://foresea.ink/mcp/\"\n }\n }\n}\n```\n\nFor a local HTTP MCP endpoint:\n\n```bash\n.venv-mcp/bin/analyze-llm-rationale mcp-server \\\n --transport streamable-http \\\n --host 127.0.0.1 \\\n --port 8787\n```\n\nConnect MCP clients to `http://127.0.0.1:8787/mcp`. If a private deployment\nrequires auth, set `FORESEA_API_KEY` or pass `--api-key`; the wrapper forwards it\nas `X-API-Key`.\n\nQuick verification:\n\n```bash\n.venv-mcp/bin/python - <<'PY'\nimport importlib.metadata as md\nfrom analyzing_llm_rationale.mcp_server import create_mcp_server\n\nprint(md.version(\"mcp\"))\nprint(create_mcp_server().name)\nPY\n```\n\n### Fetch live market prices\n\nPull the current market-implied probability straight from a venue, then feed it\ninto `/predict` as `market_probability` to compute an edge.\n\n```bash\n# Polymarket — by market slug (or ?id=)\ncurl \"https://foresea.ink/markets/polymarket?slug=will-the-fed-cut-rates-in-2026\"\n\n# Kalshi — by market ticker\ncurl \"https://foresea.ink/markets/kalshi?ticker=KXFED-26SEP-C\"\n```\n\nBoth return a normalised quote:\n\n```json\n{\n \"platform\": \"Polymarket\",\n \"question\": \"Will the Fed cut rates in 2026?\",\n \"market_url\": \"https://polymarket.com/market/...\",\n \"outcome\": \"Yes\",\n \"probability\": 0.54,\n \"outcomes\": [\n {\"label\": \"Yes\", \"probability\": 0.54},\n {\"label\": \"No\", \"probability\": 0.46}\n ]\n}\n```\n\n`probability` is `null` for unpriced/illiquid markets. Quotes are cached briefly\n(`MARKET_CACHE_TTL`, default 30s).\n\n### Trading execution: Polymarket and Kalshi\n\nForesea can submit guarded prediction-market orders, but live execution is\ndisabled by default. Keep this separate from `/agent/analyze`: the agent can\nrecommend `buy_yes`/`buy_no`, but order submission requires a signed-in user,\nserver-side exchange credentials, `FORESEA_ENABLE_TRADING=true`, `execute=true`,\nand the exact confirmation phrase `PLACE REAL ORDER`.\n\nCredentials are read only from the server environment, so use Cloud Run Secret\nManager mounts or environment secrets. Do not collect private keys in the\nbrowser or store exchange secrets in Datastore.\n\n```bash\n# Global guardrails\nexport FORESEA_ENABLE_TRADING=false # must be true for live orders\nexport FORESEA_MAX_ORDER_NOTIONAL=50 # local cap per order, USD\nexport FORESEA_ALLOW_MARKET_ORDERS=false # separate gate for IOC/FOK-style orders\n\n# Kalshi authenticated REST (RSA-PSS signing)\nexport KALSHI_API_KEY_ID=\nexport KALSHI_PRIVATE_KEY_FILE=/secrets/kalshi-private-key.pem\nexport KALSHI_BASE_URL=https://external-api.kalshi.com/trade-api/v2\n\n# Polymarket CLOB SDK\nexport POLYMARKET_PRIVATE_KEY=\nexport POLYMARKET_API_KEY=\nexport POLYMARKET_API_SECRET=\nexport POLYMARKET_API_PASSPHRASE=\nexport POLYMARKET_FUNDER_ADDRESS=\nexport POLYMARKET_SIGNATURE_TYPE=\n```\n\nInstall the optional SDKs in production with:\n\n```bash\npip install -e \".[serve,trading]\"\n```\n\nThe Docker image installs `trading`, so Cloud Run only needs secrets/env vars.\n\nCheck configured venues:\n\n```bash\ncurl https://foresea.ink/trading/accounts \\\n -H \"Authorization: Bearer $FORESEA_SESSION\"\n```\n\nPreview a Kalshi order without execution:\n\n```bash\ncurl -X POST https://foresea.ink/trading/preview \\\n -H \"Authorization: Bearer $FORESEA_SESSION\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"platform\": \"kalshi\",\n \"ticker\": \"KXFED-26SEP-C\",\n \"action\": \"buy\",\n \"outcome\": \"yes\",\n \"price\": 0.42,\n \"quantity\": 1\n }'\n```\n\nSubmit a live order only after reviewing the preview:\n\n```bash\ncurl -X POST https://foresea.ink/trading/orders \\\n -H \"Authorization: Bearer $FORESEA_SESSION\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"platform\": \"kalshi\",\n \"ticker\": \"KXFED-26SEP-C\",\n \"action\": \"buy\",\n \"outcome\": \"yes\",\n \"price\": 0.42,\n \"quantity\": 1,\n \"execute\": true,\n \"confirmation\": \"PLACE REAL ORDER\"\n }'\n```\n\nFor Polymarket, pass the CLOB `token_id` for the exact outcome, or pass\n`slug`/`market_id` plus `outcome` and Foresea will resolve the token id from the\npublic market record. Limit orders use `quantity` as shares. Market-buy orders\nuse `max_cost` as USD spend when supplied and remain blocked unless\n`FORESEA_ALLOW_MARKET_ORDERS=true`.\n\n### Request fields\n\nRequired:\n\n- `question`: forecasting question, such as `\"Will X happen by date Y?\"`,\n `\"Who will win X?\"`, `\"What will X be?\"`, or `\"When will X happen?\"`.\n\nOptional:\n\n- `question_type`: `binary`, `multiple_choice`, `numeric`, or `date`. If omitted,\n the model attempts to infer the type.\n- `options`: answer choices for `multiple_choice` questions.\n- `description`: extra context for the question.\n- `resolution_criteria`: how the question should resolve or be measured.\n- `categories`: list of topic labels.\n- `news_articles`: caller-supplied evidence articles. If provided, automatic\n evidence retrieval is skipped.\n- `attach_evidence`: defaults to `true`. When true and `news_articles` is empty,\n the API fetches current evidence from GDELT, Google News RSS, and Stooq.\n- `evidence_top_k`: number of evidence articles to attach, capped by the server.\n- `market_platform`: prediction market venue such as `Polymarket`, `Kalshi`,\n `Manifold`, or `Metaculus`.\n- `market_url`: URL for the market being analyzed.\n- `market_outcome`: outcome whose market price is supplied. Defaults to `Yes`\n for binary markets.\n- `market_probability`: current market-implied probability for\n `market_outcome`. Use `0.42` or `42`; the API normalizes percentages.\n- `variant`: prompt variant. Defaults to `variant0_neutral_baseline`.\n- `created_time`, `publish_time`, `resolve_time`, `days_open`: optional\n forecasting metadata.\n- `openrouter_api_key` + `openrouter_model`: run the forecast on your own model\n instead of the server default (see \"Bring your own model\" below).\n- `provider_base_url`: optional OpenAI-compatible `/chat/completions` endpoint to\n use with your key/model instead of OpenRouter. Must be public HTTPS.\n\n### Bring your own model\n\nBy default `/predict` runs on the server's hosted model. To use your own:\n\n- **Via OpenRouter** — pass `openrouter_api_key` and `openrouter_model` (e.g.\n `openai/gpt-4o`, `anthropic/claude-sonnet-4-5`). The request is proxied through\n OpenRouter.\n- **Via any OpenAI-compatible endpoint** — also pass `provider_base_url` (e.g.\n `https://api.openai.com/v1/chat/completions`) with the matching `openrouter_model`\n (here just the provider's model ID, e.g. `gpt-4o`) and your key.\n\nFor safety, `provider_base_url` must be public HTTPS; loopback, private,\nlink-local, and cloud-metadata hosts are rejected. In the web app, the sidebar's\n**\"Use your own model\"** panel exposes the provider, endpoint, key, and model.\n\n```bash\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"Will X happen by 2027?\",\n \"question_type\": \"binary\",\n \"openrouter_api_key\": \"YOUR_KEY\",\n \"openrouter_model\": \"gpt-4o\",\n \"provider_base_url\": \"https://api.openai.com/v1/chat/completions\"\n }'\n```\n\n### Binary request\n\n```bash\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"Will the Federal Reserve cut interest rates at least once before September 30, 2026?\",\n \"question_type\": \"binary\",\n \"market_platform\": \"Polymarket\",\n \"market_probability\": 42\n }'\n```\n\n### Multiple-choice request\n\n```bash\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"Who will win the 2026 Formula 1 drivers championship?\",\n \"question_type\": \"multiple_choice\",\n \"options\": [\"Max Verstappen\", \"Lando Norris\", \"Charles Leclerc\", \"Lewis Hamilton\", \"Other\"],\n \"attach_evidence\": false\n }'\n```\n\n### Numeric request\n\n```bash\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"What will US CPI inflation be in December 2026?\",\n \"question_type\": \"numeric\",\n \"resolution_criteria\": \"Use the year-over-year CPI-U inflation rate for December 2026.\"\n }'\n```\n\n### Request with caller-provided evidence\n\n```bash\ncurl -X POST https://foresea.ink/predict \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"question\": \"Will Company X report positive net income in Q4 2026?\",\n \"description\": \"Resolve using the company earnings release.\",\n \"resolution_criteria\": \"Yes if reported GAAP net income is positive.\",\n \"attach_evidence\": false,\n \"news_articles\": [\n {\n \"title\": \"Company X raises full-year guidance\",\n \"source\": \"Example Business News\",\n \"url\": \"https://example.com/company-x-guidance\",\n \"publish_date\": \"2026-05-29\",\n \"summary\": \"Company X raised revenue guidance and reported margin expansion.\"\n }\n ]\n }'\n```\n\n### Python client example\n\n```python\nimport requests\n\npayload = {\n \"question\": \"Will the Federal Reserve cut interest rates at least once before September 30, 2026?\",\n \"question_type\": \"binary\",\n \"attach_evidence\": True,\n \"evidence_top_k\": 3,\n \"market_platform\": \"Polymarket\",\n \"market_probability\": 42,\n}\n\nresponse = requests.post(\n \"https://foresea.ink/predict\",\n json=payload,\n timeout=180,\n)\nresponse.raise_for_status()\nprediction = response.json()\n\nprint(prediction[\"predicted_answer\"], prediction[\"confidence\"])\nprint(prediction[\"model_rationale\"])\nif prediction.get(\"market_analysis\"):\n print(prediction[\"market_analysis\"][\"summary\"])\nfor source in prediction[\"evidence_sources\"]:\n print(source[\"source\"], source[\"url\"])\n```\n\n### Response fields\n\n- `question_type`: detected or requested type: `binary`, `multiple_choice`,\n `numeric`, or `date`.\n- `predicted_answer`: `\"Yes\"`, `\"No\"`, the top multiple-choice option, or the\n median numeric/date estimate.\n- `confidence`: model confidence as a number from 0 to 1 for binary and\n multiple-choice forecasts; `null` for numeric/date forecasts.\n- `options`: per-option probabilities for multiple-choice forecasts.\n- `range_forecast`: `p10`, `p50`, `p90`, and optional `unit` for numeric/date\n forecasts.\n- `rationale`: model-generated explanation.\n- `model_rationale`: alias for the model-generated explanation, intended for API\n clients.\n- `evidence_sources`: compact source list with article title, URL, publication\n date, and relevance score.\n- `evidence_articles`: full evidence records attached to the prompt.\n- `evidence_error`: retrieval error message, or `null` when evidence retrieval\n succeeds.\n- `market_analysis`: optional comparison against a supplied market price:\n `market_probability`, `model_probability`, `edge`, `stance`, and a short\n summary. `edge` is `model_probability - market_probability`.\n\n## Repository Contents\n\n- `src/analyzing_llm_rationale/`: packaged inference, provider, validation, and CLI logic.\n- `configs/`: model and rationale-variant definitions.\n- `prompts/`: system prompt and the nine rationale-variant prompts.\n- `scripts/`: evaluation, recovery, SHAP, plotting, and utility scripts.\n- `slurm/`: HPC launchers for the variant/temperature sweeps.\n- `results/`: model outputs and run metadata.\n- `analysis/`: aggregate metric tables and rationale-analysis outputs.\n- `paper/`: paper figures, Draw.io sources, PDFs, and qualitative case studies.\n- `tests/`: unit tests for the package and metric parsing.\n\nSee `ARTIFACT_MANIFEST.md` for the submission checklist and file-level notes.\n\n## Install\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npython -m pip install -e \".[dev,analysis]\"\n```\n\nUse `.[dev]` for the core runner and tests only. Use `.[analysis]` when\nregenerating plots or SHAP analyses.\n\n## Quick Validation\n\n```bash\nPYTHONPATH=src python -m analyzing_llm_rationale validate-dataset\npython -m unittest discover -s tests\nruff check src tests scripts/*.py\n```\n\n`PYTHONPATH=src` is useful when the repository has not been installed yet or an\nolder user-local install shadows the working tree.\n\n## Primary Entry Point\n\nRun the variant 3 pipeline with the packaged CLI:\n\n```bash\nanalyze-llm-rationale run-batch --variant variant3_reasoning_type\n```\n\nFor a remote OpenAI-compatible provider:\n\n```bash\nexport PROVIDER_API_KEY=your_token\nanalyze-llm-rationale run-batch --variant variant3_reasoning_type --model llama-3.3-70b-instruct\n```\n\nIf you do not want to install the package into the environment, invoke it directly:\n\n```bash\nPYTHONPATH=src python -m analyzing_llm_rationale run-batch --variant variant3_reasoning_type\n```\n\nUseful options:\n\n- `--variant variant6_step_by_step_reasoning`: choose the prompt/output contract.\n- `--model qwen2.5-7b-instruct`: choose a configured model definition.\n- `--temperature 0.7`: control generation temperature and output directory.\n- `--max-records 10`: process only a bounded number of records.\n- `--reprocess-nulls`: rerun existing rows with `predicted_answer = null`.\n- `--drop-article-text`: remove raw article text from prompts before inference.\n- `--device auto`: select `cuda` when available, otherwise `cpu`.\n- `verify-results --variant ...`: verify completeness, duplicates, malformed rows, and missing IDs.\n- `validate-dataset`: validate the dataset schema before a run.\n\n## Foresea Autoresearch\n\nForesea has a Karpathy-style autoresearch harness for prompt experiments: edit\none candidate prompt, run a fixed benchmark slice, score one metric, and append\nan auditable experiment log. The research surface is\n`autoresearch/candidate_prompt.txt`; agent instructions live in\n`autoresearch/program.md`. The default `--model gpt-oss-120b` uses the\nSCADS-hosted OpenAI-compatible endpoint from `configs/models.yaml`\n(`SCADS_AI_API_KEY` or `SCADS_AI_API_KEY.txt`).\n\nRun one candidate experiment:\n\n```bash\nPYTHONPATH=src python -m analyzing_llm_rationale autoresearch \\\n --model gpt-oss-120b \\\n --candidate-prompt-path autoresearch/candidate_prompt.txt \\\n --max-records 50 \\\n --metric brier_score\n```\n\nCompare against a baseline and promote only if the candidate improves:\n\n```bash\nPYTHONPATH=src python -m analyzing_llm_rationale autoresearch \\\n --model gpt-oss-120b \\\n --candidate-prompt-path autoresearch/candidate_prompt.txt \\\n --baseline-results-path results/GPT-OSS-120B/temperature_00/results_variant0_neutral_baseline.json \\\n --promote-to prompts/variant0_neutral_baseline.txt \\\n --max-records 50 \\\n --metric brier_score \\\n --min-delta 0.001\n```\n\nEach run writes `analysis/autoresearch/runs//score.json` and appends a\nmachine-readable row to `analysis/autoresearch/experiments.jsonl`.\n\n## Reproducing Core Outputs\n\nValidate an existing result file:\n\n```bash\nPYTHONPATH=src python -m analyzing_llm_rationale verify-results \\\n --model qwen2.5-7b-instruct \\\n --variant variant3_reasoning_type \\\n --temperature 0.0 \\\n --temperature-tag temperature_000\n```\n\nRegenerate aggregate metrics from `results/`:\n\n```bash\npython scripts/evaluate_metrics.py\n```\n\nRun the DuckDB SQL analytics suite over the real Metaculus-style dataset and\nsaved model outputs:\n\n```bash\npython scripts/sql_analytics.py \\\n --db analysis/forecasting_analytics.duckdb \\\n --ingest --replace \\\n --output-dir analysis/sql_analytics\n```\n\nThis writes a markdown report plus one CSV per query for 10 medium-level SQL\nproblems: model accuracy, best variants, calibration bins, Brier score,\nconsensus/disagreement cases, prompt lift over baseline, temperature sensitivity,\noverconfident errors, and category difficulty.\n\nRun the LangChain-powered news retrieval wrapper:\n\n```bash\nPYTHONPATH=src analyze-llm-rationale fetch-and-rank \\\n --question \"Will X happen by date Y?\" \\\n --source gdelt \\\n --source google-news \\\n --source stooq \\\n --top-k 5\n```\n\nThe news pipeline uses LangChain for a query-planning step, article\nsummarization, and embedding-based relevance ranking before inference. Evidence\nsources are configurable with `--source` for the CLI and `--evidence-source`\nwhen serving the API.\n\nRun or schedule the Prefect DAG for RSS/news fetch, inference, and DuckDB\nlogging:\n\n```bash\n# One question\npython flows/forecasting_flow.py --question-id 124 --top-k 5\n\n# Small batch from the dataset\npython flows/forecasting_flow.py --limit 3 --top-k 5\n\n# Daily scheduled deployment at 06:00 UTC\nprefect server start\npython flows/forecasting_flow.py --deploy --limit 3 --cron \"0 6 * * *\"\n```\n\nRegenerate paper figures after metrics are present:\n\n```bash\npython scripts/plot_model_variant_metric_heatmap.py\npython scripts/plot_variant_delta_from_v0.py\npython scripts/plot_temperature_frontier.py\npython scripts/plot_frs_ablation_slopegraph.py\npython scripts/plot_uncertainty_language_calibration_disconnect.py\npython scripts/plot_shap_importance_attribute_gaps.py\n```\n\n## Scripts\n\nCommon runner and verification commands:\n\n- `python scripts/run_variant.py --variant variant5_key_conditions`\n- `python scripts/run_variant.py --variant variant3_reasoning_type --temperature 0.7 --temperature-tag temperature_07`\n- `python scripts/run_variant.py --variant variant4_credibility --model llama-3.3-70b-instruct`\n- `python scripts/verify_results.py --variant variant3_reasoning_type`\n- `python download_qwen_model.py`\n- `python test_local_inference.py`\n\nRepo layout:\n\n- `scripts/`: modular runner entrypoint\n- `slurm/`: batch launchers\n\nAuditability:\n\n- Each run writes `run_metadata_.json` next to the results file.\n- Metadata includes provider, model key, resolved model identifier, temperature, output fields, and prompt SHA-256 hashes.\n- Existing malformed results JSON now fails fast instead of being silently ignored.\n\n## Quality checks\n\n```bash\npython -m unittest discover -s tests\nruff check src tests scripts/*.py\n```\n\n## Data, Models, and Secrets\n\nThe included dataset is `forecasting_qa_news_metaculus_2025-02-01_to_today.metaculus_frs_format.json`.\nModel access is configured in `configs/models.yaml`. Open-weight Qwen models run\nlocally through Hugging Face; hosted models use OpenAI-compatible endpoints and\nrequire API keys through environment variables or local key files.\n\nNever commit key files or tokens. Large local caches (`.cache/`, `envs/`, `.venv/`)\nare intentionally ignored and excluded from source archives.\n\n## Citation\n\nIf this repository supports a publication, cite the artifact with the metadata in\n`CITATION.cff` and cite the upstream datasets/models according to their licenses.\n","changelogMarkdown","videoKey","requiredSecretsJson","publishedAt","priceCents","trialDays","categoryDisplayName","Integrations","upstream",{"_94":95,"_96":97,"_98":99,"_100":101,"_102":-5},"requiredSecrets",[],"tags",[],"assets",[],"stats",{"_92":57,"_93":57},"ratingAvg","reviewCount","topics",[],"license","MIT","language","Python","registryName","ink.foresea/forecasting","homepage","ENV",{"_105":106,"_107":108,"_109":110},"ENVIRONMENT","production","TURNSTILE_SITE_KEY","0x4AAAAAAB8-rEI2gNDf_QhU","SENTRY_DSN","https://45fd1229f78089d5c3ee78be128030e1@o4510583261757440.ingest.de.sentry.io/4510584253448272"]