# Tidings docs > A private finance journal built from the transaction emails you already receive. Self-hosted with Docker or AWS Lambda; data stays on your machine. --- # Tidings Source: https://docs.gettidings.com/index.md > A private finance journal from the transaction emails you already receive. ## Pick your path ### Just looking? The [live demo](https://gettidings.com/demo) is eleven months of a fictional household's spending — no install, no signup. Prefer a guided version? [A month in Tidings →](/tour/) ### Ready to run it? Three commands and a browser tab. First run boots in demo mode with seeded data, so you can look around before wiring up your own. [Quickstart →](/quickstart/) ### Coming from Firefly III or Actual Budget? Tidings is deliberately narrower — a journal that maintains itself, not an accounting system. What you give up, what you gain, and when to pick which. [Read the comparison →](/switching/) ### Building on the API? Every route is a versioned `/api/v1/` endpoint with bearer-token auth, and the docs themselves are served in machine-readable form. [Tidings for agents →](/for-agents/) ## Or let an agent introduce you Running Claude Code, Cursor, Codex, or another capable agent? Paste this: ```text Introduce me to Tidings. Read https://docs.gettidings.com/agent-guide.md, then use the live demo API to show me what a month of spending looks like. ``` The guide briefs the agent on how Tidings works, then points it at the read-only demo API. The agent pulls a real month of sample data and narrates it — the journal, the budget pace, the merchants — before you install anything. When you are ready, there is [a prompt for installing too](/for-agents/#install-by-prompt). ## How it works Tidings reads the transaction alerts your bank already emails you, parses each one, and shows you where your money goes. It runs on your machine and keeps your data there. The only thing that leaves is what you choose to send an AI provider — for category labels and the optional written summaries — and that stays off until you add a key. *Figure: a bank alert email travels bank → forwarder inbox → parser → categorization → local store, and every page — Journal, Transactions, Summary, Budgets, Insights, Merchants — reads from that one store.* Built for a single household — one or two people sharing a forwarding inbox, a single Gmail account that collects everyone's bank alerts. No Plaid, no broker linking, no shared multi-user tracking. Parsers ship today for five Canadian institutions: RBC, CIBC, MBNA, Simplii, and PC Financial. If yours isn't here, [open an issue](https://github.com/tvhahn/tidings/issues) with a handful of sample alert emails — or write the parser yourself: the [add-a-parser tutorial](/add-a-parser/) walks from zero to a working example, the architecture is country-neutral, and contributions for any institution are welcome. ## Core guides ### Quickstart Three commands and a browser tab. The default is demo mode, with seeded sample data — no real accounts touched. [Run the demo →](/quickstart/) ### Using Tidings What each page does and the rules underneath it — the journal, budgets, categorization, insights, and the rest. [Read the manual →](/using/how-it-works/) ### Self-hosting Docker Compose is the recommended path. AWS Lambda is the advanced serverless variant. Same parsers, same notifications, different storage. [Self-host options →](/self-hosting/docker/) ### Architecture IMAP poller → bank parser → categorizer → SQLite or DynamoDB → FastAPI → React. The shape, the boundaries, and the design decisions. [How it fits together →](/architecture/) ### API reference Versioned `/api/v1/` routes with a unified error shape. Generated from the live OpenAPI spec, drift-checked in CI. [Browse the API →](/api/) ### FAQ Does Tidings have your banking password? (No.) What leaves your machine, what it costs, and how to take your data out. [Short answers →](/faq/) --- # Quickstart Source: https://docs.gettidings.com/quickstart.md > Three commands and a browser tab. The default is demo mode with seeded sample data. Try Tidings in about a minute, with no real bank accounts. The demo runs from a seeded SQLite database — no IMAP credentials, no AWS, no external services. Click around, then wire up your own data when you're ready. > **Or hand it to an agent** > > Paste this into Claude Code, Cursor, Codex, or another capable agent: > > ```text > Help me install Tidings. Read https://docs.gettidings.com/install.md first, then walk me through it. > ``` ## Run the demo 1. Clone the repository and start the stack. ```bash git clone https://github.com/tvhahn/tidings.git cd tidings docker compose up -d ``` No published image for your platform yet? The same command builds from source instead — first run takes a few minutes. 2. Open the dashboard. ## Set your timezone Tidings defaults to `America/Los_Angeles`. Change it under **Settings → Timezone** (or the `timezone` key in `data/config.json` before first launch), and it applies to the whole instance. Details and the changing-it-later caveat are in [settings](/using/settings/). ## Wire up your own data When you move to real data, three pages cover the wiring, in order: 1. [Self-hosting with Docker](/self-hosting/docker/) — copy `.env.example` to `.env`, set `demo_mode: false` in `data/config.json`, and run the stack against your own SQLite database. 2. [Email setup](/self-hosting/email/) — create a dedicated Gmail account, generate an App Password, and point your bank alerts at it. 3. [Notifications](/notifications/) — pick a push or SMS provider. The recommended default is ntfy: free, no account, works on iOS and Android. > **AI categorization is off until you add a key** > > Tidings can use OpenAI to label each transaction's category. Adding an `OPENAI_API_KEY` turns on two switches under Settings → Intelligence — categorization and email rescue — and either can be turned back off. Categorization sends only the amount and the merchant name, never the account number, card number, or balance; email rescue sends the subject and body of an email no parser could read, so OpenAI can read the transaction out. Run without a key and nothing leaves your machine. ## Common gotchas The three failures people hit first — port 8000 already in use, IMAP authentication, and permissions on `data/` — are covered in [Docker setup's troubleshooting](/self-hosting/docker/#troubleshooting), each with its one-line fix. For the full JSON API while the stack is running, see the [API reference](/api/). --- # A month in Tidings Source: https://docs.gettidings.com/tour.md > A five-minute, screenshot-led walk through one month of spending — journal, summary, budgets, insights, merchants. Every screen below comes from the [live demo](https://gettidings.com/demo) — eleven months of a fictional household's spending. This is the shape of a month once the pipeline is wired: alerts arrive by email, parsers file them, and these pages assemble themselves. ## The journal A month, read one day at a time, newest first. Each day carries its own total, and the month accumulates as you scroll. Only spending you can act on appears here — refunds, transfers, and anything you ignore stay in the full ledger. [The journal →](/using/journal/) ## The summary One page scoped to the month in the picker. The Flow view is a Sankey diagram of where the month went — income on one side, category groups on the other; toggle to Trend for the month against recent history. The headline sentence is assembled from the page's own figures, not written by a model. [Summary →](/using/summary/) ## Budgets Budgets are annual. You set a target per category, and the dashboard reads year-to-date spending against the elapsed fraction of the year — two months in, roughly a sixth of the year has passed. A separate ceiling caps total spending, and pace tells you whether the year is running ahead or behind. [Budgets →](/using/budgets/) ## Insights The monthly briefing workspace. The cards on top — momentum against last month, and the month's quiet anomalies — are computed directly from your transactions; no model is involved. The narrative below is AI-written and opt-in, follows the same fixed shape every month, and after generation every figure in it is traced back to the underlying numbers. [Insights →](/using/insights/) ## Merchants Who you pay, across the trailing six months. Recurring merchants are detected and typed — fixed, variable, lumpy — and the fixed ones sum to a recurring burn rate: the spending locked in before the month begins. Price changes, new merchants, and churned ones are surfaced alongside. [Merchants →](/using/merchants/) ## Where next - [Try the demo](https://gettidings.com/demo) — the same data, live, no install. - [Quickstart](/quickstart/) — three commands and a browser tab. - Or let an agent introduce you: ```text Introduce me to Tidings. Read https://docs.gettidings.com/agent-guide.md, then use the live demo API to show me what a month of spending looks like. ``` --- # Coming from Firefly III or Actual Budget? Source: https://docs.gettidings.com/switching.md > What Tidings deliberately leaves to those tools, what it does that they don't, and when to pick which. Firefly III and Actual Budget are the established self-hosted finance tools, and both are good at what they set out to do. Tidings sets out to do something narrower. This page is the honest comparison. ## What those tools are **Firefly III** is a full double-entry finance manager — accounts, balances, rules, reports. Data comes in through its companion Data Importer: CSV or CAMT files, or third-party bank-data providers where they cover your bank. **Actual Budget** is a local-first envelope-budgeting app in the YNAB tradition — you give every dollar a job and reconcile against accounts. Bank sync is user-initiated through providers such as SimpleFIN Bridge (North America, about $15 a year) or Enable Banking (Europe), alongside QIF, OFX, QFX, and CSV imports. Both are organized around accounts and balances, and getting data in means bank sync where a provider covers your bank, file exports where it doesn't, and manual entry in between. ## What Tidings does instead Tidings builds a spending journal from the transaction alert emails your bank already sends. You forward the alerts to an inbox you control; a parser reads each one within a minute of arrival and files it once. No bank credentials, no sync provider, no import ritual — and no accounting system underneath. The trade is deliberate: give up account-balance bookkeeping, get a record of spending that maintains itself. ## What you give up - **Double-entry accounting.** There are no accounts and no balances. Tidings records spending and income as a journal, not a ledger that reconciles to the penny. PDF statement upload exists to catch transactions that never arrived by email, not to manage accounts. - **Envelope budgeting.** Budgets in Tidings are annual targets read against the pace of the year, plus a top-level spending ceiling. There are no envelopes, no monthly rollover, no zero-based allocation. - **Bank coverage by sync.** Tidings needs a bank that emails transaction alerts. Parsers for five Canadian banks ship today; alerts from other banks can be rescued with AI extraction, and any bank that sends alert emails can be added by [writing a parser](/add-a-parser/). ## What you gain - **No manual entry, and no credentials.** Nothing to type in, no aggregator holding your bank login, no sync subscription to maintain. - **A journal you actually read.** A month arrives one day at a time, with budgets paced to the year, a monthly briefing, and merchant signals — closer to a statement worth reading than a spreadsheet to maintain. - **An API built for agents.** Every route is a versioned `/api/v1/` endpoint with bearer-token auth, and the docs ship in machine-readable form. See [Tidings for agents](/for-agents/). ## Run both if you like Tidings does not compete for your data — it reads emails the other tools never see. Keeping an accounting tool for balances and net worth alongside Tidings for the daily journal is a legitimate setup, not a compromise. Next: [A month in Tidings](/tour/), or the [Quickstart](/quickstart/). --- # Docker setup Source: https://docs.gettidings.com/self-hosting/docker.md > The everyday self-host path: SQLite on a shared Docker volume, IMAP poller as a sidecar, zero cloud dependencies. **For self-hosters with Docker installed who want zero cloud dependencies.** By the end you'll have the dashboard running with IMAP email polling, everything stored locally in SQLite. Docker Compose is the recommended self-host path. One container runs FastAPI plus the prebuilt React dashboard; a sidecar runs the IMAP poller. SQLite lives on a shared named volume. Start here in demo mode, then [configure real Gmail IMAP](/self-hosting/email/) and [set up notifications](/notifications/). ## What you get - FastAPI backend at `http://localhost:8000` - React dashboard served from the same container at `/` - IMAP poller sidecar that polls Gmail every 60 seconds once credentials are set — and idles quietly until then, so a bare `docker compose up` never crash-loops - SQLite at `data/finance.db` (real mode) or `data/demo.db` (demo mode, the default on first run), on a shared named volume ## Run it If you already followed [Quickstart](/quickstart/), the stack is up — skip to "Where data lives" below. Otherwise: 1. Clone the repository. ```bash git clone https://github.com/tvhahn/tidings.git cd tidings ``` 2. Start the stack. ```bash docker compose up -d ``` No published image for your platform yet? The same command builds from source instead — first run takes a few minutes. 3. Open the dashboard at . Out of the box this runs in **demo mode** — a seeded SQLite database (`data/demo.db`) with sample transactions, no real accounts touched. To wire up your own data: copy `.env.example` to `.env` and add your Gmail IMAP credentials ([email setup](/self-hosting/email/) walks through it), set `demo_mode: false` in `data/config.json`, and restart. Your transactions land in `data/finance.db`; the seeded demo data stays in its own database and never mixes with yours. ## Where data lives | Path | Contents | | ------------------ | ------------------------------------------------------------------------ | | `data/config.json` | Timezone, demo mode, storage backend, AI on/off, dashboard password hash | | `data/finance.db` | Real transactions, categories, overrides, budgets | | `data/demo.db` | Seeded sample data shown in demo mode | Credentials — the Gmail App Password, OpenAI key, and notification provider tokens — live in `.env` as environment variables, not in `data/`. Every `data/config.json` key is documented in the [configuration reference](/configuration/). The `data/` directory is gitignored and lives in a Docker named volume (`tidings_finance_data`, prefixed by your Compose project name). To inspect or back it up from outside the container: ```bash docker run --rm \ -v tidings_finance_data:/data \ -v "$(pwd)/backup":/out \ alpine cp -r /data /out ``` Substitute your actual volume name from `docker volume ls` if your project directory isn't `tidings`. To start over, `docker compose down -v` removes the volume along with the containers. ## Updating ```bash git pull docker compose pull docker compose up -d ``` Tidings is pre-1.0; APIs and schemas may shift between minor versions. Read [`CHANGELOG.md`](https://github.com/tvhahn/tidings/blob/main/CHANGELOG.md) before upgrading. ## Health check The sync dot beside the month picker carries the same state `curl http://localhost:8000/api/v1/health` returns: - **`ok`** — the poller checked in within the last 5 minutes (or IMAP isn't configured), and no parser looks stuck. - **`degraded`** — the last poll was 5–30 minutes ago, a bank email failed to parse in the last 7 days, or AI categorization is failing. A failed parse is the early warning for template drift. Check `docker compose logs imap-poller`. - **`stale`** — no poll in over 30 minutes, or the most recent transaction is more than 14 days old. The dot turns red and pulses. The usual cause is a bank changing its email format so the parser silently errors; look in the logs for `Failed to parse email`. ## Troubleshooting - **Port 8000 already in use** — copy `docker-compose.override.yml.example` to `docker-compose.override.yml`, uncomment the "Re-home the API port" block with a free host port (for example `"8001:8000"`), and open `http://localhost:8001`. - **`exec format error` on Raspberry Pi / Apple Silicon** — images are multi-arch (amd64 + arm64), so this is usually a stale cache. `docker compose pull`, or build locally. - **IMAP poller can't connect** — Gmail needs an App Password and 2-Step Verification, not your account password. [Email setup](/self-hosting/email/) walks through it. - **Permission errors on `data/`** — the named volume is owned by the container user. After switching between a bind mount and the named volume, reset with `docker compose down -v` (this deletes the volume's data). - **Transactions stopped appearing** — the usual cause is a bank changing its email template. Look in `docker compose logs imap-poller` for `Failed to parse email`, then open a [parser-broken issue](https://github.com/tvhahn/tidings/issues/new?template=parser_broken.md) with a redacted email body. ## Looking for AWS instead? The same parsers also run as an AWS Lambda triggered from S3. Storage moves to DynamoDB, notifications move to SNS. See [AWS deployment](/self-hosting/aws/). > **Architecture overview** > > For the full data flow — IMAP poll, parser pipeline, dual-storage selection, notifier dispatch — see [Architecture](/architecture/). --- # Email setup Source: https://docs.gettidings.com/self-hosting/email.md > Wire a dedicated Gmail inbox to the IMAP poller. This guide walks you through wiring up the self-hosted finance dashboard to a Gmail inbox so the IMAP poller daemon can ingest bank alert emails and turn them into transactions. **For self-hosters with the project cloned and Docker installed.** No prior Gmail IMAP or email-forwarding experience required. By the end, your IMAP poller will fetch bank emails from a dedicated Gmail inbox, parse them, and display new transactions on the dashboard. Follow the steps in order — later steps assume earlier ones are complete. **How it works at a glance.** 1. Your bank sends transaction alerts to your personal Gmail. 2. A Gmail filter on your personal account forwards those alerts to a dedicated finance-only Gmail account. 3. The `imap-poller` Docker service logs into that dedicated account over IMAP, fetches new bank emails, and runs them through the same parser pipeline as the AWS Lambda stack. 4. Parsed transactions land in `data/finance.db` and show up on the dashboard. The supported banks for direct purchase / transaction alerts are RBC, CIBC, MBNA, Simplii, and PC Financial. Interac e-Transfer notifications are handled separately — any Canadian bank that uses the standard `payments.interac.ca` sender will be parsed. --- ## 0. Set your timezone (non-Pacific users) The app defaults to `America/Los_Angeles`. Before you start ingesting transactions, change it from the dashboard: **Settings → Timezone**. The selector lists every IANA zone your browser knows about, and the **Detect from browser** button picks your local one in a click. The setting is instance-wide and controls how transactions are bucketed into days and months, and how "latest transaction age" is computed. If you'd rather configure it before launching the dashboard, edit the `timezone` key in `data/config.json` directly: ```json { "timezone": "Europe/Berlin" } ``` A note on switching mid-stream: the sort key is local-time and is written once at ingest, so changing `timezone` does not rewrite history. New transactions bucket in the new zone; older rows keep their original prefix. For most users this is invisible. For users who move continents, midnight-adjacent transactions ingested before the switch may group on the wrong calendar day in the Journal until they age out. --- ## 1. Create a dedicated Gmail account Sign up for a new Gmail account that will be used **only** by the dashboard — for example, `yourname.finance@gmail.com`. Why a dedicated account, instead of pointing the poller at your main inbox? - **Isolation.** The poller marks messages as `\Seen` and stores a last-seen UID cursor. Running it against your personal inbox would mutate state you care about and race with the way you read mail. - **Blast radius.** If the App Password is ever leaked, you can burn the whole account and forwarding rule without touching your primary email. - **Easier revocation.** To disconnect the dashboard later, delete the App Password — or the whole Gmail account — with no risk of clobbering unrelated settings on your main account. Pick a strong, unique password and store it in a password manager. You will also create an App Password later (step 3), which is what the poller actually uses. Signed in as the dedicated account, move on to step 2. ## 2. Enable 2-Step Verification on the dedicated account Google will not let you create App Passwords until 2-Step Verification (2SV) is enabled on the account. 1. While signed in to the **dedicated** account, go to [myaccount.google.com/security](https://myaccount.google.com/security). 2. Under **How you sign in to Google**, click **2-Step Verification**. 3. Follow Google's walkthrough — detailed instructions live at [support.google.com/accounts/answer/185839](https://support.google.com/accounts/answer/185839) (the canonical "Turn on 2-Step Verification" article). Google's walkthrough now recommends Google prompts on a signed-in phone or a passkey as the primary method; SMS/voice codes and authenticator apps are still supported fallbacks. 4. Finish the setup until the page reports "2-Step Verification is **On**". Double-check you are configuring 2SV on the *dedicated* finance account, not your personal Gmail. ## 3. Generate a Gmail app password The **App passwords** option only appears *after* 2-Step Verification has been enabled. If you don't see it, go back and finish step 2. 1. Visit [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords) (or navigate to **Google Account → Security → 2-Step Verification → App passwords**). 2. Re-enter your password if prompted. 3. In the **App name** field, enter something recognizable like `finance-dashboard-imap`. This is a label for your own reference; Google does not use it for anything other than display. 4. Click **Create**. Google will display a 16-character password (the UI shows it as four groups of four with spaces — the spaces are cosmetic and may be omitted when pasting). 5. **Copy it now and store it in your password manager.** Google will not show it again. Notes: - The poller strips spaces from the App Password automatically, so pasting it with or without spaces both work. - This App Password is what goes into `IMAP_PASSWORD` in step 6 — *not* your regular Google account password. ## 4. Confirm IMAP is enabled For **personal Gmail accounts**, IMAP is always on as of January 2025 — Google removed the Enable IMAP / Disable IMAP toggle, and the option no longer appears under **Forwarding and POP/IMAP**. You can skip this step if you signed up at `gmail.com`. The poller will connect to `imap.gmail.com:993` over SSL without any further configuration. For **Google Workspace accounts** (custom domains administered through admin.google.com), IMAP access is still controlled by an admin. If your dedicated account is on a Workspace domain, an admin needs to enable POP and IMAP under **Apps → Google Workspace → Gmail → End User Access**. Reference (Workspace admin docs): [support.google.com/a/answer/105694](https://support.google.com/a/answer/105694). For end-user Gmail forwarding setup (separate from IMAP), see [support.google.com/mail/answer/10957](https://support.google.com/mail/answer/10957). ## 5. Forward bank alerts from your personal Gmail Switch accounts: sign in to your **personal** Gmail (the one that actually receives bank alerts today). ### 5a. Add the forwarding address 1. Gear icon → **See all settings** → **Forwarding and POP/IMAP**. 2. Click **Add a forwarding address** and enter your dedicated account (e.g. `yourname.finance@gmail.com`). 3. Google sends a confirmation email to the dedicated account. Sign in there, open the confirmation, and click the verification link. 4. Back on the personal account, refresh the settings page and confirm the dedicated address is listed as verified. Do **not** change the default ("Forward a copy … and keep Gmail's copy in the inbox") globally — you only want to forward bank alerts, not all email. Use a filter for that, below. ### 5b. Create a filter that matches bank senders 1. In Gmail, click the filters icon in the search bar (or use **Settings → Filters and Blocked Addresses → Create a new filter**). 2. In the **From** field, paste one of the sender expressions below depending on which banks you use. To cover multiple banks with a single filter, join them with ` OR `. 3. Click **Create filter**. 4. Check **Forward it to** and choose your dedicated address. 5. Optionally also check **Skip the Inbox (Archive it)** and/or **Apply the label** → `Forwarded/Finance` so the clutter stays out of your personal inbox. 6. Click **Create filter** to save. Sender patterns, matching the senders the pipeline parses — by sender-domain match, or body-text detection for Interac-routed alerts (`src/finance/email_pipeline.py`): | Bank | Suggested Gmail `From:` filter | | ------------ | ------------------------------------------------------------------ | | RBC | `from:(@alerts.rbc.com)` | | CIBC | `from:(@cibc.com)` | | MBNA | `from:(@mbna.ca)` | | PC Financial | `from:(@pcfinancial.ca)` | | Simplii (e-transfers only) | `from:(@payments.interac.ca)` | | Interac | `from:(catch@payments.interac.ca OR @payments.interac.ca)` | **Simplii note.** The Simplii parser only handles Interac e-Transfer notifications, which arrive from `payments.interac.ca` rather than `simplii.com`. Filter on the Interac sender domain for Simplii. (Direct purchase alerts on `@simplii.com` are not currently supported by the parser.) One catch-all filter covering every supported institution looks like: ``` from:(@alerts.rbc.com OR @cibc.com OR @mbna.ca OR @pcfinancial.ca OR @simplii.com OR @payments.interac.ca) ``` Interac e-Transfer notifications from RBC, Simplii, or any other Canadian bank all come from `payments.interac.ca` — keep that line even if you don't think you receive them today. `@simplii.com` is in the filter on purpose, even though those direct alerts aren't parsed yet: forwarded copies land in the **Needs review** queue instead of being dropped, AI extraction (if enabled) can recover the transactions, and the captured emails are exactly the samples a future Simplii parser needs. If you see bank alerts in your inbox from a slightly different domain (e.g. a regional RBC address), add it to the filter's `from:` list — the pipeline matches on substrings of the domain, so anything ending in one of the domains above is picked up automatically. ### 5c. Sanity check After saving the filter, send yourself a small real transaction, or wait for the next bank alert. Within a minute or two, the email should appear in the dedicated account's inbox. If it lands in **Spam**, open it and click **Not spam** so future alerts skip the spam folder. ## 6. Configure `.env` `.env` is a local file at the repo root that stores configuration as environment variables. It's gitignored — never commit it. The Docker Compose stack reads it automatically when you run `docker compose up`. At the repo root you'll find `.env.example`. Copy it to `.env` (if you haven't already) and fill in the IMAP block. The variable names below are the exact ones read by `src/finance/imap_poller.py::main()` and passed through by `docker-compose.yml` — don't rename them. ```dotenv # IMAP Polling (for self-hosted email ingestion) IMAP_SERVER=imap.gmail.com # Gmail's IMAP host; rarely needs changing IMAP_PORT=993 # IMAP-over-SSL; default is correct for Gmail IMAP_USER= # the DEDICATED account from step 1 IMAP_PASSWORD= # the APP PASSWORD from step 3 (NOT your Google password) IMAP_FOLDER=INBOX # which mailbox to poll; INBOX is correct for the filter above IMAP_POLL_INTERVAL=60 # seconds between polls; 60 is a reasonable default ``` Optional but recommended: ```dotenv OPENAI_API_KEY= # enables AI category suggestions; without it, categories default to "Miscellaneous" ``` Defaults baked in: if you omit `IMAP_SERVER`, `IMAP_PORT`, `IMAP_FOLDER`, or `IMAP_POLL_INTERVAL`, the poller falls back to `imap.gmail.com`, `993`, `INBOX`, and `60` respectively. `IMAP_USER` and `IMAP_PASSWORD` are required — the process exits immediately if either is missing. **Never commit `.env`.** It's already in `.gitignore`; keep it that way. ## 7. Start the stack From the repo root: ```bash docker compose up -d ``` This brings up two services defined in `docker-compose.yml`: - `finance` — the FastAPI dashboard on port 8000 (open [http://localhost:8000](http://localhost:8000)) - `imap-poller` — the long-lived IMAP polling daemon Both containers share the `finance_data` volume, so the poller writes transactions directly into the same SQLite DB the dashboard reads. To stop the stack: ```bash docker compose down ``` ## 8. Verify it's working Stream the poller logs: ```bash docker compose logs -f imap-poller ``` (You can also use `docker logs -f ` directly, where the container name is usually `_imap-poller_1` or `-imap-poller-1` — `docker ps` shows the exact name.) On a healthy startup you should see, in order: ``` Starting IMAP poller: server=imap.gmail.com, user=yourname.finance@gmail.com, folder=INBOX, interval=60s Connected to imap.gmail.com:993 as yourname.finance@gmail.com ``` After that, one of two things happens on each poll cycle: - **Idle tick** — nothing is logged until new mail arrives. That's normal; the daemon only emits a line per fetched message. - **New mail** — you'll see `Fetching uid=` followed by `uid=: new transaction stored as …` and, once per batch, `Processed message(s)`. The first real transaction usually appears on the dashboard within one to two poll intervals (so ~1–2 minutes on the default 60-second interval) after the bank email actually arrives in the dedicated account. Refresh the dashboard in your browser to see it. If you want to force a fast end-to-end test, send yourself a small e-Transfer from one of your supported banks — the alert should flow through forwarding → dedicated inbox → poller → dashboard within a couple of minutes. --- ## Troubleshooting **"Invalid credentials" / authentication failure on connect.** Gmail rejects your regular account password for IMAP — you *must* use an App Password. Re-check that `IMAP_PASSWORD` is the 16-character value from step 3 and not the password you log in to Google with. If in doubt, generate a new App Password and paste it in fresh. Spaces in the App Password are fine; the poller strips them. **Server rejects LOGIN with an "IMAP disabled" message.** On personal Gmail this is rare — IMAP is always on since January 2025. The likely cause is a Google Workspace account where an admin has disabled IMAP for the org. Ask your admin to enable IMAP under **Apps → Google Workspace → Gmail → End User Access**, then restart the poller: `docker compose restart imap-poller`. **Not receiving emails / the dashboard stays empty.** Three common causes, in order of likelihood: 1. The bank alert landed in **Spam** on the dedicated account. Open the message and click *Not spam* so future alerts go to `INBOX`. 2. Your Gmail forwarding filter on the personal account doesn't match the sender. Check step 5b and confirm each bank you use is in the `from:` list. Don't forget `payments.interac.ca` for e-Transfers. 3. The forwarding address was added but never verified. Go back to **Forwarding and POP/IMAP** on the personal account and make sure the dedicated address shows as verified, not pending. **"Connection refused" or timeout errors in the poller log.** Something between the container and Gmail is blocking port 993. Confirm: - `IMAP_SERVER=imap.gmail.com` and `IMAP_PORT=993` (no typos, no `imaps://`, no trailing spaces). - Your host firewall or corporate network allows outbound TCP 993. - If you run behind a VPN, try disabling it briefly to isolate the problem. The poller automatically retries with exponential backoff (5s → 10s → … up to 5 minutes), so transient network blips self-heal — persistent errors every cycle indicate a config or firewall issue. **App Password option missing, even though 2-Step Verification is on.** Some Google Workspace administrators disable App Passwords organization-wide via Admin Console → Security policy. This is common on work or school accounts and can only be re-enabled by the admin. The simplest workaround is a dedicated personal `@gmail.com` account (step 1). App Passwords are available on personal Gmail accounts with 2SV enabled, except for accounts enrolled in Advanced Protection or whose 2SV is set up with security keys only. --- ## AI categorization (optional) The dashboard can use OpenAI to auto-assign a category to each new transaction. AI categorization is enabled by default when `OPENAI_API_KEY` is set; toggle in Settings → Intelligence to disable it. Manual category overrides and merchant aliases run **before** OpenAI, so local rules always win — OpenAI is only consulted when no override matches. ### What gets sent to OpenAI When a built-in parser recognizes the bank, it extracts the fields locally and OpenAI sees only two things, for the category lookup: - the transaction amount (e.g. `42.50`) - the merchant name as it appears in the bank email (e.g. `STARBUCKS #1234`) That is the whole payload. Your account number, card number, balance, email address, and every other transaction stay on your machine. ### When the full email is sent If an email can't be read by a built-in parser — an unrecognized sender, or a known bank that changed its format — and email rescue is on, Tidings falls back to OpenAI to recover it. The email's **subject and body** are sent so the model can decide whether the message is a transaction and pull out the amount, merchant, and type. Whatever the bank put in that email goes with it. Email rescue is its own consent (`ai_extraction_enabled`), separate from categorization — turn it off under Settings → Intelligence and unparsed emails are held for review with no AI call, whatever the categorization switch says. Like categorization, it comes on when `OPENAI_API_KEY` is set. The surest way to keep everything local is to run without an OpenAI key at all: the pipeline then never calls out, unparsed emails are simply held for review, and recognized transactions fall back to the `Miscellaneous` category. ### How to turn it off - **UI**: Settings → Intelligence — the AI categorization and email rescue toggles (live; takes effect on the next new transaction) - **Config file**: set `"ai_categorization_enabled": false` and `"ai_extraction_enabled": false` in `data/config.json` - **No API key**: if you never set `OPENAI_API_KEY`, both flags default off and neither feature ever runs When disabled, transactions receive the `Miscellaneous` category. You can re-categorize them by hand in the transaction list, or add a Category Rule under Categorize → Rules so future transactions from the same merchant get the right category automatically — all without any data leaving your machine. --- ## Next step Your inbox is wired. The last step is notifications: [pick a provider](/notifications/) so you get alerted when transactions arrive. --- # Notifications Source: https://docs.gettidings.com/notifications.md > Pick a push or SMS provider for new-transaction alerts. The app sends push/SMS notifications for every new transaction in real time (and, on the AWS Lambda path, for the monthly summary job). Pick **one** provider and set its env vars in `.env`. The recommended default for self-hosters is [ntfy.sh](https://ntfy.sh) — free, no account needed, and works on iOS and Android. If no provider is configured, the app falls back to a log-only mode and writes notifications to stdout — handy for local testing before wiring up a real provider. ## Provider comparison | Provider | Cost | Best for | |----------|------|----------| | **Ntfy** (default) | Free | Self-hosters; everyone who doesn't specifically need SMS | | **Twilio** | ~$5–10/mo | You want SMS delivery and don't use AWS | | **AWS SNS** | ~$0.012/SMS US (base $0.0077 + carrier fee ~$0.0042) + 10DLC fees | You already deploy on AWS | | Log-only | Free | Local testing; no provider configured | --- ## Ntfy (recommended) Ntfy is a free push notification service. It's anonymous — no signup, no account. Messages are delivered to any device subscribed to your *topic*. ### Security: pick a hard-to-guess topic Your topic name **is** the auth. Anyone who knows it can read your notifications, and anyone can publish to it. Do **not** use something guessable like `finance` or `my-transactions`. Use a random slug: ```text finance-dashboard-x8k2m9p4 ``` A shell one-liner to generate one: ```bash echo "finance-dashboard-$(openssl rand -hex 6)" ``` ### Steps 1. Install the ntfy mobile app ([iOS](https://apps.apple.com/us/app/ntfy/id1625396347) / [Android](https://play.google.com/store/apps/details?id=io.heckel.ntfy)) or use the [web app](https://ntfy.sh/app). 2. Subscribe to your chosen topic — e.g. `finance-dashboard-x8k2m9p4`. 3. Set the URL in `.env`: ```bash NOTIFICATION_URL=https://ntfy.sh/finance-dashboard-x8k2m9p4 ``` 4. Test-send from your terminal: ```bash curl -d "hello from ntfy" https://ntfy.sh/finance-dashboard-x8k2m9p4 ``` If the push arrives on your phone, the connection is working. ### Self-hosting If you want full control (no reliance on ntfy.sh), you can run your own ntfy server in a container. Point `NOTIFICATION_URL` at your instance instead. See [docs.ntfy.sh/install](https://docs.ntfy.sh/install/). --- ## Twilio (SMS) Twilio is a paid SMS provider. Pick this if you specifically want text messages rather than push notifications. ### Steps 1. Sign up at [twilio.com](https://www.twilio.com/), verify your phone, and buy a phone number (minimum ~$1.15/mo for a US local number). 2. From the Twilio console, grab: - **Account SID** - **Auth Token** - The **From** number you just purchased 3. Add to `.env` (exactly these env-var names — the service reads them directly): ```bash NOTIFICATION_PROVIDER=twilio TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx TWILIO_AUTH_TOKEN=your-auth-token TWILIO_FROM_NUMBER=+15551234567 TWILIO_TO_NUMBER=+15557654321 ``` > **Note:** Twilio is **not** auto-detected from its env vars — you must set `NOTIFICATION_PROVIDER=twilio` explicitly. (Only `SNS_TOPIC_ARN` and `NOTIFICATION_URL` auto-detect.) > **Security note:** Auth Token works for self-hosting, but Twilio recommends [API Keys](https://www.twilio.com/docs/iam/api-keys) (or OAuth 2.0) for production — they can be scoped and revoked individually. The current Tidings notifier uses Auth Token; that's fine for personal self-hosting but rotate regularly. 4. Test-send: ```bash uv run python -c "from src.finance.notification_service import send_raw; send_raw('Test', 'hello from twilio')" ``` You should get an SMS within a few seconds. ### 10DLC (US recipients) US carriers require **10-digit long-code** registration for application-to-person (A2P) SMS. Twilio will prompt you through brand + campaign registration when you add the number. This takes a few business days for approval. - **Canadian numbers sending only to Canadian numbers** may not need 10DLC. As of March 2025, Canadian carriers require Canadian A2P SMS registration for newly purchased numbers. Confirm current Twilio guidance for your region. - **If you skip 10DLC for US traffic**, messages may silently fail to US recipients (no bounce, no error — they just don't arrive). See *Troubleshooting* below. ### Cost Roughly $0.0083 per SMS segment plus US carrier surcharges (~$0.004) — about $0.012 per outbound segment in practice. For typical personal use, budget $5–10/month: the SMS volume itself is cents, but the US 10DLC campaign monthly fee (~$1.50–10/mo depending on campaign type) dominates the total. --- ## AWS SNS (advanced) > **Most users should pick Ntfy.** Only use SNS if you're already deploying the Lambda email parser on AWS and want notifications to go through the same account. ### Steps 1. Create an SNS topic in your preferred AWS region (e.g. `us-west-2`). 2. Subscribe your phone number to it as an SMS subscription. 3. **For US numbers:** you'll hit 10DLC requirements, same as Twilio. Register a brand + campaign through the AWS End User Messaging SMS console — AWS now manages SMS origination IDs and 10DLC registration there rather than in the SNS console. See [Mobile text messaging with Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/sns-mobile-phone-number-as-subscriber.html) for the broader overview. 4. Set in `.env`: ```bash SNS_TOPIC_ARN=arn:aws:sns:us-west-2:YOUR_ACCOUNT_ID:your-topic-name AWS_REGION=us-west-2 ``` Setting `SNS_TOPIC_ARN` alone is enough — the provider auto-detects from it. `AWS_REGION` defaults to `us-west-2` if unset. 5. AWS CLI credentials must be available to the running process (or container). Use an IAM role if running on EC2/ECS/Lambda, or mount `~/.aws/credentials` into the container for local self-hosted use. ### Cost Roughly $0.012 per SMS to US destinations (base $0.0077 + carrier fee ~$0.0042) plus 10DLC registration fees. Canadian SMS is slightly cheaper. No monthly number fee — pay per message. --- ## Log-only fallback If none of `SNS_TOPIC_ARN`, `NOTIFICATION_URL`, or an explicit `NOTIFICATION_PROVIDER` are set, the app logs notifications to stdout: ```text NOTIFY [Transaction]: Transaction | Company: TIM HORTONS | Amount: $4.85 | ... ``` Useful for verifying the parser and categorization pipeline before configuring a real provider. **Nothing is sent to your phone in this mode.** --- ## Provider auto-selection The service picks a provider at startup in this order: 1. **`NOTIFICATION_PROVIDER`** — if set explicitly to `ntfy`, `twilio`, `sns`, or `log`, that wins. 2. **`SNS_TOPIC_ARN`** — if set, use SNS. 3. **`NOTIFICATION_URL`** — if set, use ntfy. 4. **Log-only.** > **Twilio is only selected via the explicit `NOTIFICATION_PROVIDER=twilio` setting.** Its env vars (`TWILIO_ACCOUNT_SID`, etc.) are not auto-detected. To force a specific provider regardless of what else is in the environment: ```bash NOTIFICATION_PROVIDER=ntfy # or twilio | sns | log ``` An unrecognized value logs a warning and falls back to log-only. --- ## Troubleshooting **"I'm not getting notifications."** The log-only fallback engages silently whenever required env vars are missing. Check your `.env`: - Ntfy: is `NOTIFICATION_URL` set? - SNS: is `SNS_TOPIC_ARN` set? - Twilio: is `NOTIFICATION_PROVIDER=twilio` set (not just the `TWILIO_*` creds)? Then tail the daemon log and look for a `NOTIFY [Transaction]: ...` line — if you see one, you're in log-only mode. **"SNS 10DLC blocked."** Register a brand and campaign in the AWS SNS console. Until approved, SMS to unregistered US numbers will be dropped. **"Twilio error 30034."** 10DLC registration is incomplete or rejected — finish brand and campaign signup in the Twilio console. (Error 30032 is the related Toll-Free Verification error if you're using a toll-free number instead.) **"Ntfy says it delivered but no push on my phone."** Your phone's battery optimizer or notification settings are blocking delivery. - Android: enable **Instant delivery** in the ntfy app settings (uses a foreground service for prompt delivery), then whitelist ntfy in Settings → Apps → ntfy → Battery → Unrestricted. Without instant delivery, FCM doze mode can delay pushes by minutes or hours. - iOS: ensure notifications are enabled for the ntfy app under Settings → Notifications. APNs delivers regardless of Background App Refresh; if pushes still don't arrive, check Focus modes / Do Not Disturb. **"Unknown NOTIFICATION_PROVIDER=... — falling back to log-only."** Typo in `NOTIFICATION_PROVIDER`. Valid values: `ntfy`, `twilio`, `sns`, `log`. --- ## Next step Provider configured? Return to [Docker setup](/self-hosting/docker/) or [AWS deployment](/self-hosting/aws/) to wire it in. --- # How Tidings works Source: https://docs.gettidings.com/using/how-it-works.md > The path a bank email takes to become a categorized transaction, and the rules that decide what changes when. Tidings turns the transaction alerts your bank already sends into a private finance journal. Nothing is scraped, and no bank credentials are stored. Everything downstream reads from one local store, so the concepts on this page recur on every other page in this section. ## The pipeline *Figure: a bank alert email travels bank → forwarder inbox → parser → categorization → local store, and every page reads from that one store.* Each transaction follows the same path: 1. Your bank emails a transaction alert. 2. Your [forwarder](/using/settings/) sends that email to Tidings. 3. A deterministic, per-bank parser reads the amount, merchant, and date from the email. 4. Your [category rules](/using/categorization/) (overrides) are checked first — matching the merchant by exact name, alias, or fuzzy similarity. A confident match applies the category and skips everything else. 5. If you have opted into AI [categorization](/using/categorization/) under Settings → Intelligence, the categorizer runs next; otherwise the transaction stays uncategorized. 6. The result is stored locally — SQLite on the self-hosted setup, DynamoDB on the AWS variant. Every page you see is a different read of that single store. ## Two queues that sound alike The most confusable pair in Tidings is the difference between an email that never became a transaction and a transaction that has no category. *Figure: a bank email either fails to parse — landing in Needs review as a quarantined email with no transaction — or becomes a transaction, which sits in the attention queue on Transactions until its category is confirmed.* - [Needs review](/using/needs-review/) holds raw emails no parser could read. They are quarantined, not transactions — no transaction exists for them until you recover one. This is where you go when a bank is not yet supported. - The attention queue on [Transactions](/using/transactions/) holds transactions that already exist but still need a category. One is about emails that failed to parse; the other is about transactions that need a decision. Keeping them apart saves a lot of confusion. ## What changes when you edit Different edits reach different distances. Some rewrite history, some only shape what happens next. | Action | What it changes | Retroactive? | | ---------------------------------------------- | ---------------------------------------------------------- | ------------------ | | Rename a category | Transactions, overrides, and budget groups everywhere | Yes, always | | Add a category rule (override) | New transactions from then on | No | | Add an auto-ignore rule | New transactions, unless you choose to apply it to history | No, unless applied | | Add a [merchant alias](/using/categorization/) | The display name only — nothing stored changes | No stored change | The distinction to hold onto: renaming a category is the one edit that reaches backward through everything you have already recorded. Rules and aliases shape the future or the surface, not the record. ## The second way in Email misses things — a card that sends no alerts, a month before you set up forwarding. [Statements](/using/statements/) are the second ingestion path: you upload a bank PDF and Tidings parses the transactions it missed. ## Time Your instance timezone decides which day and month a transaction belongs to, which is why the boundary matters for month totals. You set it in [settings](/using/settings/). ## Where to go next - [Architecture](/architecture/) for the technical deep dive on the pipeline and storage. - [Quickstart](/quickstart/) to set up your own instance. --- # The journal Source: https://docs.gettidings.com/using/journal.md > The day-grouped view of a month's spending, and why its totals differ from the full ledger. The journal is the default view: a single month read one day at a time, newest day first. ## What the journal shows, and what it leaves out The journal shows only spending you can act on: transactions that are a spending type with an amount above zero, and that are neither deleted nor [ignored](/using/categorization/). Refunds, deposits, and transfers are filtered out here and appear on [Transactions](/using/transactions/) instead. > **note** > > Because the journal drops refunds, deposits, and transfers, a day's total here reflects spending only. If a number looks smaller than a raw count of that day's activity would suggest, the difference is the non-spending rows the journal sets aside. ## Reading a month Each day carries its own total; the month accumulates as you scroll. The monthly ceiling shown at the top is your annual [spending ceiling](/using/budgets/) divided by twelve — it is derived from the annual target, not a separate monthly budget you set. The month-over-month figure compares this month's spending against the prior month up to the same day. Early in the month you are compared against the same early stretch of last month, not the whole of it, so the two are a fair match rather than a partial month against a full one. ## Day summaries A day summary is a short piece of prose about that day's spending, written by the AI provider you configure. You can generate one on request, or let Tidings write them on a daily schedule. - **Summarize month** fills only the days that are still missing a summary, so it never overwrites what you have already generated. - A per-day refresh regenerates that one day. - Only one generation runs at a time; a second request while one is in flight is skipped rather than queued. Each summary is saved as a plain text file under `data/journal//`, so the writing lives on disk alongside your data rather than in a database. Day summaries need a daily-summary provider set under Settings → Intelligence — see [settings](/using/settings/). ## On a phone Below 768 pixels the journal switches to a single-column layout with a bottom navigation bar, keeping the same day-by-day read within reach of a thumb. --- # Transactions Source: https://docs.gettidings.com/using/transactions.md > The full ledger, including the refunds and transfers the journal filters out, plus the attention queue and the trash. Transactions is the complete ledger. Everything the [journal](/using/journal/) sets aside — refunds, deposits, transfers, zero-amount rows — is here, alongside your spending. ## Month view and range search By default you look at one month, filtered and sorted right in the browser for an immediate response. Pick a date range instead and Tidings switches to a server-backed search across months. That search returns up to roughly a thousand rows; when a range matches more, a notice shows how many you are seeing and you export the full set to CSV. ## The attention queue The banner reading "N transactions need a category" is the attention queue: transactions that still need a human decision. A row lands here when it is still Miscellaneous with no one having set it, or when the AI extracted or categorized it and no one has confirmed the result yet. **Confirm** marks the category as reviewed without changing it — it clears the row from the queue by agreeing with what is already there. > **note** > > The attention queue is transactions that need a category. It is not [Needs review](/using/needs-review/), which holds raw emails that never parsed into transactions at all. One is a decision about an existing transaction; the other is an email waiting to become one. ## Changing a category versus changing behavior Editing a transaction's category changes that one transaction and nothing else. To change how future transactions from the same merchant are categorized, you make a rule — see [categorization](/using/categorization/). Your manual corrections are also what teach Tidings the rules it suggests, so a careful edit now shapes the suggestions later. ## Deleting and the trash Delete is a soft delete. The transaction moves to the trash, drops out of every report and total, and stays there until you decide what to do with it — there is no automatic purge and no retention clock. - **Restore** brings a transaction back from the trash into your active ledger. - **Permanent delete** removes it for good and cannot be undone. You browse the trash a month at a time at `/transactions/trash`. ## Receipts to file A receipt — a photo, an image from the library, or a PDF — can be attached directly from a transaction's row, or uploaded on its own. Standalone uploads collect under **Receipts to file** at the top of the page until you file them: filing matches the receipt to the transaction it belongs to, with ranked candidates by date and amount, and parses its contents when receipt-parsing consent is on in [settings](/using/settings/). Attached receipts also serve as the evidence rows in [tax receipts](/using/tax/). --- # Categorization Source: https://docs.gettidings.com/using/categorization.md > How the Categorize page names and categorizes your transactions, and how each mechanism reaches into your history differently. The Categorize page holds four tabs, each a different lever over how a transaction gets named and filed. They look similar, but they reach into your data in very different ways. The distinction that matters most: some changes touch only transactions that arrive from now on, and one touches everything you already have. ## Categories This tab is your category vocabulary, grouped into super-categories. Those groups are the same ones [Summary](/using/summary/) and [Budgets](/using/budgets/) read from, so the list here is the shared spine of the whole app. Renaming a category is the one retroactive action in the product. When you rename, Tidings cascades the new name across every transaction that carried the old one, across your overrides, and across your budget entries in a single pass. Every other tab on this page changes future transactions only. > **note** > > Renaming is a rename, not a merge — a rename to a name that already exists is refused. To fold one category into another, recategorize its transactions to the target category, then delete the empty one. ## Rules Rules are category overrides: a merchant pattern pinned to a category. An override applies to new transactions at ingestion, and it skips the AI categorizer entirely — the pattern wins before the model is ever asked. Transactions already in your history keep whatever category they were saved with; a new rule does not reach back and recategorize them. Matching runs in tiers. An exact merchant name matches first, then a normalized or aliased form, and only then a fuzzy similarity score. A fuzzy match auto-applies when the similarity reaches 0.90 or higher. *Figure: a new transaction falls through three tiers — your rules (exact, normalized, alias, then fuzzy match) always win; the AI categorizer runs only if you opted in; anything left lands in Miscellaneous and joins the attention queue.* Tidings also learns. When you correct a merchant's category by hand, that correction becomes a suggested rule. Dismiss a suggestion and it stays quiet, but if you correct the same merchant again it resurfaces. Where you have accumulated several overrides that point at the same merchant, Tidings offers to consolidate them. ## Auto-ignore Auto-ignore marks a merchant pattern as ignored at the moment a transaction is written. An ignored transaction is excluded from totals, from budget pace, and from [briefings](/using/insights/) — the mechanism exists for credit-card payments, transfers, and other movements that would double-count real spending. By default an ignore rule affects new transactions only. Choose apply to history and Tidings backfills the rule across your existing rows, with one deliberate exception: it never re-ignores a transaction you previously restored by hand. Suggestions surface for merchants whose recent activity you have mostly been ignoring one transaction at a time. ## Aliases An alias maps a raw bank description to a display name. This is purely cosmetic. It changes how a merchant reads on [Merchants](/using/merchants/) and the [income statement](/using/income-statement/), and it drives how transactions group there, but the stored transaction and its category are untouched. Rename a merchant here and nothing about its classification moves. Read [how Tidings works](/using/how-it-works/) for where categorization sits in the ingestion path, and see [Needs review](/using/needs-review/) for the emails that never became transactions in the first place. --- # Needs review Source: https://docs.gettidings.com/using/needs-review.md > The quarantine for bank emails Tidings could not parse, and how retrying or setting them aside works. Needs review is where bank emails go when Tidings receives them but cannot read them. The parsers could not find a merchant or an amount, so the raw email is kept intact and no transaction exists yet. Nothing is lost, but nothing is counted either until a retry succeeds. ## Not the same as the attention queue This is the pair most worth keeping straight. The attention queue on [Transactions](/using/transactions/) holds transactions that exist but have no category — real rows waiting to be filed. Needs review holds emails that never became transactions at all. One is a categorization gap; the other is a parsing gap. If a purchase is missing from your totals entirely, look here, not there. ## Retrying a row Retry re-runs the deterministic parsers against the stored email — the same regex-based parsers, never AI. It exists to confirm a parser fix, so the outcome tells you whether the fix took: - If the email now parses, Tidings creates the transaction and clears the row. - If the result matches something you already have, the row is marked handled as a duplicate. - If the parsers still cannot read it, the email stays quarantined for you to try again later. When a retry does create a transaction, categorization runs through your override rules, so a merchant you have already pinned lands in the right category on arrival. > **note** > > Retry deliberately skips AI. AI extraction is the ingestion-time recovery path; the retry here is the manual-confirmation path for a parser you have changed. ## Retry all When two or more quarantined emails share one institution or sender domain, Tidings offers a retry all for that group. This is the batch you run right after a new parser lands: point it at the backlog from that bank and recover the whole set in one pass. ## Setting a row aside Set aside dismisses a row you do not intend to recover. It removes the email from the active list without deleting the captured email itself, so the record stays on disk. ## When a bank keeps failing If the same bank lands here again and again, the fix is not more retries — it is a parser that can read that bank's emails. The [add a parser](/add-a-parser/) guide walks through writing one, and these quarantined emails are exactly the raw material that workflow uses. Once the parser exists, retry all clears the backlog it was built from. --- # Summary Source: https://docs.gettidings.com/using/summary.md > The month-scoped overview — stat cards, a Trend or Flow visualization, and a category table with six months of context. The summary is one page scoped to the month in the picker: stat cards, a visualization you can toggle between Trend and Flow, and a per-category table carrying six months of context. ## The one-line headline The sentence under the title reads the numbers for the month and states them plainly. It is deterministic, assembled from the same figures on the page, not written by the AI briefing you find in [insights](/using/insights/). It says what the month looks like; it does not judge it. ## Pace, only for the current month When the month in the picker is the current calendar month, the cards describe pace — where spending stands partway through the month. You see "Spent so far", a "Projected month end" with a typical range, and a "vs typical pace" figure that compares today's total against what a typical month looks like by this day. Pace needs a running month and some history to project into. Past months are already finished, so they show completed-month cards instead — the month's total against the month before. If pace cannot be computed — too little history, for instance — the page still loads, just without the pace cards. ## Trend and Flow The visualization has two views, and which one you are looking at lives in the page's URL, so a Flow view stays Flow when you share or reload the link. Trend charts the month in the context of recent months. Flow is a Sankey diagram of where the month went — income on one side, category groups on the other, built from the same figures as the cards, so the two never disagree. On narrow screens, below 768 pixels, the Sankey gives way to a text breakdown of the same numbers. ## The category table and groups Below the visualization, the per-category table lists the month's spending with six months of history alongside each category, so a number reads against its own recent past rather than in isolation. Category groups — the way categories collect into larger buckets — are editable here. Those groups are shared: the same grouping shapes the Flow diagram, the [budgets](/using/budgets/) dashboard, and the [income statement](/using/income-statement/). Edit them once and every surface that leans on them follows. For how a transaction lands in a category in the first place, see [categorization](/using/categorization/). --- # Budgets Source: https://docs.gettidings.com/using/budgets.md > Annual per-category targets assessed by proration, a top-level spending ceiling, and the editor where you set them. Budgets in Tidings are annual. You set a target for each category for the year, and the dashboard reads year-to-date spending against it. There is no month-to-month rollover and no per-month balance to fund — a target is a yearly number, and pace is how far into that number you are. ## How pace is assessed Pace compares what you have spent so far this year against the elapsed fraction of the year. Two months in, roughly a sixth of the year has passed, so a category on an even pace has spent about a sixth of its target. Spend ahead of that fraction and the category reads as off pace; behind it, ahead of pace. That even-pace assumption does not fit every category, so each one carries a type: - **Fixed** — steady month to month, like a subscription. - **Variable** — present every month but varying, like groceries. - **Lumpy** — a once-or-twice-a-year expense, like an annual fee or insurance. A lumpy category is judged against its full-year target rather than a monthly pace, because a single large charge in one month is expected, not an overrun. Fixed and variable categories are assessed against the elapsed-year fraction. ## The spending ceiling The ceiling is a separate, top-level annual cap on total spending. It is not the sum of the category targets — you can set category targets that add up to less, and the ceiling stands on its own as the outer limit. Once spend crosses it, the over-state reads plainly, as in "$112 over ceiling". The ceiling also feeds the [journal](/using/journal/): the journal's monthly ceiling is the annual ceiling divided by twelve. ## Views The dashboard shows either a year-to-date or a monthly view, with an optional column comparing against the prior year. Categories you have spent in but never budgeted collect in their own unbudgeted section, so nothing goes missing just because it lacks a target. ## Setting targets The first time you open budgets for a year, you choose where to begin: from scratch, or from your last twelve months of historical averages. The editor's pre-fill uses those same averages — it fills in targets, suggests a type for each category, and proposes a ceiling, all of which you can adjust. In the editor, a target can be entered monthly or annually, and the other figure is derived from it — enter $200 a month and the annual reads $2,400. If you save against a version that changed underneath you, in another tab for instance, the save is refused and you are asked to reload rather than quietly overwrite the other edit. For where the category types also surface, see the [income statement](/using/income-statement/) and [merchants](/using/merchants/). --- # Insights Source: https://docs.gettidings.com/using/insights.md > The monthly briefing workspace — always-on structured cards, a generated narrative, and exactly what Tidings sends to a model. The Insights page is where a month gets read back to you. It is the one page you sit with rather than act on. ## Two layers The cards at the top — momentum against last month, and the month's quiet anomalies — are computed directly from your transactions. No model is involved, so they are always present once the month has data, and they read the same every time. The briefing sits below them. It is the generated part, and it is optional: you can look at the cards for a month you have never generated a briefing for. ## What a briefing covers A briefing follows a fixed shape, the same every month: the month in brief, what changed, where the month went, worth attention, your notes, and looking ahead. The sections are set in the prompt, so successive months read as the work of one steady hand rather than a fresh take each time. Generation runs in the background, one month at a time, and usually finishes in about 30 seconds. The provider, model, and reasoning effort come from task routing in [settings](/using/settings/) — a briefing runs only once you have configured a provider there. ## What Tidings sends The briefing is the one place Tidings hands a slice of your finances to a model, so it is worth knowing exactly what goes across. The context is: this month's summary and the previous month's, six months of trend, your budget targets and the pace math derived from them, the notable category changes and anomalies, the month's largest transactions, any notes you wrote on individual transactions, and the continuity signals below. You can inspect the exact context a briefing sees — it is served read-only in the app, so nothing is hidden from you. ## Continuity Alongside the numbers, Tidings gathers a set of continuity signals so a briefing can speak across time rather than in isolation: the same month one year ago, categories that recur only once or twice a year, flat recurring charges like a mortgage or strata, merchants you usually mark ignored, an excerpt of last month's briefing so this one can follow up on what it raised, and your standing briefing memo from Settings → Intelligence. The memo is free text you write once; it shapes every briefing after. ## Every figure is checked After a briefing is generated, Tidings traces each dollar amount and percentage in it back to the underlying context. If a figure does not match, the page says so, naming how many did not trace. Silence is the success state — when nothing is flagged, every figure checked out. A flagged briefing is still saved; the check rides alongside it rather than blocking it. ## Where briefings live Briefings are written to disk on your machine, under `data/insights/`, as plain markdown. The page lists what has been saved for the month, newest first, and you can regenerate at any time — each run is saved separately, so a regeneration never overwrites the last one. --- # Merchants Source: https://docs.gettidings.com/using/merchants.md > Read-only merchant intelligence over a trailing six months — recurring detection, burn rate, and price-change signals. The merchants page looks at who you pay, across the trailing six months. It is read-only — a view over your spending, not a place to edit it — and every figure is derived from data Tidings already has. ## Where the numbers come from Nothing new is collected here. The page is built entirely from your existing monthly summaries, the same per-month figures the rest of Tidings runs on. Results are cached for an hour, so a very recent edit — a re-categorized transaction, a merged alias — can lag until the cache refreshes. Merchants group by canonical alias, so the several raw names one merchant sends collect into a single entry. ## How merchants are classified Over the six-month window each merchant is sorted by how it recurs: - **Fixed** — present every month at a steady amount. These read as subscriptions. - **Variable** — present every month, but the amount moves. - **Lumpy** — present in some months, not others. The recurring burn rate is the sum of the fixed merchants' typical amounts — the spending locked in before the month begins. Discretionary spending is then this month's total minus that burn rate — what landed this month outside the subscriptions. ## Price changes, new, and churned For a fixed merchant, Tidings compares its last two active months. If the amount changed by at least 5 percent and at least a dollar, the change is surfaced — a subscription that crept up, seen without you watching for it. A merchant is marked **new** when it appears in the last two months but was absent earlier in the window — two consecutive months, so a single one-off charge does not read as a new relationship. A merchant is marked **churned** when it was active earlier in the window but has gone silent for the last two months. For the types these classifications share with your budget, see [budgets](/using/budgets/); for how merchant aliases are set, see [categorization](/using/categorization/). --- # Income statement Source: https://docs.gettidings.com/using/income-statement.md > An annual spreadsheet of income against expenses by month, grouped by budget category type. The income statement lays a year out as a spreadsheet: income across the top, expenses below, one column per month, with annual totals down the side. ## What counts as income Income here is the deposit stream — the e-transfers and deposits Tidings has actually seen arrive. It is not payroll parsing: Tidings does not detect a salary or read a pay stub. If a deposit reaches your account and Tidings sees the email for it, it lands in the income section; if money arrives some other way, it does not. Read the income figures as "deposits observed", not "everything earned". ## How expenses are grouped Expenses are grouped into sections by budget category type — fixed, variable, and lumpy — the same types you set in [budgets](/using/budgets/). A category with no budget defaults to variable. The fixed section's annual total surfaces on its own as a fixed-commitments figure, the yearly and monthly weight of the expenses that do not move. Within each category, company sub-rows break the spending down by merchant, using the same merchant aliases that shape [categorization](/using/categorization/) — so a merchant that appears under several raw names collects into one row. ## Savings rate and the current year The savings rate is net over income — what is left after expenses, as a share of what came in. The current year is shown in progress rather than as a finished ledger. Future months are hidden, the current month is marked as still in progress, and the annual figures are relabeled year-to-date so a partial year is never mistaken for a full one. Alongside them sits an "on pace for … by December" projection that annualizes the year-to-date figures — a straight-line read of where the year lands if the rest of it looks like the part already seen. --- # Statements Source: https://docs.gettidings.com/using/statements.md > Uploading a bank-issued PDF statement to reconcile and import transactions that never arrived by email. A statement is a bank-issued PDF you upload for reconciliation. It is the second way transactions enter Tidings, for purchases that never came through as email alerts — cash-like charges, older history, or a bank whose alerts you never wired up. > **note** > > Statement upload is self-hosted only. It runs on your own machine against your own data; the hosted demo shows a sample statement so you can see the review screen, but the feature itself is local. ## Upload and reconcile When you upload a PDF, Tidings parses it and reconciles every row against the transactions you already have, sorting them into buckets. Nothing is written yet — the buckets are a proposal you review. The PDF itself is stored under `data/raw/statements//`. *Figure: each statement row lands in one of five buckets — matched (already in your ledger), ambiguous (a close match you decide), suspected duplicate (a cross-type twin to check), new (offered for import), or previously imported (from an earlier upload).* If no built-in parser covers the bank, Tidings falls back to your AI provider — but only when you have enabled statement-parsing consent in Settings → Intelligence. When that happens the review screen says the statement was read by AI and asks you to verify dates and descriptions before importing. Every amount is checked against the PDF text, but read the rows anyway. ## Review, then import The review screen is per-row. For each one you choose how it lands: - **Import** creates a transaction, aware of duplicates so it will not write one twice. - **Enrich** adds the statement's detail to a matched existing transaction. - **Update** overwrites a row you previously imported from a statement. A category you set during review is recorded as a manual choice, so later automation will not overwrite it. That makes the review screen a good place to file a merchant correctly once and have it stick. ## Reparsing and deleting Reparse re-runs the parser on the stored PDF while keeping the edits you already made, so a parser improvement does not cost you your manual review. Deleting a statement removes it and its parsed rows, but it does not remove transactions you already imported — those are real transactions now and stand on their own. Imported rows carry a tag for the statement they came from and appear on [Transactions](/using/transactions/) for that period, alongside everything that arrived by email. See [how Tidings works](/using/how-it-works/) for where this second ingestion path fits, and [configuration](/configuration/) for the settings that govern AI consent. --- # Tax receipts Source: https://docs.gettidings.com/using/tax.md > A year-scoped organizer that buckets claimable spending into lines, keeps the evidence, and exports a zip — records only, not advice. Tidings organizes your records; it doesn't give tax advice. The Tax receipts page gathers a calendar year's claimable spending into lines and keeps the evidence attached to each row, so that at filing time you hand your records — or your accountant — something already sorted. It does not compute deductions, estimate refunds, or decide what qualifies. That judgment stays with you. ## Turning it on Tax receipts is off until you enable it under Settings → Features — see [settings](/using/settings/). Turning it on adds the page and a per-transaction flag control on the Transactions and Journal pages. Turning it off hides both, but keeps every flag you set; they return exactly as they were when you switch the feature back on. Nothing is deleted by the toggle. ## How lines are filled Each row lands in a line based on its category. The seed mapping is shaped around Canadian CRA lines — charitable donations, medical, child care, moving, tuition, union and professional dues, and tax paid by instalments — and a personal copy overrides the seed when you want different category-to-line rules. Because membership is derived from category, most rows sort themselves; you adjust only the exceptions. ## Adjusting individual rows Two per-transaction overrides handle the exceptions. An **include** forces a row into a line you choose, even when its category would not have placed it there; an include that maps to no seed line lands in a catch-all "Other claimable" line, which appears only when it holds something. An **exclude** takes a row the category placed and moves it to that line's removed list, where it no longer counts toward the total. Overrides are per transaction, so one adjustment never reshuffles the rest. ## Evidence on every row Each row carries its own proof of the spend, in one of three forms: a receipt you attached, the source bank email the transaction came from, or the statement it was imported from. A statement-imported row has no source email, since it never arrived as one — the statement itself lives on the Statements page. ## Downloading the pack The download button streams the year as a single zip: a summary CSV rolling up each line, one CSV per line listing its rows, the receipt files themselves, and the email bodies for rows whose evidence is an email. It is a complete, offline copy of the year's claimable records, assembled from your own machine. The export is disabled in the demo, which reads only sample data. --- # Settings Source: https://docs.gettidings.com/using/settings.md > One page over every setting, organized by scope — what lives in your browser, what applies to the whole instance, and what controls access. Settings is one page, organized by a single idea: scope. Some settings live in the browser you set them from, some apply to everyone using the instance, and some control who gets in. Knowing which scope a setting belongs to tells you where it takes effect and who it reaches. ## Device settings Display and Navigation are yours alone — they persist in the browser, not on the server, so each device you open Tidings on keeps its own. **Display** covers the mode (Light, Dark, or System), the palette — Warm Paper (the default), Plain, Nord, Midnight, Solarized, or Gruvbox — and the date format. Change these on your laptop and your phone keeps whatever you set there. **Navigation** lets you reorder or hide the tabs in the sidebar. A hidden tab is only hidden from the rail; it stays reachable by its URL, so nothing you turn off becomes unreachable. ## Workspace settings These apply instance-wide. Change one and it changes for every browser pointing at the same dashboard. **Timezone** decides how transactions bucket into days and months, so it shapes every total on every page. Changing it later is safe, but past rows keep the day grouping they were first filed under — new transactions bucket in the new zone while old ones stay where they were. **Features** is where you turn the tax organizer on or off; see [tax receipts](/using/tax/) for what that adds and hides. **Intelligence** is the section to read carefully, because it governs whether AI runs at all and, when it does, where your data goes. It has four parts: - **Connections** — where you attach a provider. Add an OpenAI API key, which is tested before it saves; sign in to ChatGPT for Codex; or let Tidings detect the Claude Code and Gemini command-line tools, which authenticate through their own tools rather than a key you paste here. - **Task routing** — the provider, model, and reasoning effort to use per task: daily summaries, monthly briefings, categorization and email rescue, and document parsing. Each task can point at a different provider. - **Briefing memo** — standing free-text context, up to 2000 characters, injected into every monthly briefing. Use it to tell briefings something the numbers cannot show. - **AI consents** — four independent switches that decide whether AI touches your data, each stating exactly what it sends: the merchant name and amount for categorization; the subject and body of an email no parser can read, for email rescue; a statement's text for statement parsing; and receipt content for receipts. The two email switches come on when you add an OpenAI key and can be turned back off; the two document switches stay off until you enable them — a key alone never turns them on. *Figure: your inbox, parser, journal, and dashboard all live on your machine. Four consents — categorization (merchant and amount), email rescue (subject and body of an unparsed email), statement parsing (statement text), and receipt parsing (receipt content) — are the only crossings to an AI provider, and nothing crosses until the matching consent is on. The two email consents come on with an OpenAI key; the two document consents only by hand.* > **note** > > Consents and task routing are two separate layers. A consent decides *whether* AI runs for a task; task routing decides *which* provider runs it. With every consent off — the default — no data leaves for AI regardless of what routing is set to. **Backup** downloads a single zip holding every transaction field, including the original email bodies — treat that file as sensitive, since it is a full copy of your data. Restore previews the incoming rows as new versus duplicate before it commits anything, so an import never silently doubles your history. See [backup and restore](/backup-and-restore/) for the operational walkthrough and the [configuration reference](/configuration/) for the underlying keys. If AWS credentials are already present on the machine, Backup also shows an **S3 backup** section — a bucket and optional prefix, a Verify button, an enable toggle, and a status row. Turned on, it mirrors your receipt attachments and statement PDFs (the files the backup zip leaves out) to a bucket you own, syncing hourly and restorable from the command line. The section stays hidden without AWS credentials. See [S3 backup](/s3-backup/) for setup, the minimal IAM policy, and the restore walkthrough. ## Account settings These control access to the instance. **Password** gates the dashboard and must be at least eight characters. It guards the browser session only — agents and scripts keep authenticating with bearer tokens instead, covered in [agent access](/agent-access/). A local-development bypass exists and is labelled as such; it is for local development only, never a deployed instance. **Sessions** lets you sign out this device, or sign out everywhere — other devices are signed out on their next request. **Activity** is the write journal: every change that lands through the dashboard or the API is recorded with who made it — a named agent token, your browser session, or this device before a password is set. Each entry shows what changed, most entries can be reverted in place, and a revert is recorded as a new entry rather than an erasure. Entries are kept for 90 days. The feed does its real work once agents hold bearer tokens; [agent access](/agent-access/) covers the full contract. **System** holds the demo-mode toggle, which loads the sample dataset, and the storage backend status, which tells you whether you are on SQLite or DynamoDB. --- # Configuration Source: https://docs.gettidings.com/configuration.md > Every data/config.json key: type, default, behavior, and what lives in .env instead. Every runtime setting lives in one file: `data/config.json`. The app creates it on first boot with auto-detected defaults and rewrites it whenever you change a setting in the dashboard (Settings) or via the API (`PUT /api/v1/config`). You can also edit it by hand while the app is stopped. The schema is defined by `AppConfig` in [`src/finance/app_config.py`](https://github.com/tvhahn/tidings/blob/main/src/finance/app_config.py) — if this table and the code ever disagree, the code wins (and a docs fix is welcome). Release stability: pre-1.0, keys may be renamed or removed between minor versions. [`releases.md`](https://github.com/tvhahn/tidings/blob/main/docs/guides/releases.md) asks every release to call out config changes in the changelog. ## Keys | Key | Type | Default | What it does | |-----|------|---------|--------------| | `storage` | `"sqlite"` \| `"dynamodb"` | `"sqlite"` | Storage backend. SQLite uses `data/finance.db`. DynamoDB is an explicit opt-in for the AWS path and is never auto-selected — set it here if you run the Lambda stack. Region comes from the `AWS_REGION` env var (falls back to `us-west-2`). | | `demo_mode` | bool | `true` on first run | Serves the seeded demo database (`data/demo.db`) instead of your real data, and blocks uploads/imports. The dashboard shows a demo banner. Flip to `false` when you wire up real email ingestion. | | `user_id` | string | `"default"` | Partition owner for budget/override records. Single-user app — leave it alone unless you know why you're changing it. | | `timezone` | IANA zone name | `"America/Los_Angeles"` | Drives day/month bucketing, "latest transaction" age math, and the daily-summary schedule. Set it to your zone (e.g. `"Europe/Berlin"`); invalid names fall back to Pacific with a logged warning. | | `ai_categorization_enabled` | bool | auto: `true` iff `OPENAI_API_KEY` is set | Master switch for AI transaction categorization. Off, every new transaction lands as miscellaneous for manual categorization. | | `ai_extraction_enabled` | bool | auto: `true` iff `OPENAI_API_KEY` is set | Consent to rescue unreadable emails with AI: when no parser can read an email, its full body is sent to the configured AI provider for a validated extraction. Distinct from `ai_categorization_enabled` — either can be on without the other. Off, unreadable emails are captured to Needs review without any AI call (no email body reaches the AI, not even the relevance classifier). Absent from an existing config, it preserves the prior consent: it inherits a persisted `ai_categorization_enabled` (which gated extraction before the two were split), falling back to the key check only when neither AI key was ever persisted. | | `ai_statement_parsing_enabled` | bool | `false` | Consent to parse statement PDFs with AI: when no built-in bank parser can read an uploaded statement, its extracted text is sent to the provider in `document_parsing_provider` and the reply is validated — every amount must appear verbatim in the PDF text — before anything is saved. Never auto-enables (a statement is the most sensitive document the app touches), so it is always an explicit opt-in from Settings → Intelligence. | | `ai_receipt_parsing_enabled` | bool | `false` | Consent to parse receipt attachments with AI: sends a receipt photo or PDF to the provider in `document_parsing_provider`, only when you ask (the parse action in the attach dialog). The reply is validated fail-closed and amounts are checked against your transactions before anything is linked. Never auto-enables — always an explicit opt-in from Settings → Intelligence. | | `s3_backup_enabled` | bool | `false` | Master switch for mirroring receipt/document attachments (`data/raw/attachments/`) and statement PDFs (`data/raw/statements/`) to a user-owned S3 bucket. Sync runs only when this is on, `s3_backup_bucket` is set, and AWS credentials resolve from the standard boto3 chain. Never auto-enables — always an explicit opt-in from Settings → Backup. See [`s3-backup.md`](/s3-backup/). | | `s3_backup_bucket` | string \| null | `null` | The user-owned bucket the S3 backup mirrors to. `null` means no bucket configured (sync stays idle). The bucket is reached in the region from `AWS_REGION`/`AWS_DEFAULT_REGION` (falls back to `us-west-2`). Set and cleared from Settings → Backup. | | `s3_backup_prefix` | string \| null | `null` | Optional key prefix inside `s3_backup_bucket`. `null` (or empty) writes to the bucket root; a value like `"tidings"` nests the backup under `tidings/attachments/…`, `tidings/statements/…`, and `tidings/manifest.json`. The backup only ever touches keys under those three names. | | `tax_tracking_enabled` | bool | `true` | Whether tax receipt tracking is active. When on, the Tax receipts workspace tab, its `/tax` route, and the per-row "Flag as tax item" control on the Transactions and Journal pages are all available. Turning it off (Settings → Features) hides all three; existing tax flags are preserved and reappear if re-enabled. Defaults on so upgrading users keep their tax flags. | | `daily_summary_provider` | `"openai"` \| `"claude_cli"` \| `"codex"` \| `"gemini_cli"` \| `"disabled"` | `"disabled"` | Which AI provider writes journal daily summaries. The CLI providers shell out to a locally installed, authenticated CLI; `openai` uses the API key. | | `insights_provider` | same enum | `"disabled"` | Which AI provider writes monthly insights briefings. | | `categorization_provider` | `"openai"` \| `"codex"` \| `"disabled"` | auto: `"openai"` iff `OPENAI_API_KEY` is set | Which provider runs AI categorization and email rescue (still gated by the two consents above). Codex is never auto-selected — it needs an explicit pick plus a ChatGPT login. | | `document_parsing_provider` | same enum as `daily_summary_provider` | `"disabled"` | Which provider parses statement PDFs and receipt attachments (still gated by the two parsing consents above). | | `daily_summary_model`, `insights_model`, `categorization_model`, `document_parsing_model` | string \| null | `null` | Per-feature model override. `null` means the provider's default: `gpt-5.4-nano` on the `openai` path, the CLI's own default otherwise (`sonnet` for Claude Code). Any model your provider serves is accepted — e.g. `gpt-5.6-sol` / `gpt-5.6-terra` / `gpt-5.6-luna`. | | `daily_summary_reasoning_effort`, `insights_reasoning_effort`, `categorization_reasoning_effort`, `document_parsing_reasoning_effort` | `"none"` \| `"minimal"` \| `"low"` \| `"medium"` \| `"high"` \| `"xhigh"` \| `"max"` \| null | `null` | Per-feature reasoning-effort override; `null` uses the provider default (no flag sent). Support varies by provider (OpenAI API none–xhigh, Codex low–xhigh, Claude Code low–max); an unsupported combination surfaces as a provider error. Gemini ignores model and effort overrides. | | `insights_user_memo` | string \| null (≤ 2000 chars) | `null` | Standing free-text context injected into every AI briefing — household facts, known annual bills, savings goals, anything the briefing should already know. Edit it in Settings → Intelligence. `null` (or empty) means no memo. Values over 2000 characters are rejected with a 422. | | `enable_daily_summaries` | bool | `true` | Whether the in-process scheduler generates journal daily summaries at all (only takes effect when `daily_summary_provider` is not `"disabled"`). | | `daily_summary_schedule_time` | `"HH:MM"` (24h) | `"19:00"` | When the daily summary runs, evaluated in `timezone`. Invalid values fall back to the default with a logged warning. | | `agent_tokens` | list | `[]` | Bearer-token records for agent/API access — sha256 hashes only, never raw tokens. Manage via `make agent-token` / `make agent-token-show` / `make agent-token-revoke`; don't hand-edit. See [`agent-access.md`](/agent-access/). | | `app_password_hash` | string \| null | absent | Argon2id hash of the dashboard password. Absent means TOFU mode: the API allows unauthenticated access and the dashboard nags you to set a password. Set it from Settings → Password. To reset a forgotten password, stop the app and remove this key from `data/config.json` — the dashboard returns to TOFU mode. | | `session_version` | int | `0` | Bumped by "sign out everywhere" — invalidates every existing session cookie without rotating the secret. | | `session_signing_secret` | 64-char hex | auto-generated on first use | HMAC key for session cookies. Treat it like a password; rotate only if it leaks (rotation signs everyone out). | | `auth_bypass_for_dev` | bool | `false` | Dev-only escape hatch: skips cookie auth on `/api/v1/*` even when a password is set (bearer enforcement still applies). Never enable on an instance that binds beyond localhost — anyone on the network gets full read/write access. | ## What does *not* live here - **Secrets and infrastructure env vars** — `OPENAI_API_KEY`, `IMAP_*`, notification provider settings (`NOTIFICATION_URL`, `NOTIFICATION_PROVIDER`, `TWILIO_*`, `SNS_TOPIC_ARN`), `AWS_REGION`, `SERVE_FRONTEND`, `CORS_ALLOWED_ORIGINS` — come from the environment / `.env`. [`.env.example`](https://github.com/tvhahn/tidings/blob/main/.env.example) is the complete, commented list. - **Budgets, category overrides, merchant aliases** — stored in the database, managed in the dashboard. - **User mappings** (`ForwardedTo` → `UserId`) — live in `data/config/user_mappings.csv`. They load once into an in-memory cache with no TTL (unlike the 5-minute config caches), so edits to the CSV only take effect after a server restart. ## First-run behavior When `data/config.json` is absent, the app writes one with `storage: "sqlite"` and `demo_mode: true` — even when AWS credentials are present on the machine. `ai_categorization_enabled`, `ai_extraction_enabled`, and `categorization_provider` all auto-detect from `OPENAI_API_KEY`. After that first write, the file is authoritative: auto-detection never overrides a persisted value. ## Upgrading from the single-provider key Configs written before per-feature routing carry one `summary_provider` key. On the first boot after upgrading it is migrated automatically: daily summaries, insights, and document parsing inherit its value; `categorization_provider` becomes `"codex"` when the legacy value was codex, otherwise the API-key auto-detect above. The legacy key is then removed from the file. Keys already present are never overwritten by the migration. --- # Backup and restore Source: https://docs.gettidings.com/backup-and-restore.md > Back up the data directory, restore it, and leave cleanly. Everything Tidings knows lives in one place: the `data/` directory — `config.json` plus the SQLite database, and the receipt files under `data/raw/attachments/` if you use receipts. A backup is a copy of that data; a restore is putting it back. This guide covers both, plus leaving cleanly. Under the default Docker stack, `data/` is a **named volume** (`docker volume ls` shows it as `_finance_data`, e.g. `tidings_finance_data`) — it is *not* the `data/` folder in your checkout. Copying the repo directory backs up nothing. ## Back up ### The backup zip (recommended) **Settings → Backup → Download backup** produces `finance-backup-.zip`: every transaction plus your configuration (categories, overrides, merchant aliases, budgets, and the Needs review queue). It is the portable format the restore path understands, and it works the same on SQLite and DynamoDB. The same thing over the API, for a cron job or an agent: ```bash curl -sf -X POST http://localhost:8000/api/v1/data/export \ -H "Authorization: Bearer $FINANCE_API_TOKEN" \ -o "tidings-backup-$(date +%F).zip" ``` Bearer tokens are covered in the [agent access guide](/agent-access/). Backup is disabled in demo mode — there is nothing of yours to back up yet. **What the zip does not carry:** receipt and statement attachment *files*, and instance settings (timezone, dashboard password, AI toggles — those live in `data/config.json` and take a minute to re-set on a fresh install). The transactions and their links are in the zip; the images and PDFs themselves live only in the volume. If you use receipts, take a volume copy (below) as well. If you already use AWS, you can instead mirror those files to a bucket you own with [S3 backup](/s3-backup/) — the durability answer for the attachment and statement files the zip leaves out. ### The volume copy (full fidelity) A byte-level copy of the whole volume — databases, config, attachments, saved briefings. Stop the stack first so SQLite isn't mid-write: ```bash docker compose stop docker run --rm \ -v tidings_finance_data:/data \ -v "$(pwd)/backup":/out \ alpine cp -r /data /out docker compose start ``` Substitute your volume name from `docker volume ls`. The result in `./backup/data/` is the entire state of your install. ## Restore ### From the backup zip **Settings → Backup → Restore from backup.** Upload the zip (or a plain transactions CSV from the Search tab's export). Nothing is written until you confirm: a dry-run preview first shows how many rows are new versus duplicates, and you choose whether duplicates are skipped and whether the zip's configuration is applied. This is also the migration path — the zip restores into a fresh install on another machine, or across the SQLite/DynamoDB boundary. ### From a volume copy Onto a fresh install: ```bash docker compose up -d # first boot creates the volume docker compose stop docker run --rm \ -v tidings_finance_data:/data \ -v "$(pwd)/backup/data":/from \ alpine sh -c "rm -rf /data/* && cp -r /from/. /data/" docker compose start ``` Open the dashboard and check the sync dot beside the month picker; the [troubleshooting guide](/troubleshooting/) covers anything that looks off. ## A reasonable habit Download a backup zip before every upgrade (the [upgrading guide](/upgrading/) starts there) and on some calendar rhythm that matches how much re-entry you can tolerate losing. One zip in your existing backup system — a synced folder, an external drive — is enough; this is a single household's journal, not a database fleet. ## Leaving Tidings Everything is yours and everything is local, so leaving is short: 1. Download a backup zip (transactions as portable JSON/CSV) if you want the history. 2. `docker compose down -v` — stops the stack and deletes the volume. 3. Delete the cloned repo directory. 4. Revoke the Gmail App Password ([myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords)) and turn off the bank-alert forwarding rules you created during [email setup](/self-hosting/email/). No accounts to close, nothing left behind. --- # Upgrading Source: https://docs.gettidings.com/upgrading.md > Pull-and-restart upgrades, automatic migrations, and rolling back. Tidings is **pre-1.0**: a minor version may change APIs or the database schema. The upgrade itself is three commands; the care is in the two steps around them. ## Check where you are The health endpoint shows the running version: ```bash curl -s http://localhost:8000/api/v1/health | jq .version ``` ## Upgrade 1. **Read the [CHANGELOG](https://github.com/tvhahn/tidings/blob/main/CHANGELOG.md) first.** Anything that changes a schema, a config key, or a compose default is called out per release — that's the contract the [release ritual](https://github.com/tvhahn/tidings/blob/main/docs/guides/releases.md) enforces. 2. **Take a backup.** One zip from Settings → Backup, per the [backup guide](/backup-and-restore/). Pre-1.0 this is the step to skip least. 3. **Pull and restart:** ```bash git pull # compose file and docs move with releases docker compose pull docker compose up -d ``` Database migrations run automatically on startup — the app creates missing tables and applies pending migrations before serving requests. There is no separate migrate command to run. Then open the dashboard and hover the sync dot beside the month picker: a recent sync time means the poller re-attached and the schema settled. Anything off, start with the [troubleshooting guide](/troubleshooting/). ## Roll back Images are published per version, so rolling back is pinning a tag: 1. `docker compose down` (without `-v` — keep your data). 2. Edit the two `image:` lines in `docker-compose.yml` from `:latest` to the previous version, e.g. `ghcr.io/tvhahn/tidings:v0.1.0` and `ghcr.io/tvhahn/tidings-imap-poller:v0.1.0` (the committed file spells the owner with the `GITHUB_REPOSITORY_OWNER` compose variable — only the tag needs to change). 3. `docker compose up -d`. If the newer version already migrated the schema, the older code may not read it — that's what the pre-upgrade backup is for: restore it per the [backup guide](/backup-and-restore/) and you're back exactly where you were. ## Following along between releases `docker compose pull` follows `:latest`, which moves with tagged releases. Running from `main` (building locally after `git pull`) is the explicit bleeding-edge opt-in — any commit may break an install, and the changelog's `[Unreleased]` section is the only warning you get. --- # Troubleshooting Source: https://docs.gettidings.com/troubleshooting.md > Diagnose the failures self-hosters actually hit, most common first. The short version: the sync dot beside the month picker tells you *whether* something is wrong, `docker compose logs` tells you *what*. This page collects the failures self-hosters actually hit, most common first. ## First checks ```bash docker compose ps # both services up? curl -s http://localhost:8000/api/v1/health | jq . docker compose logs --tail=50 finance # the API + dashboard container docker compose logs --tail=50 imap-poller # the email poller sidecar ``` The health endpoint is unauthenticated and returns the same state as the sync dot: | Status | Meaning | |---|---| | `ok` | Poller checked in within 5 minutes (or IMAP isn't configured), nothing quarantined. | | `degraded` | Last poll 5–30 minutes ago, an email failed to parse in the last 7 days — the early signal that a bank changed its template — **or** AI categorization is failing (provider, quota, or auth errors). | | `stale` | No poll in over 30 minutes, or no new transaction in 14 days. | In the dashboard, the dot turns red and pulses only on `stale`; a `degraded` poller shows up in the dot's hover text rather than as a separate color. `degraded` and `stale` describe the *email pipeline*, not the API — the dashboard keeps working either way. ## Transactions stopped appearing The most common failure in normal operation, and it's usually not your setup: a bank changed its email template. 1. The dot goes amber; unparsed alerts collect under **Needs review** rather than being dropped. 2. `docker compose logs imap-poller` shows `Failed to parse email`. 3. Open a [parser-broken issue](https://github.com/tvhahn/tidings/issues/new?template=parser_broken.md) with a redacted email body — or fix it yourself with the [add-a-parser guide](/add-a-parser/), which is exactly the tutorial for this. Meanwhile, entries recovered by AI extraction (if enabled) and manual entries from Needs review keep the journal complete. ## The IMAP poller can't connect Gmail sign-in failures, `[AUTHENTICATIONFAILED]`, repeated reconnects: - Gmail needs an **App Password**, not your account password, and 2-Step Verification has to be on first. - The poller backs off automatically (5s doubling to 5 minutes) and recovers on its own once credentials are right — no restart needed after fixing `.env`, but `docker compose restart imap-poller` applies it immediately. - The [email setup guide](/self-hosting/email/#troubleshooting) has the per-symptom detail, including IMAP being disabled on the account. ## The dashboard won't load - **Port 8000 already in use** — uncomment the `ports: !override` snippet in `docker-compose.override.yml` (copy the file from its `.example` if you don't have one) and pick a free host port, e.g. `"8001:8000"`. Then open `http://localhost:8001`. Prefer the override to editing `docker-compose.yml` directly — the override is gitignored, so upgrades via `git pull` never conflict with your port choice. - **Container restarting or exited** — `docker compose ps`, then `docker compose logs finance` and read the last screen; the failing line is almost always in it. - **`exec format error` on Raspberry Pi / Apple Silicon** — images are multi-arch (amd64 + arm64), so this is usually a stale cache: `docker compose pull`, or build locally. ## Permission errors on `data/` The named volume is owned by the container user. This appears after switching between a bind mount and the named volume. The reset is `docker compose down -v` — **this deletes the volume and your data**, so take a [backup](/backup-and-restore/) first if there's anything real in it. ## Forgotten dashboard password Stop the app, remove the `app_password_hash` key from `data/config.json`, start it again — the dashboard returns to trust-on-first-use and prompts for a new password. Under the Docker stack the file lives in the volume: ```bash docker compose exec finance python -c \ "from src.finance.app_config import update_config; update_config({'app_password_hash': None})" docker compose restart finance ``` ## Notifications aren't arriving - With no provider configured, notifications go to the logs only — look for `NOTIFY [Transaction]:` lines in `docker compose logs finance`. That's the fallback, not a bug. - More than one provider's variables set? Auto-detection has a precedence order; pin it with `NOTIFICATION_PROVIDER`. The [notifications guide](/notifications/#troubleshooting) walks each provider's failure modes. ## Demo mode won't go away Demo mode is a config key, not an env var: set `demo_mode: false` in `data/config.json` (or from Settings), then restart. Your real data lives in `data/finance.db`; the seeded demo stays in `data/demo.db` and never mixes. ## Still stuck - Health state, last poll, and version in one line: `curl -s http://localhost:8000/api/v1/health | jq .` - "How do I…" questions go to [GitHub Discussions](https://github.com/tvhahn/tidings/discussions); reproducible bugs go to [Issues](https://github.com/tvhahn/tidings/issues) — include the health JSON and the last ~20 log lines, redacted. --- # FAQ Source: https://docs.gettidings.com/faq.md > Short answers on privacy, bank support, costs, and importing history. Short answers, with links into the guides for depth. ## Does Tidings have my banking password? No. Tidings never talks to your bank. It reads the alert emails your bank already sends you, from a Gmail inbox you control, using an App Password scoped to that inbox. There is no Plaid, no screen scraping, no bank credential anywhere in the system. ## Is it OK to forward my own bank emails? This is your own mail, about your own accounts, forwarded by you to another inbox you own, processed on your own hardware. That is a long way from the things bank terms of service actually prohibit — sharing credentials, granting third parties account access, screen scraping. Email forwarding of alerts the bank chose to send you is not credential sharing, and no third party is involved. Two caveats. First, alert emails contain personal financial information, so the inbox that receives them is worth protecting like the account itself — a dedicated Gmail account with 2-Step Verification, which is what the [email setup guide](/self-hosting/email/) builds. Second, this is a description of how Tidings works, not legal advice; if your bank's alert terms say something unusual, they win. For Canadian users wondering about PIPEDA: it governs how *organizations* handle your personal information. Running Tidings yourself, for yourself, on your own machine is personal use of your own data — the situation the law exists to protect, not restrict. ## What leaves my machine? Without an AI key, nothing. Add an OpenAI key and two switches come on, both yours to turn back off under [Settings → Intelligence](/using/settings/): categorization, which sends the amount and merchant name of each parsed transaction, and email rescue, which sends the subject and body of an email no parser could read. Every other AI feature — the monthly briefing, receipt reading, statement parsing — is a separate opt-in that a key never turns on. ## Which banks work? Five ship today: | Institution | Email alerts | PDF statements | Notes | |---|---|---|---| | RBC | Yes | Yes (chequing) | Purchase, withdrawal, e-transfer | | CIBC | Yes | No | Purchase, preauth payment | | MBNA | Yes | No | Credit card purchase | | Simplii | Yes | Yes (chequing) | E-transfer (Interac sender) | | PC Financial | Yes | No | Money Account purchase alerts | Alerts from any other bank land in **Needs review** instead of being dropped: AI extraction (if enabled) can read them, you can enter them manually, and the [add-a-parser guide](/add-a-parser/) turns them into a proper parser — the most useful contribution to the project. ## How is Tidings different from Firefly III or Actual Budget? They solve adjacent problems in different ways. Firefly III is a full double-entry finance manager; Actual Budget is a local-first envelope budgeting app. Both are built around bank sync, file imports, or entering transactions yourself. Tidings reads the alert emails your bank already sends — no bank credentials, no aggregator, no manual entry — and turns them into a daily journal. Pick those projects for accounting or strict budgeting; pick Tidings for a spending record that maintains itself. Both projects are solid; this is a difference in shape, not a ranking. ## Can I import my history? Two paths. Bank PDF statements upload from the Statements page (built-in parsers for RBC and Simplii chequing; AI statement parsing is an opt-in for others). And the restore path accepts a transactions CSV, previewed before anything is written — see [backup and restore](/backup-and-restore/). ## Why a dedicated Gmail account? The poller reads the inbox you point it at, and an App Password grants mail access — both are reasons to keep bank alerts in their own account rather than your personal one. It also gives every bank one stable forwarding target. The [email setup guide](/self-hosting/email/) walks through creating it. ## Can two people use it? One household, yes — point both people's bank alerts at the same forwarding inbox and the journal is shared. Separate logins, per-person views, or multi-tenant hosting are out of scope, and staying out — see [ROADMAP — Not doing](https://github.com/tvhahn/tidings/blob/main/ROADMAP.md#not-doing). ## Can I put it on the internet? It's built for loopback or a network you control — Tailscale is the recommended way to reach it away from home. There's no rate limiting and TLS is a reverse proxy's job, so the open internet is the wrong place for it. The [agent access guide](/agent-access/#exposing-the-api-on-a-lan) has the exposure checklist. ## What does it cost to run? The software is free (MIT). Self-hosting costs whatever your machine already costs — a Raspberry Pi is plenty. Optional extras: OpenAI categorization (cents per month at household volume), ntfy notifications (free), the AWS path (see the [cost analysis](https://github.com/tvhahn/tidings/blob/main/docs/guides/dynamodb-cost-analysis.md)). ## Is my data locked in? It's a SQLite file on your disk. Settings → Backup exports your transactions, categories, rules, and budgets as a zip; the Search tab exports CSV. Delete the volume and the repo and Tidings is gone — [leaving Tidings](/backup-and-restore/#leaving-tidings) is four steps. ## Do I need AWS? No. The default is Docker Compose on your own machine with SQLite. The AWS serverless path exists for people who already live there — same parsers, different plumbing. Run the [Docker path](/self-hosting/docker/) unless you know you want Lambda. --- # Email-to-S3 setup Source: https://docs.gettidings.com/self-hosting/email-to-s3.md > Route bank alert emails into the S3 bucket the AWS Lambda watches, using SES, a verified domain, and a receipt rule. Route bank alert emails into the S3 bucket the AWS Lambda watches, using SES, a verified domain, and a receipt rule. :::note[Prerequisites] SES inbound only works on a domain you control. You will need to add MX and DKIM records to its DNS — that means owning a domain at a registrar, not just an email address. If you do not own a domain, the [Docker + IMAP path](/self-hosting/docker/) does the same job without DNS. Most users start there. ::: **For AWS users who own a registered domain and want bank alerts to arrive via email at that domain.** By the end, you'll have an S3 bucket receiving inbound emails from your bank, ready for the [Lambda parser](/self-hosting/aws/) to process. The AWS path needs your bank alert emails to land in an S3 bucket before any of the parser scripts can run. The Lambda function is triggered by S3 object-created events, so if nothing is writing to the bucket, nothing parses. The IMAP path on Docker does not have this prerequisite — it polls Gmail (or any IMAP mailbox) directly and skips S3 entirely. This page covers the AWS path specifically — the configuration steps between deploying the Lambda and transactions showing up in the dashboard. If you do not already use AWS, or want the shortest path to a working setup, the [Docker self-hosting guide](/self-hosting/docker/) and the [Email setup guide](/self-hosting/email/) cover the IMAP path end to end. ## The shape of the pipeline Before stepping through the configuration, it helps to hold the full path in mind: ```text bank → Gmail (or direct) → forwarder → SES inbound → S3 → Lambda → DynamoDB → dashboard ``` The work on this page covers everything from **forwarder** through **S3**. The Lambda, DynamoDB table, and dashboard are deployed separately by the scripts in `docker/email_parsing/` and the rest of the repo. SES is the only piece that needs a verified domain and DNS records — everything to the right of S3 is plain AWS resources. The reason this is more involved than IMAP polling is that SES inbound is a "real" mail server: it has to convince other mail servers (Gmail, your bank) that it owns a domain, has a published MX record, and accepts mail at known addresses. That is the cost of running on AWS infrastructure rather than logging into a mailbox you already own. ## What you need before you start ### Hard requirements - **A registered domain you control DNS for.** Any registrar works — Route 53 is convenient if you already use AWS, but the rest of this guide assumes you can add MX and CNAME records wherever your domain currently lives. - **An AWS account** where you can create S3 buckets, configure SES rule sets, and attach IAM policies. This is the same account you will deploy the Lambda into. ### Recommended choices - **An SES inbound region.** SES inbound is region-restricted, but the supported list has grown over time — the AWS console shows the live list when you open the SES email-receiving page, and the canonical reference is the [Email Receiving endpoints table](https://docs.aws.amazon.com/general/latest/gr/ses.html#ses_inbound_endpoints) in the AWS General Reference. `us-east-1`, `us-west-2`, and `eu-west-1` are the long-standing safe defaults if you have no other constraint. Whichever region you pick, the S3 bucket and the Lambda function must live in the same region. Cross-region routing is not supported for SES inbound rules. ## 1. Create the S3 bucket In the AWS S3 console, create a new bucket: - **Name:** something distinctive, since bucket names are globally unique. `tidings-inbound-` works as a naming pattern. - **Region:** the same region you picked for SES inbound. - **Block public access:** keep all four block settings on. SES writes to the bucket via a service principal, not via public access. - **Default encryption:** SSE-S3 is enough. SSE-KMS works too but adds an extra IAM grant on the SES side — skip it unless you have a policy reason to use KMS. - **Versioning:** off, unless you want to keep historical copies of inbound mail. The parser is idempotent, so versioning mostly costs storage. You do not need to pre-create any prefixes. SES will write objects directly to the bucket root (or to a prefix you specify in the receipt rule, see step 5). ## 2. Verify your domain in SES Open the SES console in your chosen region and go to **Verified identities** → **Create identity** → **Domain**. AWS's [Easy DKIM walkthrough](https://docs.aws.amazon.com/ses/latest/dg/send-email-authentication-dkim-easy.html) covers the same steps with screenshots. - Enter the apex domain you control (`yourdomain.com`), even if you plan to receive mail at a subdomain like `mail.yourdomain.com`. - Turn on **Easy DKIM** with RSA 2048-bit keys. SES will give you three CNAME records to add to your DNS. Add the three CNAME records SES provides at your DNS host. They look like: ```text ._domainkey.yourdomain.com CNAME .dkim.amazonses.com ._domainkey.yourdomain.com CNAME .dkim.amazonses.com ._domainkey.yourdomain.com CNAME .dkim.amazonses.com ``` (In a small number of newer regions — for example `eu-south-1`, `eu-central-2`, `me-central-1` — SES generates a region-specific target like `.dkim..amazonses.com`. Use whatever values the SES console gives you.) Verification usually completes in under 30 minutes. Most DNS hosts propagate within a few minutes, and SES re-checks on a short interval. The identity status moves from **Pending** to **Verified** when DKIM is confirmed. ### A note on SPF and DMARC To receive mail, SES needs two things in your DNS: the domain **verified** (the DKIM CNAME records from step 2, which prove you own it) and an **MX record** (step 3). It does not need SPF or DMARC. Those records protect against spoofed mail being **sent from** your domain — an outbound concern, not an inbound one. Once the identity is verified, SES checks DKIM on incoming mail automatically, with no extra DNS work. If you already publish SPF or DMARC at your apex domain because you also send mail from it, those records do not interfere with SES inbound. They live at the apex; the MX record for SES lives at the subdomain. Different scopes. ## 3. Add an MX record for SES inbound Pick the address you want bank alerts to arrive at. The recommended pattern is a dedicated subdomain — `mail.yourdomain.com` or `inbound.yourdomain.com` — so SES inbound does not interfere with whatever already handles mail at the apex. AWS's reference for this step is [Publishing an MX record for Amazon SES email receiving](https://docs.aws.amazon.com/ses/latest/dg/receiving-email-mx-record.html). Add an MX record at the chosen subdomain pointing to the SES inbound endpoint for your region: ```text mail.yourdomain.com. MX 10 inbound-smtp.us-east-1.amazonaws.com. ``` The endpoint hostname follows the pattern `inbound-smtp..amazonaws.com`. Common values: | Region | Inbound endpoint | |---|---| | `us-east-1` | `inbound-smtp.us-east-1.amazonaws.com` | | `us-west-2` | `inbound-smtp.us-west-2.amazonaws.com` | | `eu-west-1` | `inbound-smtp.eu-west-1.amazonaws.com` | Priority `10` is conventional — it does not matter what number you pick as long as no other MX record at the same name has a lower number. If you ever add a backup MX, give it a higher number. DNS propagation for MX records is typically a few minutes, occasionally up to an hour. You can confirm with `dig MX mail.yourdomain.com` once it lands. The expected output: ```text ;; ANSWER SECTION: mail.yourdomain.com. 300 IN MX 10 inbound-smtp.us-east-1.amazonaws.com. ``` A note on subdomain choice: SES inbound only matches the recipient domain, not the apex. If you put the MX at `mail.yourdomain.com`, mail to `you@yourdomain.com` will not reach SES — it will go wherever the apex MX points (often nowhere). Pick a subdomain you do not already use for anything else. ## 4. Grant SES write access to the bucket SES needs explicit permission to write to your bucket. The bucket policy below allows the `ses.amazonaws.com` service principal to `PutObject`, scoped to your AWS account *and* the specific receipt rule so only that rule's SES delivery can drop mail in. This mirrors the example AWS publishes in [Giving permissions to Amazon SES for email receiving](https://docs.aws.amazon.com/ses/latest/dg/receiving-email-permissions.html#receiving-email-permissions-s3). You will fill in four values: `YOUR-BUCKET-NAME`, `YOUR-ACCOUNT-ID`, `YOUR-REGION` (the SES inbound region from step 2), and `YOUR-RULE-SET-NAME` / `YOUR-RULE-NAME` (the names you will create in step 5). :::note Steps 4 and 5 depend on each other. Either complete step 5 first to get the rule-set and rule names, or apply this bucket policy with placeholders now (which blocks SES until the names are filled in — the safer order). ::: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSESPuts", "Effect": "Allow", "Principal": { "Service": "ses.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*", "Condition": { "StringEquals": { "AWS:SourceAccount": "YOUR-ACCOUNT-ID", "AWS:SourceArn": "arn:aws:ses:YOUR-REGION:YOUR-ACCOUNT-ID:receipt-rule-set/YOUR-RULE-SET-NAME:receipt-rule/YOUR-RULE-NAME" } } } ] } ``` Apply it under **S3 console → your bucket → Permissions → Bucket policy**. The `AWS:SourceAccount` and `AWS:SourceArn` conditions are the key piece — together they implement the "confused deputy" pattern AWS recommends for cross-service access, restricting writes to a specific receipt rule in your account. (Older AWS examples used `aws:Referer` for this; AWS has since moved every SES inbound walkthrough over to `SourceAccount` + `SourceArn`, which is more precise and the current recommendation.) If you encrypted the bucket with KMS instead of SSE-S3, also grant `kms:GenerateDataKey` on the bucket's KMS key to the same service principal with the same condition keys. Most setups skip this by sticking with SSE-S3. ## 5. Create a receipt rule set In the SES console, go to **Email Receiving** → **Create rule set**. Name it something clear like `tidings-inbound`. SES only honors **one active rule set at a time**, so if you already have one, you will be adding rules to the active set rather than activating a new one. AWS's [Creating receipt rules console walkthrough](https://docs.aws.amazon.com/ses/latest/dg/receiving-email-receipt-rules-console-walkthrough.html) covers this same flow with current screenshots. Activate the rule set, then add a rule inside it: - **Recipient condition:** the address you want to receive bank alerts at. `alerts@mail.yourdomain.com` is a reasonable choice. You can list multiple recipients on the same rule. - **Action:** **Deliver to S3 bucket** (per AWS's current [Deliver to S3 bucket action](https://docs.aws.amazon.com/ses/latest/dg/receiving-email-action-s3.html)). - **S3 bucket:** the bucket from step 1. - **Object key prefix:** optional. Leaving it blank writes to the bucket root, which is what the Lambda script expects out of the box. If you set a prefix, update the Lambda S3 trigger filter accordingly. - **KMS key:** none, unless you set up KMS in step 4. - **SNS topic:** leave blank. The Lambda is wired to S3 object-created events directly, not to SNS. Save the rule and confirm the rule set is **Active**. SES marks the active set with a green badge in the console. A small gotcha: SES will not let you save a rule that writes to a bucket whose policy does not yet grant SES `PutObject`. If you skipped step 4 or got the policy wrong, the rule save fails with a permissions error rather than saving and then silently dropping mail. That is a feature — fix the bucket policy and try again. ## 6. Forward bank alerts to your SES address You now have an SES address that drops mail in S3. The remaining piece is getting bank alert emails to that address. - **Direct configuration.** Some banks let you change the alert destination address freely in their notifications settings. If yours does, point it straight at `alerts@mail.yourdomain.com` and skip Gmail entirely. - **Gmail forwarding filter.** Most Canadian banks restrict the alert address to one already on file or refuse to send to addresses on unfamiliar domains. The workaround is a Gmail filter: 1. In Gmail settings, add `alerts@mail.yourdomain.com` as a forwarding address. Gmail sends a verification message to the SES address. The message includes both a confirmation link and a numeric code; the link will not be clickable (SES is not a mail client), so retrieve the email from S3, copy the numeric code, and paste it back into Gmail. 2. Create a filter that matches each bank's `From:` addresses (RBC `@alerts.rbc.com`, CIBC `@cibc.com`, MBNA `@mbna.ca`, PC Financial `@pcfinancial.ca`, Interac `@payments.interac.ca` — Simplii alerts arrive via the Interac sender domain, not `@simplii.com`) and applies the action **Forward to alerts@mail.yourdomain.com**. The [Email setup guide](/self-hosting/email/) has the full sender reference if you need it. 3. Optionally add **Skip the inbox** so Gmail keeps the original but does not clutter your view. The Gmail filter approach is what most users end up with, since direct bank configuration is rarely permitted. ## 7. Confirm the loop works End-to-end test, in order: 1. Send a test email from any mailbox to `alerts@mail.yourdomain.com`. Use a short plain-text body. 2. Open the S3 bucket in the console. Within a few seconds, an object with a long random key should appear. The body is the raw RFC 5322 message including headers. 3. Forward (or wait for) a real bank alert email to the same address. Confirm a second object lands. 4. Trigger the Lambda manually: ```bash cd docker/email_parsing/ ./6_invoke_func.sh ``` The script invokes the function with a synthetic S3 event. Check CloudWatch Logs for the function — you should see parser output and a DynamoDB write. 5. Open the dashboard. The transaction from your test bank email should be in the journal. ## If the loop does not work If steps 2 and 3 work but step 4 produces nothing, the S3 event notification on the bucket is probably not wired to the Lambda. That wiring lives in `4_establish_lambda_func.sh` and is separate from the bucket policy in step 4 of this guide. To inspect it, open **S3 console → your bucket → Properties → Event notifications** and confirm there is one entry that triggers the Lambda on **All object create events**. If the entry is missing, re-run the deploy script — it is idempotent and will repair the wiring. The Lambda execution role also needs `s3:GetObject` on the bucket so the function can read each message body when triggered. The deploy scripts attach this policy automatically. If you wrote your own deploy or modified the script, confirm the role has that permission, or every invocation will fail with `AccessDenied` in CloudWatch. If the Lambda runs but the dashboard is empty, the parser ran but found no transaction. CloudWatch logs the bank it tried to match. Common cases: - The bank is not in `src/finance/parsers/` yet. Five Canadian banks ship today (RBC, CIBC, MBNA, Simplii, PC Financial). Other institutions are added by contribution. - The email is from the bank but is a marketing message, not a transaction alert. The parser logs `not a transaction email` and exits. - The forwarder rewrote the `From:` header, so the parser sees Gmail's address rather than the bank's. The repo has logic to handle Gmail's `X-Forwarded-For` header, but custom forwarders may behave differently. Inspect the raw S3 object to see what headers actually arrived. ## What comes next When this page is finished, you should be able to: - Receive a test email at `alerts@mail.yourdomain.com` and see an object appear in the S3 bucket within a few seconds. - See SES list the domain identity as **Verified** and the rule set as **Active**. - Run `dig MX mail.yourdomain.com` from any machine and get back the SES inbound endpoint for your region. - Forward (or wait for) a real bank alert and see a transaction land in the dashboard within the Lambda's typical invocation latency (a few seconds plus OpenAI categorization time). Once these steps are complete, email-to-S3 is ready. The next step is to deploy the Lambda function that processes those emails. Continue to [AWS deployment](/self-hosting/aws/). ## Why this is more complex than IMAP The AWS path scales to zero, runs on managed infrastructure, and costs pennies per month at typical email volumes — but the inbound path requires a domain you control, SES verification, an MX record, a bucket policy, and a receipt rule set. None of that is technically hard, but it is six moving parts that have to line up before any transaction reaches the parser. The IMAP path on Docker needs a Gmail account and a forwarding filter. There is no domain, no DNS, no SES, no bucket policy. For most users, that is the right starting point — see the [Docker guide](/self-hosting/docker/) and [Email setup](/self-hosting/email/). Move to AWS once the parser is doing useful work and you want to stop running a host. ## Troubleshooting - **SES verification stuck on Pending.** Re-check the three DKIM CNAME records at your DNS host. Most failures here are a copy-paste error in the token names or a DNS host that silently appended your apex domain to the record (some hosts double up, producing `._domainkey.yourdomain.com.yourdomain.com`). Use `dig CNAME ._domainkey.yourdomain.com` to confirm what is actually published. - **Test email never appears in S3.** Confirm the rule set is **Active**, not only created. Confirm the recipient condition exactly matches the address you sent to (case-insensitive on the local part, but typos count). Confirm the bucket policy is attached and the `AWS:SourceAccount` matches your account ID and the `AWS:SourceArn` matches your active rule set + rule name. SES will silently drop mail if the bucket write fails — most often because one of those two values has a typo or the rule has been renamed since the policy was applied. - **MX lookup fails.** Run `dig MX mail.yourdomain.com` from a machine outside your DNS host's caching layer. If it returns nothing, the record was not saved or has not propagated. If it returns the wrong endpoint, you typed the region wrong. - **Object lands in S3 but Lambda does not run.** The S3 → Lambda event notification is a separate piece of glue, configured by `4_establish_lambda_func.sh`. Re-run that script, or open **S3 console → bucket → Properties → Event notifications** and confirm the function is wired to **All object create events**. - **Lambda runs but no transaction shows up.** The function ran but did not parse the email. Check CloudWatch Logs for the function; the parser logs the `From:` address it tried to match. If the bank is not in `src/finance/parsers/`, the email is dropped on purpose — parsers are added by contribution. - **Mail bounces back at Gmail with "address not found."** The MX record either has not propagated, points to the wrong region's endpoint, or sits at the apex when SES is configured for a subdomain. Run `dig MX ` and confirm the answer matches the SES inbound endpoint for your region. - **Mail arrives in S3 but the object is huge and the parser logs a memory error.** Some banks include base64-encoded PDF attachments on alert emails. The Lambda's default memory of 128 MB is enough for typical alerts but not for multi-MB attachments. Bump memory to 512 MB on the Lambda function configuration if this is a recurring issue. The parser ignores attachments by design — the body text is what carries the transaction. ## After setup: operations (optional) The next two sections cover ongoing operational concerns — bucket retention and monthly cost — rather than setup steps. They're optional reading; skip ahead to [AWS deployment](/self-hosting/aws/) if you want to keep momentum. ### Bucket lifecycle and retention Once mail is flowing, the S3 bucket grows by the size of every email SES drops in. At typical alert volumes — a handful per day — the bucket grows by a few MB per month. Over years, that adds up to noticeable but inexpensive storage. Two reasonable retention strategies: - **Keep everything.** Storage is cheap. The raw S3 objects are useful for replaying parser changes against historical data — if a parser bug is fixed, you can re-invoke the Lambda over old objects and rebuild the corrected transaction history. - **Expire after 90 days.** Add an S3 Lifecycle rule that deletes objects 90 days after creation. The Lambda has already extracted the transaction into DynamoDB by then; the raw email is no longer load-bearing. This keeps the bucket bounded. Pick based on how much you trust the parsers. If you are still tuning a new bank parser, keep everything. If the setup has been stable for a while, the 90-day expiry suffices. ### Costs at typical volumes SES inbound charges $0.10 per 1,000 emails received, plus $0.09 per 1,000 incoming 256 KB mail chunks, plus standard S3 storage. For typical bank alerts (well under 256 KB each) the chunk fee rounds to zero. For a single user with two or three banks, that is well under $0.10 per month — the round-up to a billable cent is the dominant cost. The Lambda function tier covers the parser invocations free at this volume. The DynamoDB on-demand table costs a few cents per month at most. Most of the AWS bill, if there is one, is the existing S3 minimums on the account. --- # AWS deployment Source: https://docs.gettidings.com/self-hosting/aws.md > Build the Docker image and deploy the parser to AWS Lambda. **For AWS users with a registered domain who want to scale beyond the local Docker self-host path.** By the end, you'll have a Lambda function triggered by S3 events processing emails into DynamoDB, plus a monthly-summary Lambda triggered on schedule by EventBridge. > **Prerequisite — a registered domain.** This path requires a domain you control DNS for. SES inbound depends on MX and DKIM records on that domain; without one, nothing in this guide works. If you do not already own a domain, the IMAP/Docker path ([Email setup](/self-hosting/email/)) does the same job without DNS. ## Prerequisites Before running the deployment scripts: 1. **Two IAM roles in the AWS console.** Create `email-parser-iam` (with `AWSLambdaBasicExecutionRole` plus `s3:GetObject` on the inbound bucket) and `lambda-s3-trigger-role` (used by the monthly-summary function). The deployment scripts assume both already exist. 2. **Email-to-S3 routing.** Complete [Email-to-S3 setup](/self-hosting/email-to-s3/) so your S3 bucket already receives bank emails — a separate setup step covering domain verification, SES inbound, MX record, and the S3 receipt rule. Lambda is downstream of that pipeline; without inbound emails landing in S3, the function has nothing to trigger on. ## Docker development (Lambda function) **Note:** The Lambda Docker setup (`docker/email_parsing/`) is separate from the devcontainer. The devcontainer is for local development; the Lambda Docker is for AWS deployment. ### Building and testing locally ```bash # Build docker image for Lambda (from repo root, using -f flag) docker build --platform linux/amd64 -f docker/email_parsing/Dockerfile -t docker-image:test . # Or use the build script (can be run from any directory): bash docker/email_parsing/2_build_image.sh # Run container locally for testing docker run -it --rm docker-image:test # Test Lambda function with sample event # (Ensure test_event.json is configured properly) ``` The Lambda loads `OPENAI_API_KEY` from SSM Parameter Store first (`/email-parser/openai-api-key`), falling back to the function's environment variables — see the loader note under the deployment workflow. It is not baked into the Docker image. ## AWS deployment workflow Run the deployment scripts in order from `docker/email_parsing/`: ```bash cd docker/email_parsing/ ./0_authenticate_to_ecr.sh # Authenticate to AWS ECR ./1_aws_repo_create.sh # Create ECR repository ./2_build_image.sh # Build and tag Docker image ./3_push_image.sh # Push image to ECR ./4_establish_lambda_func.sh # Create Lambda function ./5_update_func.sh # Update existing Lambda function ./6_invoke_func.sh # Test invoke Lambda function ./7_establish_summary_lambda.sh # Create monthly-summary Lambda function ./8_create_schedule.sh # Create EventBridge monthly schedule ./9_establish_coverage_lambda.sh # Create coverage-check Lambda function ./10_create_coverage_schedule.sh # Create EventBridge daily coverage schedule ``` For routine updates to an existing deployment, you typically only need steps 0, 2, 3, and 5. ### What each step relies on If you want to read the AWS documentation alongside the scripts, the canonical references are: - Steps 0–3: [Pushing a Docker image to an ECR private repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html). The build script targets `linux/amd64` because Lambda runs x86_64 by default; `aws ecr create-repository` uses `--image-tag-mutability MUTABLE` and `--image-scanning-configuration scanOnPush=true`. (Note: AWS now recommends configuring image scanning at the registry level via `PutRegistryScanningConfiguration` rather than per-repo with `--image-scanning-configuration`. The per-repo flag still works and is acceptable for a single-account hobby deployment.) - Step 4: [Creating Lambda functions defined as container images](https://docs.aws.amazon.com/lambda/latest/dg/images-create.html). The script invokes `aws lambda create-function --package-type Image --code ImageUri=`. - Step 5: [Updating Lambda function code with a container image](https://docs.aws.amazon.com/lambda/latest/dg/images-create.html). Uses `aws lambda update-function-code --image-uri` against both the email-parser and monthly-summary functions. (The same `images-create.html` page covers both create and update workflows.) - Step 7: Same Lambda container-image flow for the monthly-summary function. Assumes the role `lambda-s3-trigger-role` already exists. - Step 8: [Creating a schedule with EventBridge Scheduler](https://docs.aws.amazon.com/scheduler/latest/UserGuide/getting-started.html). Cron `cron(0 17 8 * ? *)` is "minute 0, 17:00 UTC, day-of-month 8, any month, day-of-week placeholder, any year" — the 6-field format EventBridge Scheduler requires. The Lambda function loads `OPENAI_API_KEY` from SSM Parameter Store at `/email-parser/openai-api-key` ([Working with Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-working-with.html)). Falls back to env var, then `.env` for local/CI use. ## Verify the deployment Send a test email to your SES inbound address. Within ~30 seconds: 1. CloudWatch Logs for the email-parser function should show parser output. 2. The `Transactions` table in DynamoDB should have a new item with the parsed transaction. 3. The dashboard should display the parsed transaction. If any of those don't happen, check CloudWatch Logs first — the Lambda function logs every step from S3 trigger through DynamoDB write. ## Next step Your Lambda is deployed. Now wire notifications: [pick a provider](/notifications/) so you get alerted when transactions arrive. --- # S3 backup Source: https://docs.gettidings.com/s3-backup.md > Mirror receipt attachments and statement PDFs to a bucket you own — the durability answer for the files the backup zip leaves out. Mirror your receipt attachments and statement PDFs to an S3 bucket you own, so the one part of `data/` the backup zip cannot carry has a durable copy. :::note[Prerequisites] This section only appears in Settings → Backup when AWS credentials are already present on the machine — environment variables, a `~/.aws` profile, or an IAM role. Detection uses the standard boto3 chain. If you have no AWS credentials, the S3 backup section stays hidden and the [backup zip](/backup-and-restore/) is your backup path. You bring your own bucket. Tidings never creates one, and it never touches AWS credentials — it only reads whatever is already configured. ::: **For self-hosters who already use AWS and want a durable copy of their uploaded files.** The backup zip from Settings → Backup carries every transaction and your configuration, but it deliberately leaves out the receipt and statement *files* — those live only in `data/raw/`. This feature is the durability answer for those files. It works on either storage backend (SQLite or DynamoDB). ## What it backs up Two stores, mirrored to your bucket: - **Receipt and document attachments** — `data/raw/attachments/`. - **Statement PDFs** — `data/raw/statements/`. Alongside the files, every successful sync uploads a `manifest.json`: a metadata snapshot of the attachment rows (including which transaction each is linked to and any parse results) and the statement rows (including the reconciled transaction list for each statement). The manifest is what lets a restore rebuild the links, not just the raw files. The bucket layout is: ```text /attachments// /statements// /manifest.json ``` `` is the optional key prefix you configure; without one, the three names sit at the bucket root. The backup only ever reads, writes, or deletes keys under those three names. Anything else in the bucket is left alone, so an existing bucket you already use for other things is safe. ## What it does not do - **It is not the backup zip.** The zip still does not include attachment or statement files — this is the separate durability answer for AWS users. Keep taking the zip for transactions and configuration; see [backup and restore](/backup-and-restore/). - **No bucket creation in-app.** You create the bucket yourself (below). - **No manual "back up now" button.** Sync runs on a schedule, described below. - **No write-through.** A newly uploaded receipt is mirrored on the next scheduled pass, not the instant you attach it. ## Create the bucket Any region works, but Tidings talks to the bucket in the region it resolves from `AWS_REGION`, then `AWS_DEFAULT_REGION`, falling back to `us-west-2`. Create the bucket in that region, or set the environment variable to match wherever you make it — a region mismatch is the most common verify failure. ### Console In the AWS S3 console, create a bucket: - **Name:** globally unique. `tidings-backup-` is a reasonable pattern. - **Region:** the region Tidings resolves (see above). - **Block public access:** keep all four settings on. Tidings writes with your credentials, never via public access. - **Versioning:** enable it. This is the recommended accidental-delete safety net — see [Versioning is your safety net](#versioning-is-your-safety-net) below. - **Default encryption:** SSE-S3 is fine. SSE-KMS works too; it adds no extra step on the Tidings side. ### CLI The same bucket from the command line, in `us-west-2`: ```bash aws s3api create-bucket \ --bucket tidings-backup-example \ --region us-west-2 \ --create-bucket-configuration LocationConstraint=us-west-2 aws s3api put-public-access-block \ --bucket tidings-backup-example \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true aws s3api put-bucket-versioning \ --bucket tidings-backup-example \ --versioning-configuration Status=Enabled ``` `us-east-1` is the one region that rejects the `LocationConstraint` argument — drop the `--create-bucket-configuration` line there. ## Minimal IAM policy Whatever credentials Tidings resolves need these permissions on the bucket. The first three are required; the last two are read-only and used only by the Verify button's warnings. Omitting them degrades those two warnings to "could not check" — it does not fail verification or sync. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "TidingsBackupList", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" }, { "Sid": "TidingsBackupObjects", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::/*" }, { "Sid": "TidingsBackupVerifyWarnings", "Effect": "Allow", "Action": [ "s3:GetBucketVersioning", "s3:GetBucketPublicAccessBlock" ], "Resource": "arn:aws:s3:::" } ] } ``` Replace `` with your bucket name in both resource lines. ## Configure and verify Open **Settings → Backup → S3 backup**: 1. Enter the **bucket** name and, optionally, a **prefix** (a key prefix inside the bucket — useful if the bucket holds other things). 2. Choose **Verify**. Tidings checks, in order: AWS credentials are present, the bucket exists and is reachable in the resolved region, and a write probe succeeds (it puts and then deletes a tiny `/.tidings-write-probe` object). It also raises non-blocking warnings if S3 Block Public Access is not fully on or bucket versioning is not enabled. 3. When verification passes, turn the **enable** toggle on. The first sync runs within a minute; after that it runs hourly. Below the controls, a status row shows the last sync time, the file counts, and the last error if there was one. ## How sync behaves - **Hourly reconcile, inside the app process.** No separate service to run. The first pass fires within a minute of enabling, then hourly. - **New and changed files upload.** The reconcile compares what is on disk against what is in the bucket and uploads anything new or changed. - **Deletions are mirrored.** A receipt or statement you delete locally is deleted from the bucket on the next pass. This keeps the bucket a faithful mirror rather than an ever-growing pile — which is exactly why versioning matters. - **The manifest refreshes every successful pass**, so the metadata snapshot stays current with your files. ### Versioning is your safety net Because sync mirrors deletions, a file removed locally leaves the bucket on the next pass. S3 **versioning** is the recommended protection against an accidental local delete: with versioning on, a delete writes a delete marker and the prior version is still recoverable in the bucket's version history. Tidings recommends versioning but never enables or enforces it — that stays your choice on the bucket. If you skip it, a mirrored deletion is permanent. ### When sync fails On the first failure in a run of failures, Tidings sends one notification through your configured notification provider (see [notifications setup](/notifications/)). It then retries every five minutes until a pass succeeds, at which point it returns to the hourly cadence. Further failures in the same run stay silent, and recovery is silent. The current state — last sync, counts, last error — is also on the status row in Settings → Backup, and over the API at `GET /api/v1/data/s3-backup-status`. ## Restore Restore is a command-line step, meant for a fresh or empty install — it downloads the files and manifest from the bucket and rebuilds the attachment and statement metadata databases, including the receipt-to-transaction links: ```bash uv run python -m src.finance.s3_backup_restore --bucket --prefix ``` `--bucket` and `--prefix` default to the values in `data/config.json` when omitted, so on the machine that wrote the backup you can usually run it with no arguments. Preview first with `--dry-run` — it reports what would be downloaded and rebuilt without writing anything: ```bash uv run python -m src.finance.s3_backup_restore --bucket --dry-run ``` Restore is oriented at a fresh target. Run it against an empty install, confirm the dry run looks right, then run it for real. ## Troubleshooting Verify reports the specific failure rather than a generic error: - **No credentials.** Tidings found no AWS credentials in the standard chain. The S3 backup section only shows when credentials are present, so this usually means they were removed or an IAM role expired. Confirm your environment, `~/.aws/credentials`, or role is in place. - **Bucket not found.** The bucket name is wrong, or the bucket lives in a different AWS account than your credentials. Check the name for typos. - **Access denied.** The credentials reached the bucket but lack permission. Apply the [minimal IAM policy](#minimal-iam-policy) above. - **Wrong region.** The bucket exists but in a different region than the one Tidings resolved (`AWS_REGION`, then `AWS_DEFAULT_REGION`, then `us-west-2`). Either recreate the bucket in the resolved region or set the environment variable to match the bucket's region. - **Warnings could not be checked.** If the credentials lack `s3:GetBucketVersioning` or `s3:GetBucketPublicAccessBlock`, the two bucket warnings show as "could not check" instead of a result. Verification and sync still work — add those two read-only permissions if you want the warnings. Sync failures surface in two places: the status row in Settings → Backup (last error and last sync time), and the one notification sent on the first failure in a run of failures through your [notification provider](/notifications/). --- # Add a parser Source: https://docs.gettidings.com/add-a-parser.md > Write a parser so Tidings recognizes a new bank's alert emails. Tidings turns the transaction-alert emails your bank already sends into a private spending journal. Each supported bank has a small parser that reads one email and returns structured fields (amount, merchant, type). Five Canadian banks ship today (RBC, CIBC, MBNA, Simplii, PC Financial), but the architecture is country- and language-neutral: a parser for any bank, anywhere, is the most useful contribution you can make. This guide is self-contained. Part 1 covers the email parser (the primary path). Part 2 covers the PDF statement parser (a secondary path, for banks that don't send alert emails or for reconciling uploaded statements). ## When your bank isn't supported yet An alert from a bank Tidings doesn't recognize is not lost. If the body looks like a transaction alert (a `$`-amount plus at least two alert keywords), it is captured to the **Needs review** queue even with AI turned off. From there the user can enter it by hand, or — with "Rescue unreadable emails with AI" enabled in Settings — the app recovers it automatically by sending the body to the configured AI provider (institution lands as `Other`). Those captured emails are also the raw material for a real parser: the flow is capture → review → rescue-or-author, and this guide is the *author* step. Once your parser lands, `POST /api/v1/parse-failures/retry-all` recovers the whole backlog through it. ## Minimal setup You don't need the full DevContainer to write a parser — a clone and `uv` are enough: ```bash git clone tidings && cd tidings uv sync # creates .venv, installs Python 3.12 + deps uv run pytest tests/unit/ -v # confirm a green baseline ``` Always run Python through `uv run` (`uv run pytest`, `uv run python`), never bare `python`/`pip` — only `uv run` sees the project's dependencies. Full environment notes: [`environment-management.md`](https://github.com/tvhahn/tidings/blob/main/docs/guides/environment-management.md). ## Start from your own quarantined emails The best fixtures are the real alerts your instance already captured. List the quarantined rows through the store factory (works on both storage backends): ```bash uv run python -c ' from src.finance.storage import create_parse_failure_store store = create_parse_failure_store() for r in store.list_failures("quarantined", 1000): frm = r.get("from_email") or "" domain = frm.rsplit("@", 1)[-1] if "@" in frm else frm print("id=" + str(r["id"]), "domain=" + str(domain), "subject=" + str(r.get("subject"))) ' ``` With the dev server running, turn a captured email straight into a scrubbed fixture pair (PII redacted, amounts kept; refuses to overwrite, and requires a git checkout with demo mode off): ```bash curl -X POST http://localhost:8000/api/v1/parse-failures//to-fixture \ -H "Content-Type: application/json" -d '{"institution": ""}' ``` It writes `tests/test_data//.txt` (the scrubbed body) plus a `.json` skeleton whose `"TODO"` fields you complete from the body. Without a server, scrub manually with `src/finance/fixture_scrub.scrub_body`. Agents (and users running Claude Code) should prefer the `.claude/skills/add-a-parser/` skill — it walks this whole guide end-to-end from the captured emails, enforces the evidence rule (a transaction type with no real sample is not parsed; it falls through to AI extraction), and finishes with a bulk `retry-all` of the backlog. --- ## Part 1 — Email parser ### The contract A parser lives under `src/finance/parsers/` and subclasses `TransactionParser` from `src/finance/parser_base.py`. The abstract contract is one method: ```python # src/finance/parser_base.py class TransactionParser(ABC): @abstractmethod def parse_email(self, email_body_text: str, email_details: dict[str, Any]) -> dict[str, Any]: ... ``` `parse_email` receives the plain-text email body and the `email_details` dict already extracted from the message (sender, date, etc.). It returns that dict merged with the transaction fields you parse out: `name`, `amount`, `company`, `transaction_type`, and `institution`. Use the `merge_details` and `parse_amount` helpers exported from `parser_base.py` — don't reinvent them. `AMOUNT_PATTERN` (also exported) matches comma-grouped or plain amounts (`1,234.56`, `1000.00`) so `$1000.00` parses as `1000.0`, not `100.0`. Reference parser: `src/finance/parsers/rbc_parser.py`. It's short, covers purchases, withdrawals, and e-transfers, and is the cleanest thing to clone. ### Create the parser Create `src/finance/parsers/_parser.py` (e.g. `td_parser.py`, `chase_parser.py`, `barclays_parser.py`). Subclass `TransactionParser` and implement `parse_email`: ```python # src/finance/parsers/chase_parser.py import logging import re from typing import Any from src.finance.parser_base import ( AMOUNT_PATTERN, TransactionParser, merge_details, parse_amount, ) logger = logging.getLogger(__name__) class ChaseParser(TransactionParser): def parse_email(self, email_body_text: str, email_details: dict[str, Any]) -> dict[str, Any]: parsed_data = None amount_match = re.search(rf"purchase of \$({AMOUNT_PATTERN})", email_body_text) if amount_match: parsed_data = { "name": None, "amount": parse_amount(amount_match.group(1)), "company": None, # regex out the merchant here "transaction_type": "purchase", } email_details["institution"] = "Chase" return merge_details(email_details, parsed_data) ``` `merge_details` returns `email_details` unchanged when `parsed_data` is `None`, so an email the parser doesn't recognise still flows through with its `institution` stamped. Keep regexes readable and log matches at `DEBUG`. See how `RBCParser.parse_purchase` splits amount and company into named patterns. ### Locale considerations The five shipping parsers all handle Canadian-dollar emails with the `$1,234.56` thousand-separator format. Parsing emails from elsewhere, watch: - **Currency symbol.** `$` is hard-coded in the existing regexes. Yours may need `€`, `£`, `¥`, `kr`, `₹`, `R$`, or a three-letter code (`EUR 42.00`). Don't assume `$`. - **Decimal and thousand separators.** `parse_amount` strips `,` as a thousand separator and treats `.` as the decimal point — fine for Canadian, US, and UK formats. If your bank writes `1.234,56` (German, Italian, Spanish, Portuguese conventions), do the swap inside your parser before calling `parse_amount`. - **Date formats.** Bank emails vary widely (`DD/MM/YYYY`, `YYYY-MM-DD`, `15 Jan 2026`, `Jan 15`). Parse to a Python `datetime` early and let the existing pipeline format it. - **Non-English bodies.** Regexes anchored on English keywords ("Purchase authorized", "You sent") won't match a Deutsche Bank or BNP Paribas email. Localize the keyword set in your parser; don't assume English in shared helpers. ### Register the parser Everything routes through `src/finance/email_pipeline.py`. Wire your parser into **three** places there: 1. **`PARSER_KEYS`** (module-level tuple) — the source of truth for which institution names can appear in an email body. Phase-2 body-text detection iterates it in order, and `parse_recovery` uses it to decide whether an unparsed email is even relevant. **A parser missing from here is invisible to body-text detection** (Interac e-transfers, emails with no matchable sender): ```python PARSER_KEYS: tuple[str, ...] = ("CIBC", "RBC", "MBNA", "Simplii", "PC Financial", "Chase") ``` 2. **The `parsers` dict** built by `build_parsers()` in `email_pipeline.py` (parser imports sit at module top) — instantiates each parser. Both detection phases look the parser up here by key: ```python from src.finance.parsers.chase_parser import ChaseParser # ... parsers = { "CIBC": CIBCParser(), "RBC": RBCParser(), "MBNA": MBNAParser(), "Simplii": SimpliiParser(), "PC Financial": PCFinancialParser(), "Chase": ChaseParser(), } ``` 3. **`_detect_institution_by_sender`'s `domain_map`** — maps the sender's email domain to your key, so Phase-1 sender detection runs before the body-text fallback: ```python domain_map = { "cibc.com": "CIBC", "alerts.rbc.com": "RBC", "mbna.ca": "MBNA", "pcfinancial.ca": "PC Financial", "chase.com": "Chase", } ``` If your bank routes alerts through a generic clearing service like Interac, leave the domain out — body-text detection (step 1) picks them up. That's why `payments.interac.ca` is intentionally omitted. ### Add test fixtures Drop sanitised email bodies (strip real names, account numbers, URLs) and their expected parsed output under `tests/test_data//` — a `.txt` file for the raw body plus a matching `.json` for the expected fields. Example pair already in the repo: - `tests/test_data/rbc/2024.10.22_15.45_abc123def456_rbc_purchase.txt` — the raw email body. - `tests/test_data/rbc/2024.10.22_15.45_abc123def456_rbc_purchase.json` — the expected output: ```json { "institution": "RBC", "name": "Demo User", "amount": 127.53, "company": "Costco Wholesale", "transaction_type": "purchase", "email_filepath": "tests/test_data/rbc/2024.10.22_15.45_abc123def456_rbc_purchase.txt" } ``` Cover at least one fixture per transaction type you support (purchase, withdrawal, e-transfer, pre-auth, etc.) plus an `edge_case_*` fixture for anything nasty (large amounts, unusual wording, non-ASCII merchant names). Put Chase fixtures in `tests/test_data/chase/`. Then add a parametrised test at `tests/unit/test__parser.py` — copy `tests/unit/test_rbc_parser.py`. `load_test_data("chase")` (from `tests/conftest.py`) picks up every JSON in your fixture dir automatically. ### Run the tests ```bash uv run pytest tests/unit/ -v -k chase # tight loop make verify # full gate before you open a PR ``` `tests/property/test_parser_invariants.py` runs shared invariants against every parser in its explicit `PARSERS` list — register yours there, add a matching `BODY_FACTORIES` entry (a body template with the amount slotted in; the whitespace-invariant test KeyErrors without it), and add any common-word trigger to `TRIGGER_SUBSTRINGS`. A parser missing from the list gets no property coverage. --- ## Part 2 — PDF statement parser For banks that don't send transaction alerts, or for users who want to reconcile uploaded PDF statements against their email-derived history, Tidings also parses statement PDFs. Two banks ship today (`RBCChequingParser`, `SimpliiChequingParser`). This is a secondary path — email-first ingestion is the product's core — but a real one that reaches users whose banks send no alert emails at all. For any other bank, the upload falls back to the AI statement parser (`src/finance/statement_parser_ai.py`) when the user has opted in via `ai_statement_parsing_enabled` — it prompts the configured AI provider with the PDF's extracted text and fail-closed-validates the reply (every amount must appear verbatim in the text). That fallback makes unsupported banks usable immediately, but a deterministic parser is still the destination: it is free, instant, offline, and its output never needs a "check this against the PDF" caveat. Use the AI fallback's results as ground truth while building one. ### The contract Statement parsers subclass `StatementParser`, defined in `src/finance/statement_parser_base.py` (the public module `src/finance/statement_parser.py` re-exports it and everything below, so the `from src.finance.statement_parser import ...` surface stays stable): ```python # src/finance/statement_parser_base.py class StatementParser: institution: str = "" account_type: str = "" def parse(self, pdf_bytes: bytes) -> StatementParseResult: ... def validate_pdf(self, pdf_bytes: bytes) -> str | None: ... # size/magic-byte check ``` `parse` returns a `StatementParseResult` (`transactions`, `metadata`, `raw_descriptions`, `cleaned_descriptions`). Reference parser: `src/finance/parsers/rbc_statement_parser.py` (`RBCChequingParser`). It uses `pdfplumber` to read word positions and reuses the shared helpers imported from `statement_parser_base` — `_detect_columns`, `_extract_statement_period`, `_parse_page`, `clean_statement_description` — plus the base's `validate_pdf()` and `_setup_logging()`. ### Two ways to extract the table Statement PDFs vary in how cleanly they expose their text. Two slash-command skills help during development: - **`parse-statement-text`** (`.claude/commands/parse-statement-text.md`) — read pdfplumber's text output and structure it into JSON. Use this when the PDF has selectable text and recognizable column layouts. - **`parse-statement-vision`** (`.claude/commands/parse-statement-vision.md`) — render each page to PNG and parse the image. Use this when the PDF is scanned or when text extraction loses the column structure. Start with text extraction; fall back to vision when the text is unusable. Either way, the goal is a list of `{date, description, amount, type, balance}` records that your parser converts into a `StatementParseResult`. ### Create and register the parser Create `src/finance/parsers/_statement_parser.py`, subclass `StatementParser`, set `institution` and `account_type`, and implement `parse` (clone `RBCChequingParser`). Then register it in `src/finance/statement_parser.py`: 1. Import your parser class at the top of the module. 2. Add its name to `__all__`. 3. Extend `select_parser(pdf_bytes)` to detect your bank's PDF — typically by searching the first page or two for an institution marker. See how Simplii is detected (a `"simplii"` text match or a `"trans."` header word); anything not matched falls through to `RBCChequingParser`. ### Reconciliation `src/finance/statement_reconciler.py` matches statement transactions against the existing transaction store (DynamoDB or SQLite). It compares by amount, date, and direction, so your parser must set each transaction's `type` to `withdrawal` (outflow) or `deposit` (inflow) correctly — the reconciler maps those onto the DB's `purchase`/`withdrawal`/`preauth` (outflow) and `deposit` (inflow) types. The shared page-parsing helpers already emit the right types for the shipping parsers. ### Add fixtures and run Statement fixtures live alongside email fixtures under `tests/test_data//`: a `.pdf` plus a matching `.json` of the expected output. Existing example: - `tests/test_data/rbc/Rbc_Chequing_2025-02-24_to_2025-03-24.pdf` - `tests/test_data/rbc/Rbc_Chequing_2025-02-24_to_2025-03-24.json` Sanitise the PDF first — replace real account numbers, balances, and merchant data with synthetic values that still exercise the parser's edge cases. The email-fixture tests filter statement JSONs out by their different schema, so the two can share a directory. ```bash uv run pytest tests/ -v -k statement make verify ``` --- When your parser is green, open a PR — see [`CONTRIBUTING.md`](https://github.com/tvhahn/tidings/blob/main/CONTRIBUTING.md) for PR conventions. One bank per PR is ideal. --- # Architecture Source: https://docs.gettidings.com/architecture.md > How Tidings turns bank emails into a transaction journal. **For contributors and curious self-hosters.** This document explains how Tidings turns bank transaction emails into categorized data, across both the everyday Docker self-host path (IMAP polling → SQLite) and the advanced AWS Lambda path (SES → S3 → Lambda → DynamoDB). After reading, you'll understand the dual-path design, where transactions enter the system, how categorization happens, and where data lives. A few domain terms used throughout: a **forwarder** (or forwarding address) is the dedicated email recipient that bank alerts are routed to; **`ForwardedTo`** is that recipient address used as the DynamoDB partition key. An **override** is a known-good company-to-category mapping that short-circuits OpenAI. A **briefing** is an AI-generated spending narrative; an **override audit** records which rule fired on a given transaction. ## System overview **Self-host path.** Gmail IMAP → Python poller → SQLite, all running in a single Docker stack: ```text Gmail inbox imap_poller (Docker) ┌─────────────┐ ┌──────────────────────────────┐ │ forwarded │ IMAP │ imap_poller.py │ │ bank emails │────────>│ │ │ └─────────────┘ │ ├─> email_pipeline.py │ │ ├─> detect institution │ │ ├─> parse transaction │ │ ├─> categorize (OpenAI) │ │ └─> write (SQLite) │ └──────────────────────────────┘ │ v data/finance.db │ v FastAPI ── React UI ``` **Advanced AWS path.** Same parser/categorizer/storage code, different transport and storage — SES inbound delivers emails into S3, Lambda processes each new object into DynamoDB: ```text Email ──> S3 (.eml) ──> Lambda (Docker) ──> DynamoDB │ └──> SNS (SMS) ``` The pipeline shared between both paths is described next; the AWS variant is detailed in [Advanced: AWS serverless variant](#advanced-aws-serverless-variant) at the end. ## IMAP polling daemon `src/finance/imap_poller.py` is a long-lived polling daemon that replaces the AWS Lambda path for self-hosted deployments. It connects to an IMAP inbox, fetches new bank transaction emails, and runs them through the same `parse_email()` pipeline used by `lambda_function.py`. ### How it works 1. **UID persistence** — `_load_last_uid()` and `_save_last_uid()` store the last-seen message UID in the `config_store` SQLite table (PK `SYSTEM#imap_poller`, SK `last_seen_uid`). New UIDs above the stored value are fetched on each poll; already-processed messages are skipped. A second row at SK `last_poll_at` is updated on every successful poll; the `/api/v1/health` endpoint reads it as the IMAP heartbeat. 2. **Per-message processing** — `process_message(raw_bytes, uid, transactions_db, context_enricher)` mirrors the Lambda handler flow: - `parse_email()` extracts transaction fields - `transactions_db.add_transaction()` writes to storage; returns `None` (invalid), `False` (duplicate), or the `DateFileName` (new) - `context_enricher.enrich()` computes month-to-date spending context; `update_context()` persists it - `notification_service.send()` dispatches the transaction notification through the configured provider — SNS, ntfy, or Twilio, log-only when none is configured (see [`guides/notifications-setup.md`](/notifications/)) — suppressing blocked companies and unknown transaction types 3. **Connect + poll loop** — `ImapPoller` maintains an `imaplib` connection, polls every `poll_interval` seconds (default 60s), and applies exponential backoff (5s → 300s) on connection errors. Designed to run as a Docker service alongside the finance dashboard. ### Testing `tests/unit/test_imap_poller.py` mocks `imaplib` + pipeline calls to verify orchestration logic. `tests/integration/test_imap_poller.py` is the opt-in end-to-end counterpart: it fetches one real message (requires `IMAP_USER` + `IMAP_PASSWORD`) and verifies the SQLite write — the self-hosted analogue of `tests/integration/test_lambda_e2e.py` on the AWS path. ## Anatomy of a request Both paths run the same parser pipeline. The entry points differ — `imap_poller.py:_process_message` for IMAP self-hosted, `lambda_function.py:handler` for AWS — but everything from `email_pipeline.parse_email()` onward is shared code. Here's what happens when a transaction email arrives, file by file: ### Transport and entry 1. **Transport trigger** — A raw email lands in the inbox (IMAP) or as a `.eml` object in S3 (AWS). The IMAP poller fetches new UIDs on its poll loop; the S3 event invokes the Lambda function. 2. **Entry point** — `imap_poller.py:_process_message` or `lambda_function.py:handler()` initializes the OpenAI client (`gpt-5.4-nano`) and calls `parse_email()`. ### Processing pipeline 3. **`email_pipeline.py:parse_email()`** — Orchestrates the full pipeline: - `extract_raw_email_details()` parses the raw bytes into structured headers (from, to, date, subject) and extracts the body text. The `X-Forwarded-To` header is used to identify the recipient; `user_mappings.csv` maps this to a `UserId` via an in-memory cache. - `parse_email_body()` detects the financial institution and delegates to the right parser (see [Parser system](#parser-system) below). 4. **Parser** — The matched parser (e.g., `RBCParser`) extracts `amount`, `company`, `name`, and `transaction_type` from the body text using regex patterns. 5. **`categorizer.py:categorize_transactions()`** — Runs the tiered `resolve_override()` matcher (exact → normalized → alias) against the overrides + aliases loaded from storage. On a hit, stamps `{source, matched_rule, confidence, reviewed_at}` onto the transaction as `_category_audit` so it rides through to `add_transaction(category_audit=...)`. Only novel phrasing the normalizer can't reach falls through to OpenAI (see [OpenAI integration](#openai-integration)). 6. **`transaction_db.py:add_transaction()`** — Generates a dedup hash, checks for duplicates, and writes to DynamoDB or SQLite (depending on backend; see [Data storage](#data-storage)). Returns the `DateFileName` sort key on success. 7. **`transaction_context.py:TransactionContextEnricher.enrich()`** — For new transactions, queries the same month's partition to compute month-to-date spending for the category, count merchant visits, and look up budget targets. The resulting `TransactionContext` is stored back on the transaction record via `update_context()`. Fail-open: returns `None` on any error. ### Outcomes 8. **Notification** — If the transaction is new (not a duplicate) and the company isn't in the blocked list, the system publishes via the configured notification provider (SNS for AWS, Ntfy/Twilio/log-only for self-hosted). When enrichment context is available, the message includes budget progress (e.g., `restaurant/dining — $340/$400 (85%)`) and merchant frequency (e.g., `3rd visit this month`). **When things don't match:** If no parser matches the email body, `email_pipeline.py:parse_email_body()` returns the raw `email_details` dict without transaction fields. The email is **not** dropped: every entry point hands it to the parse-failure quarantine (below), which either recovers it via AI extraction or captures it in the dead-letter store for later retry. The same applies when `add_transaction()` rejects parsed fields (`failure_stage: "db_validation_failed"`). ### Parse-failure quarantine `src/finance/parse_recovery.py` is the shared recovery gate all three transports call (IMAP poller, Lambda handler, `upload-eml`). When parsing yields no usable transaction: 1. **Relevance gate** — an email is relevant when `_detect_institution_by_sender()` matches, a parser name (`email_pipeline.PARSER_KEYS`) appears in the body, or the `email_is_transaction_alert()` classifier says so (classifier errors bias toward capture). Everything else is ignored, as before. 2. **AI extraction fallback** — with an AI client and `ai_categorization_enabled` (the same privacy opt-out the categorizer honors), `extractor.extract_transaction()` runs one forced tool call constrained to `{amount, company, transaction_type}`. `validate_extraction()` enforces the anti-hallucination contract: the amount must appear verbatim in the body (candidate-set membership, `$`/comma tolerant) and the company must be a whitespace-collapsed case-insensitive substring. Recovered transactions are stamped with an `ExtractionAudit` provenance map (`category_audit.build_extraction_audit()`), flow through the normal categorize → `add_transaction` → notification path, and join the attention queue until a human touches them. 3. **Quarantine** — anything relevant that wasn't recovered is persisted by the dual-backend `ParseFailureStore` pair (`parse_failure_store.py` DynamoDB / `parse_failure_store_local.py` SQLite, migration 003) with the full `email_details` dict, a `failure_stage` (`no_parser_match` | `extraction_empty` | `ai_extraction_failed` | `ai_validation_failed` | `db_validation_failed`), and a deterministic `pf_` id so Lambda redeliveries upsert. `GET/DELETE /api/v1/parse-failures[/{id}]` list and dismiss rows; `POST /api/v1/parse-failures/{id}/retry` re-runs the deterministic parsers after a parser fix; `POST /api/v1/parse-failures/{id}/resolve` hand-enters a transaction for a quarantined email the parsers can't read (the "Needs review" manual-entry path). Rows ride along in the `/api/v1/data/export` backup zip. The whole gate is fail-open (`recover_or_quarantine` never raises) — a quarantine bug must never break ingestion. The first quarantine per institution per 24h sends one calm notification via `notification_service.send_raw`; `/health` exposes the 7-day quarantine count (see [Health endpoint](#health-endpoint)). ## HTTP API The FastAPI app at `src/api/main.py` serves the dashboard frontend and all external consumers (Claude CLI subprocesses, Home Assistant automations, mobile shortcuts). The router modules in `src/api/routers/` (one per `/api/v1/*` resource) are mounted through `src/api/dependencies.py` factories so both storage backends (DynamoDB + SQLite) work transparently. ### Versioning Every route mounts under `/api/v1/`. Breaking changes will ship as `/api/v2/` rather than in-place so existing consumers (the React frontend, any external script) keep working until they migrate. The policy is documented in the root OpenAPI description in `src/api/main.py` and surfaces in `/docs`. ### Authentication Two schemes cover the two kinds of caller; the full walkthrough is [`guides/agent-access.md`](/agent-access/). - **Browser sessions (TOFU).** First run is trust-on-first-use: the dashboard works without a password until one is set via `POST /api/v1/auth/set-password` (Settings → Password). After that, `POST /api/v1/auth/login` issues a signed `tidings_session` cookie; `/logout` and `/sign-out-all` revoke sessions. Password hashing and session issue live in `src/finance/auth_session.py`; the routes in `src/api/routers/auth.py`. - **Bearer tokens (agents and scripts).** `bearer_auth_middleware` (`src/api/auth.py`, mounted in `main.py`) validates `fin_…` tokens minted by `src/finance/agent_tokens.py` — only the SHA-256 hash is persisted. Each token carries a scope checked against per-scope route allowlists (`SCOPE_ALLOWLISTS` in `src/api/auth.py`); `read` covers GET routes, `read+write` all methods. `/api/v1/health` and the other `PUBLIC_PATHS` stay unauthenticated. ### Unified error schema Every 4xx/5xx response follows the shape: ```json { "error": "Transaction not found", "code": "NOT_FOUND", "details": null } ``` Routers keep raising `HTTPException(status_code=..., detail="...")` — the handler at `main.py:http_exception_handler` rewrites every exception into the shape above using `code_from_status()` (see `src/api/errors.py` for the status→code map: `404 → NOT_FOUND`, `409 → CONFLICT`, `422 → VALIDATION_ERROR`, etc.). Pydantic `RequestValidationError` gets the full error list in `details`. Use `ApiException(status_code, code, message, details)` from `src/api/errors.py` when you need a custom machine code or structured details — the handler unpacks its richer payload automatically. ### Discovery `FastAPI(openapi_tags=...)` in `src/api/main.py` declares one tag per router domain (`_OPENAPI_TAGS`). The auto-generated schema is reachable at: - `http://localhost:8000/docs` — Swagger UI - `http://localhost:8000/redoc` — ReDoc - `http://localhost:8000/openapi.json` — machine-readable schema ### Scripts share modules, not HTTP Anything that needs domain logic outside the API — the operator CLIs in `dev/cli/`, slash commands, the AWS Lambda handlers — imports the shared modules under `src/finance/` directly rather than calling over HTTP. This is deliberate: module-level access is the break-glass path for fixing data when the API is not running. The correct DRY pattern is shared importable modules under `src/finance/`, consumed by both sides. Canonical example: `src/finance/insights_context.py` exports `gather_context(year_month)`. The `GET /api/v1/insights/context` endpoint imports it and returns the JSON; the `/spending-insights` slash command runs the same function offline. Single function, two callers, no HTTP coupling. ### Filter semantics `/api/v1/transactions/search` filters are intentionally *not* uniform across fields. `company` is a case-insensitive substring match (merchant strings arrive with payment-processor noise like `"Sq *coffee Spot Cen"`); `category`, `institution`, and `type` are case-insensitive exact match (clean enum-like values). The split is load-bearing domain knowledge, not an inconsistency — documented per-field in OpenAPI at `src/api/routers/search.py`. ### Health endpoint `GET /api/v1/health` (`src/api/routers/health.py`) is an unauthenticated liveness probe. It returns `{status, version, backend, imap_last_poll, imap_poll_age_seconds, last_transaction_at, last_transaction_age_seconds, parse_failures_7d, ai_categorization_status, checked_at}` in under 50 ms. Status thresholds: `ok` when the IMAP poller heartbeat is under 5 minutes old (or the poller isn't configured); `degraded` at 5–30 minutes, when `parse_failures_7d > 0` (one or more emails quarantined in the last 7 days — the per-institution template-drift signal, surfaced in minutes instead of weeks), or when `ai_categorization_status` is `degraded` (a run of hard provider errors filing transactions as Miscellaneous); `stale` past 30 minutes, or when no transaction has been parsed in 14 days. `stale` always wins over a quarantine-driven `degraded`. Every read is fail-open — a fresh boot with no parse-failures table reports `parse_failures_7d: null` rather than 500-ing. The React frontend polls the endpoint and surfaces the status as a sync dot beside the month picker. ## Parser system Parsers use the **Strategy pattern**. Each institution has a class inheriting from `TransactionParser` (`parser_base.py`), which requires implementing `parse_email(email_body_text, email_details) -> dict`. Beyond the five email parsers, `etransfer_parser.py` handles Interac e-Transfer notifications from any Canadian bank using `payments.interac.ca`, and statement PDF parsers (`rbc_statement_parser.py`, `simplii_statement_parser.py`) cover batch imports — detailed in the [Statement import](#statement-import) section below. ### Two-phase institution detection `email_pipeline.py:parse_email_body()` selects the parser in two phases: 1. **Sender domain** — Match `from_email` against a domain map (`cibc.com` → CIBC, `alerts.rbc.com` → RBC, etc.). This is fast and reliable for direct institution emails. 2. **Body text fallback** — If no domain matches (e.g., forwarded emails, Interac e-transfers from `payments.interac.ca`), scan the body text for institution name strings. Interac e-transfers are intentionally omitted from the domain map because multiple institutions (RBC, Simplii) use the same `payments.interac.ca` sender — the body text reveals which institution originated the transfer. ### Parser capabilities | Parser | Transaction Types | Detection Method | |--------|------------------|-----------------| | **RBC** | purchase, withdrawal, e-transfer | Sender domain or body text | | **CIBC** | purchase, preauth payment | Sender domain or body text | | **MBNA** | purchase | Sender domain or body text | | **PC Financial** | purchase | Sender domain or body text | | **Simplii** | e-transfer | Body text only (uses Interac sender) | The shared `etransfer_parser.py:parse_e_transfer()` function handles e-transfer parsing for both RBC and Simplii. ### Adding a new parser 1. Create `src/finance/parsers/_parser.py` with a class inheriting from `TransactionParser` (from `parser_base.py`) 2. Implement `parse_email()` — extract `amount`, `company`, `name`, `transaction_type` using regex 3. Register it in `build_parsers()` in `email_pipeline.py` (the imports sit at module top) 4. If the institution has a unique sender domain, add it to `_detect_institution_by_sender()` 5. Add test fixtures in `tests/test_data//` (`.txt` + `.json` pairs) ## Data storage ### DynamoDB schema **Table: `Transactions`** (on-demand billing / `PAY_PER_REQUEST`) | Attribute | Type | Role | Description | |-----------|------|------|-------------| | `ForwardedTo` | String | **Partition key** | Email recipient address | | `DateFileName` | String | **Sort key** | `YYYY.MM.DD_HH.MM_.eml` — encodes temporal ordering | | `TransactionHash` | String | | SHA-256 dedup hash (see below) | | `UserId` | String | | Mapped from `ForwardedTo` via `user_mappings.csv` | | `Institution` | String | | e.g., "RBC", "CIBC" | | `Amount` | Decimal | | Transaction amount (stored as DynamoDB Decimal) | | `Company` | String | | Merchant or recipient name | | `TransactionType` | String | | "purchase", "withdrawal", "preauth", "e-transfer" | | `Category` | String | | Lowercased category from OpenAI (e.g., "groceries") | | `Name` | String | | Cardholder first name | | `Date` | String | | Timestamp in the configured `timezone` (`src/finance/app_timezone.py`, default Pacific), formatted `MM/DD/YYYY HH:MM ZZZ` where `ZZZ` is that zone's abbreviation (e.g. `PDT`/`PST`) | | `FileName` | String | | S3 object key | | `FromName`, `FromEmail` | String | | Sender details (email lowercased) | | `ToName`, `ToEmail` | String | | Recipient details (email lowercased) | | `Subject` | String | | Email subject line | | `Body` | String | | Full email body text | | `CategoryAudit` | Map | | Audit metadata: `source` (`override`/`override_normalized`/`override_alias`/`override_fuzzy`/`manual`/`audit`/`statement_import`/`statement_enrich`/`manual_edit`), `matched_rule` (original-case override key that fired), `confidence` (Decimal; `1.0` for Tiers 0–2, cosine score for Tier 3), `reviewed_at` (ISO timestamp) | | `ExtractionAudit` | Map | | Provenance for AI-recovered rows (parse-failure quarantine path): `method` (`ai_fallback`), `model`, `validated` (always `true`), `extracted_at` (ISO timestamp), `schema_version`. Built by `category_audit.build_extraction_audit()` | | `Ignored` | Boolean | | When true, transaction is excluded from summaries | | `Comment` | String | | User annotation/note on the transaction | | `DeletedAt` | String | | ISO 8601 timestamp when soft-deleted; absent if active | | `TransactionContext` | Map | | Enrichment context: `category_month_total`, `merchant_month_count`, optional `category_budget_target` and `category_budget_pct` | The table is auto-created if it doesn't exist. ### DynamoDB: BudgetConfig table **Table: `BudgetConfig`** (on-demand billing / `PAY_PER_REQUEST`) | Attribute | Type | Role | Description | |-----------|------|------|-------------| | `PK` | String | **Partition key** | `USER#` — derived from `data/config.json` `user_id` field | | `SK` | String | **Sort key** | Prefixed config key (e.g., `BUDGET#targets#2026`, `BUDGET#groups#2026`) | | `Data` | Map | | JSON config payload | | `UpdatedAt` | String | | ISO 8601 timestamp | | `Version` | Number | | Monotonically increasing; used for optimistic locking | Two items per year: - **Budget Targets** (`SK = "BUDGET#targets#2026"`) — spending ceiling and per-category targets (annual amount, input_mode, category_type, derived monthly_amount) - **Category Groups** (`SK = "BUDGET#groups#2026"`) — display-only grouping of categories for UI organization Writes use `ConditionExpression` for optimistic locking. On conflict, the API returns HTTP 409 and the frontend re-reads before retrying. ### BudgetService `src/finance/budget_service.py` — follows the same pattern as `SpendingSummary`. CRUD operations for budget configuration, historical average computation (with 1-hour in-memory cache), and category type inference (fixed/variable/lumpy). Every write also persists a JSON backup to `src/finance/config/budget_config.json`. On the self-hosted SQLite backend, the `BudgetConfig` equivalent lives in `data/finance.db` and is created automatically by `ensure_schema()`. On the AWS path, the `BudgetConfig` DynamoDB table is provisioned during initial deployment. ### Schema migrations (SQLite) The self-hosted backend owns its schema lifecycle through an idempotent migration runner at `src/finance/migrations/`. A `schema_version (version INTEGER PRIMARY KEY, applied_at TEXT NOT NULL)` table records every applied migration; the runner discovers numbered modules (`001_initial.py`, `002_...`, ...), applies each pending migration in a transaction, and records success. `ensure_schema()` is called by the IMAP poller at startup and by every SQLite-backed service constructor on first use; the FastAPI lifespan does not call it explicitly. Users see one consistent database regardless of which service wakes up first after an upgrade. ### Deduplication Transactions are deduplicated using a SHA-256 hash of six fields: ```text SHA-256( forwarded_to | institution | amount | company | date | transaction_type ) ``` Before writing, `_transaction_exists()` queries DynamoDB for a matching `TransactionHash` under the same `ForwardedTo` partition. **The check fails open** — if the query errors (DynamoDB throttling, network issue), the transaction is written anyway. This prioritizes data capture over strict uniqueness; a duplicate row is less harmful than a lost transaction. ### Timezone handling All dates are normalized to the configured app timezone (`timezone` key in `data/config.json`, default `America/Los_Angeles`) before storage. The sort key format (`YYYY.MM.DD_HH.MM_filename.eml`) uses this local time, ensuring chronological ordering within each partition. Because the sort-key prefix is local-time and is written once at ingest, changing `timezone` does **not** rewrite history. New transactions bucket in the new zone; older rows retain their original prefix. For a single-user deployment that stays in one place this is invisible. For users who move continents, midnight-adjacent transactions ingested before the switch may group on the "wrong" calendar day in the Journal until they age out. The helper lives in `src/finance/app_timezone.py` — `get_app_timezone()` returns a `ZoneInfo` and `get_tzinfos()` produces the `dateutil.parse(tzinfos=…)` map. The map always resolves `PST`/`PDT` to Pacific regardless of configured zone so legacy rows (and test fixtures) continue to parse correctly after a zone switch. ## OpenAI integration The system uses OpenAI's **function calling** (tool use) to get structured output — not free-text parsing. ### Transaction categorization The categorization flow has three stages, in order: 1. **Override resolver short-circuit.** `categorize_transactions()` runs the tiered `resolve_override()` matcher against overrides + aliases. If any tier hits, the category is returned immediately without calling OpenAI (see [Category override system](#category-override-system)). 2. **Feature gate.** The OpenAI call is gated on the `ai_categorization_enabled` flag in `data/config.json`. AI categorization is enabled by default when `OPENAI_API_KEY` is set; toggle in Settings → Intelligence to disable it. When disabled, the categorizer returns the `Miscellaneous` fallback without touching OpenAI. 3. **OpenAI call.** If no override matches and the gate is open, the function sends the transaction amount and company to `gpt-5.4-nano` with a forced function call (`categorize_transaction`) that constrains the output to one of 39 predefined categories: > Alcohol, Auto Maintenance, Baby & Kids, Bank Fees, Car Payment, Charitable Giving, Childcare, Clothing, Coffee & Cafes, Education, Entertainment, Fitness, Gasoline, Gifts, Groceries, Health Care, Hobbies, Home Goods, Home Maintenance, Insurance, Internet, Miscellaneous, Mortgage, Moving, Personal Care, Pets, Phone, Professional Services, Public Transit, Rent, Restaurant/Dining, Rideshare & Taxi, Strata/HOA, Subscriptions, Taxes, Technology, Therapy, Travel, Utilities Existing users keep whatever list is in their `CategoryConfig` storage — the JSON above is read once on first access for fresh users and then storage takes over. **Fallback:** If the API call fails, returns no completion, or the response can't be parsed, the category defaults to `"Miscellaneous"`. ### Transaction alert detection `email_is_transaction_alert()` uses a separate function call (`detect_if_transaction_alert`) to classify whether an email is actually a transaction alert vs. a credit-card payment confirmation or unrelated email. Returns `True`, `False`, or `None` on error. ## Category override system Known-good company→category mappings are managed via `OverrideService` (DynamoDB-backed) and also persisted to `data/config/category_overrides.json` as a backup. The Lambda pipeline loads overrides and merchant aliases together via `config_loader.py:get_override_context()` — a bundled 5-minute in-memory cache that returns `(overrides, aliases)` and is invalidated by override or alias mutations. ### Tiered resolver (`resolve_override`) All five override lookup sites — Lambda categorizer, `OverrideService.lookup_category`, statement reconciler, `/fix-categories` dev script, and `CategorySuggester._exact_match` — delegate to `src/finance/category_resolver.py:resolve_override(company, overrides, aliases=None)`. The resolver returns a frozen `ResolvedOverride(category, tier, matched_rule, confidence)` or `None`, applying tiers cheap-to-expensive and returning on the first hit: | Tier | Method | `source` stamp | Confidence | Available in Lambda? | |------|--------|----------------|------------|----------------------| | 0 (exact) | Case-insensitive string equality against override keys | `override` | `1.0` | yes | | 1 (normalized) | `merchant_normalizer` regex cleanup on both sides — strips store numbers (`#1234`), location markers, province/country codes, punctuation. One `BOOSTER JUICE` override catches every `BOOSTER JUICE #N` variant | `override_normalized` | `1.0` | yes | | 2 (alias) | Same as Tier 1, plus merchant alias substitution after cleanup. Skipped when aliases is empty | `override_alias` | `1.0` | yes | | 3 (fuzzy) | Cosine similarity on `CategorySuggester`'s embedded corpus. Threshold `0.85` (reconciler) / `0.90` (automatic paths) | `override_fuzzy` | cosine score | **no** — no embedding imports in Lambda module graph | **Ambiguity blacklist:** when two override keys collapse to the same normalized (or alias-resolved) key but disagree on category, the group is dropped from that tier's map and falls through to the next. Example: `SHOPPERS DRUG MART #123 → Health Care` vs `SHOPPERS DRUG MART #456 → Groceries` → neither resolves on `SHOPPERS DRUG MART #789`; the transaction falls through to OpenAI. Tier 3 is available to the webapp and statement reconciler via an optional `suggester` parameter on the resolver (Phase 2; see the dated specs under `docs/specs/` — local-only, absent in the public repo). ### Audit trail Every resolver hit stamps `CategoryAudit` on the new transaction row at insert time via the new `add_transaction(transaction_data, category_audit=...)` kwarg on both DynamoDB and SQLite backends. The audit map carries `source` (tier-derived), `matched_rule` (original-case override key that fired), `confidence` (Decimal in DynamoDB, REAL in SQLite), and `reviewed_at` (ISO timestamp). Phase 4's duplicates UI reads `matched_rule` to show which override collapsed onto each variant. SQLite schema backs the audit fields with `category_audit_reviewed_at`, `category_audit_source`, `category_audit_matched_rule` (TEXT), and `category_audit_confidence` (REAL) columns. Added idempotently on connection bootstrap via `_apply_column_migrations()` in `local_db.py`. ### Retroactive category correction `TransactionsDB` provides two methods for updating historical records: - **`update_category(forwarded_to, date_file_name, new_category, source)`** — Changes the `Category` attribute and writes a `CategoryAudit` map with `reviewed_at` (ISO timestamp) and `source` (e.g., `"manual"`, `"override"`, `"audit"`). Returns the old category value. - **`mark_category_reviewed(forwarded_to, date_file_name, source)`** — Writes `CategoryAudit` metadata without changing the category, marking the transaction as audited. ### Slash command workflow The `/review-categories` and `/fix-categories` slash commands form a review-then-fix pipeline. `/review-categories` analyzes transactions for inconsistencies and updates the overrides file. `/fix-categories` applies those overrides retroactively to DynamoDB and audits the full history. The slash-commands operational guide lives at `docs/guides/slash-commands.md` in the repo. ### Category management services The category override system is also backed by DynamoDB for webapp access, using two services that share the `CategoryConfig` table (same schema as `BudgetConfig` — PK/SK with optimistic locking). **`OverrideService`** (`src/finance/override_service.py`) — CRUD for the company→category overrides map. Stores the full map as a single DynamoDB item (`SK = "CONFIG#category_overrides"`). Supports individual put/delete, bulk replace, and suggestion dismissal tracking (a `Dismissed` map on the same item keyed by `company_lower|category_lower`). Every write persists a JSON backup to `category_overrides.json` and `dismissed_suggestions.json`. **`CategoryService`** (`src/finance/category_service.py`) — CRUD for the master category list. Stores the sorted list as a single DynamoDB item (`SK = "CONFIG#categories"`). Supports add, rename, and delete with case-insensitive uniqueness checks. `Miscellaneous` is protected from rename/delete. Falls back to the local `categories.json` file if DynamoDB is unavailable. Every write persists a JSON backup to `categories.json`. Both services fall back to their local JSON files (`category_overrides.json`, `categories.json`) when DynamoDB is unavailable, so the self-hosted SQLite path works without provisioning any AWS resources. On the AWS path, the `CategoryConfig` table is created during initial deployment; the one-time setup scripts that were used to bootstrap the original maintainer's account are not part of the public repo. **API endpoints** (`src/api/routers/overrides.py` and `src/api/routers/category_management.py`): - `GET /api/v1/overrides` — List all overrides with version - `PUT /api/v1/overrides/{company}` — Add or update a single override - `DELETE /api/v1/overrides/{company}` — Remove an override - `GET /api/v1/overrides/suggestions` — Get category suggestions derived from user edits - `POST /api/v1/overrides/suggestions/dismissed` — Dismiss a suggestion - `DELETE /api/v1/overrides/suggestions/dismissed/{key}` — Undismiss a suggestion - `GET /api/v1/categories/managed` — List categories with version - `POST /api/v1/categories` — Add a new category - `PUT /api/v1/categories/{old_name}` — Rename a category - `DELETE /api/v1/categories/{name}` — Delete a category - `GET /api/v1/categories/{name}/usage` — Check transaction count before deletion - `PUT /api/v1/categories/{name}/group` — Update a category's budget group assignment ## Monthly spending summary The `spending_summary.py:SpendingSummary` class generates monthly spending reports from DynamoDB data. ### How it works `query_month(year_month)` queries all transactions for a given `YYYY-MM` month by scanning each `ForwardedTo` partition with a `begins_with` filter on the `DateFileName` sort key prefix. Results are aggregated across all partitions. `aggregate(items)` splits transactions into spending types (purchase, withdrawal, preauth) and deposit types (e-transfer), then computes totals, counts, breakdowns by category and company, and top categories. `get_summary_with_comparison(year_month)` fetches the current and previous month summaries and calculates month-over-month deltas for total spending and category-level changes. ### Automated SMS notifications `docker/email_parsing/summary_handler.py` is a separate Lambda handler that automates the monthly SMS. It reuses the same Docker image as the email parser but with a different entry point (`summary_handler.handler`). An EventBridge Scheduler rule triggers the Lambda on the 8th of every month at 17:00 UTC (10:00 AM PDT during daylight saving, 09:00 AM PST otherwise — the cron expression is fixed in UTC, so the local time shifts seasonally). The handler calls `SpendingSummary.get_summary_with_comparison()` for the previous month, formats the result with `format_sms()`, and publishes it to the same SNS topic used for transaction notifications. Deployment scripts `7_establish_summary_lambda.sh` and `8_create_schedule.sh` create the Lambda function and schedule respectively. ## AI spending insights The system generates AI-powered spending briefings via Claude Code CLI. Generation runs as a background task; the frontend polls for status. ### How it works 1. **Context gathering** — `src/finance/insights_context.py:gather_context()` assembles a comprehensive spending snapshot: current + previous month comparison (via `SpendingSummary`), 6-month trend, budget targets and year-to-date totals (via `BudgetService`), and historical category averages. All `Decimal` values are converted to `float` for JSON serialization. The context is written to `data/insights/context_.json`. 2. **Background-task generation** — `POST /api/v1/insights/generate` (`src/api/routers/insights.py`) gathers the context and kicks off a background generation task that spawns a Claude CLI subprocess (`claude -p --output-format text --model --no-session-persistence`, via the shared transport in `src/finance/ai_cli.py`) with a 180-second timeout. The endpoint returns 202 with a task id. The frontend polls `GET /api/v1/insights/status` until the task completes, then reads the result via `GET /api/v1/insights/saved/{id}`. 3. **Saved briefings** — Completed briefings (>200 chars) are persisted to `data/insights//.md`. Two additional endpoints serve saved briefings: `GET /api/v1/insights/saved?month=YYYY-MM` (list) and `GET /api/v1/insights/saved/{id}?month=YYYY-MM` (retrieve). ### Slash command The `/spending-insights [YYYY-MM]` slash command provides an offline alternative — it runs the same `gather_context()` pipeline, then Claude Code generates the briefing directly in the conversation rather than via a subprocess. ## Spending journal The Spending Journal is a day-grouped transaction timeline that surfaces the per-transaction enrichment written by `TransactionContextEnricher` (`category_budget_pct`, `merchant_month_count`) directly in the UI, and pairs each day with an AI-generated narrative summary. ### How it works 1. **Day grouping** — `GET /api/v1/journal?month=YYYY-MM` (`src/api/routers/journal.py`) queries the month's transactions, filters out deleted/ignored rows, and groups them by the `YYYY.MM.DD` prefix of the `DateFileName` sort key. Days are ordered descending (newest first) but MTD totals are accumulated ascending so each day carries a running month-to-date figure. The budget ceiling (annual target ÷ 12) is attached when configured. 2. **Daily context gathering** — `src/finance/daily_summary_context.py:gather_daily_contexts()` takes the day-grouped list and builds a compact context dict per day: `day_of_week`, `day_total`, `transaction_count`, top merchants, `mtd_total`, top MTD categories, and budget ceiling. These dicts feed the AI prompt. 3. **Multi-provider summary generation** — `src/finance/summary_provider.py` exposes four implementations of a `SummaryProvider` ABC: - `OpenAISummaryProvider` — uses `instructor` + OpenAI for structured per-day output (one API call per day). - `ClaudeCLISummaryProvider` — spawns the Claude Code subprocess with a single batched prompt containing all days, then parses the response back into per-day summaries. - `CodexCLISummaryProvider` — spawns the OpenAI Codex CLI subprocess with the same single-batched-prompt approach. - `GeminiCLISummaryProvider` — spawns the Google Gemini CLI subprocess with the same single-batched-prompt approach. `create_summary_provider()` dispatches based on `data/config.json`'s `daily_summary_provider` field (`"openai"`, `"claude_cli"`, `"codex"`, `"gemini_cli"`, or `"disabled"`), threading the optional `daily_summary_model` / `daily_summary_reasoning_effort` overrides into the chosen provider (see [`configuration.md`](/configuration/)). 4. **Background generation** — `POST /api/v1/journal/summaries/generate` kicks off an `asyncio.Task` tracked in module-level state (single-slot; returns HTTP 409 if a job is in flight). As each day completes, the summary text is written to `data/journal//
.txt`. `GET /api/v1/journal/summaries/status` reports progress; `GET /api/v1/journal/summaries?month=YYYY-MM` returns the full saved set. The frontend Journal page (`frontend/src/pages/JournalPage.tsx`) renders each day as a card with the transactions, inline enrichment badges, a waveform pace bar, and the AI summary (collapsible). Auto-generation is scoped to the current day only. ## Cash-flow Sankey (Summary → Flow view) The Summary page's Flow view renders a Sankey diagram for the selected month: income sources split proportionally between a `Spending` hub (feeding the expense parent groups) and a `Kept` sink; when spending exceeds income, a `From savings` source covers the deficit. A three-figure strip above the diagram (`Income` · `Spent` · `Kept`, or `From savings` in a deficit month) always sums exactly. ### How it works 1. **Backend** — `GET /api/v1/summary` returns `deposits_by_company` alongside the existing aggregates. The aggregator already computed it; the `DepositSourceSummary` Pydantic model exposes it on the response. The same endpoint also carries the nullable `pace: MonthPaceInfo` block (populated only when the requested month is the current one) that drives the Summary page's mid-month cards and the chart's hatched projection. No dedicated endpoint and no DB writes — the Sankey is a frontend roll-up over `/summary` plus `/category-management` group assignments. 2. **Graph builder** — `frontend/src/lib/cashFlow.ts:buildCashFlowGraph()` joins deposits, category spend, and group memberships into the `{ nodes, links }` shape `@visx/sankey` expects, splitting each income source between spending and kept with cent-exact allocation. Pure function, no React; covered by `cashFlow.test.ts`. 3. **Render** — `SankeyCashFlow.tsx` (rendered by `SummaryPage.tsx`'s Flow view) uses `@visx/sankey` + `@visx/responsive`, with theme-aware palette via `getGroupColor` + `useChartTone`. Mobile (<768px) falls back to a text breakdown so narrow screens don't render an unreadable diagram. Phase A (this section) ships an MVP. Phase B — drill-down, dedicated endpoint, year scope, two-layer expense breakdown, and folding `IncomeStatement` into a tab — is captured in `docs/specs/_archive/2026-04-26-sankey-cash-flow-research/handoff-phase-b.md` (a local-only spec, absent in the public repo). ## Merchant intelligence (`/merchants`) The `/merchants` page surfaces structured recurring-charge detection, price-change alerts, and committed burn rate — separate from the AI narrative briefing on `/insights`. ### How it works 1. **`merchant_intelligence.py:MerchantIntelligenceService`** — runs over 6 months of `by_company` data from `SpendingSummary` (no new storage, 1-hour in-memory cache). Computes: - **Recurring detection** — fixed (same amount every month, ≥3 months) and variable (charged most months, ≥4 of 6). - **Price-change alerts** — recurring merchants whose latest amount drifted ≥5% AND ≥$1 from prior baseline. - **New / churned classification** — first appearance this month, or absent after recurring payments end. - **Committed burn rate** — sum of recurring monthly amounts, exposed alongside discretionary spend. 2. **`GET /api/v1/merchants/intelligence?month=YYYY-MM&months=6`** (`src/api/routers/merchants.py`) returns the structured payload. Demo mode computes the same shape on the fly from summary fixtures. 3. **Frontend** — `MerchantsPage.tsx` renders `MerchantSummaryCards` (committed vs. discretionary, counts), `RecurringMerchantsList` (fixed/variable with monthly amounts), and `MerchantAlerts` (price changes, new, stopped). Brand voice: "notable changes" not "alerts"; sentence case throughout. This page executes the `2026-02-24-merchant-intelligence` spec end-to-end and Phase 2 of `2026-04-26-gentle-insights-implementation`. ## Design decisions **Fail-open deduplication** — If the DynamoDB duplicate check fails (network error, throttling), the transaction is written anyway. Rationale: a duplicate row can be cleaned up later, but a lost transaction is unrecoverable. Financial data capture is prioritized over strict uniqueness. **Two-phase institution detection** — Sender domain first, body text fallback. Forwarded emails may not preserve the original sender. Interac e-transfers all come from `payments.interac.ca` regardless of originating bank, so body text is the only way to distinguish them. **Container-based Lambda** — The Docker image is built from the repo root (`docker build -f docker/email_parsing/Dockerfile .`) so it can include `src/` directly. The `OPENAI_API_KEY` is loaded at runtime from SSM Parameter Store via `src/finance/secrets.py` (SSM → env var → `data/.env` → project-root `.env` fallbacks for local/CI use), never baked into the image. **Function calling for categorization** — Using OpenAI's tool use with an enum constraint guarantees the response is one of the predefined categories. No free-text parsing or fuzzy matching needed. The `Miscellaneous` fallback handles any failure mode gracefully. **Category overrides** — Known-good company→category mappings in `category_overrides.json` short-circuit the OpenAI API call. This reduces latency and cost for recurring merchants while ensuring consistent categorization. The `/review-categories` slash command maintains the override file. **Blocked companies for SMS** — Recurring subscription charges from specific companies (e.g. YouTube, a paid newsletter) are suppressed from SMS to reduce notification fatigue. Unknown transaction types are also suppressed. ## Statement import The statement import system fills gaps in the email-based pipeline by parsing bank statement PDFs and importing missing transactions. ### Parser system `src/finance/statement_parser.py` defines the `StatementParser` base class, `StatementParseResult`, and the `select_parser()` auto-detection helper. Institution-specific parsers live in `src/finance/parsers/` and are re-exported from `statement_parser.py` for backward compatibility: - **`RBCChequingParser`** (`src/finance/parsers/rbc_statement_parser.py`) — RBC chequing account statements (single date column, CamelCase descriptions) - **`SimpliiChequingParser`** (`src/finance/parsers/simplii_statement_parser.py`) — Simplii Financial chequing account statements (two date columns: trans. date + eff. date, clean text descriptions) Each parser extends `StatementParser` and produces a `StatementParseResult` with transactions, metadata, and raw/cleaned descriptions. **Auto-detection:** `select_parser(pdf_bytes)` inspects the first two pages for Simplii markers (`"simplii"` in text or `"trans."` header word) and returns the appropriate parser. Defaults to `RBCChequingParser` if no Simplii markers are found or on error. The API upload and reparse endpoints use `select_parser()` automatically. Description cleanup (`clean_statement_description()`) strips known prefixes like `BillPayment`, `InteracPurchase`, `ATMwithdrawal`, etc., and applies CamelCase splitting. A `_PREFIX_DISPLAY_NAMES` dict maps prefixes that ARE the entire description (e.g., `Monthlyfee` → `Monthly fee`). Simplii descriptions are already clean text, so cleanup is near-no-op. ### Four-tier reconciliation `src/finance/statement_reconciler.py` matches parsed transactions against existing DynamoDB records: 1. **Tier 1 (Exact):** Same date + same amount (±$0.01) + compatible type → auto-matched 2. **Tier 2 (Suspected Duplicate):** Cross-type match (e.g., withdrawal vs e-transfer) + same direction → flagged for review 3. **Tier 3 (Fuzzy):** Date ±2 days + same amount + compatible type → flagged as ambiguous 4. **Tier 4 (New):** No match found → offered for import with category suggestion from overrides Type mapping: statement `withdrawal` matches DB `purchase`, `withdrawal`, `preauth`; statement `deposit` matches DB `e-transfer`, `deposit`. A direction filter (`_same_direction()`) rejects cross-type matches when transactions flow in opposite directions. Used-key tracking prevents double-matching. ### Statement-specific DynamoDB fields Statement-imported transactions use a subset of the Transactions schema: - **DateFileName:** `YYYY.MM.DD_00.00_stmt_RBC_.pdf` — synthetic sort key with `stmt_` prefix - **Date:** `MM/DD/YYYY 00:00 PST` — synthetic format for compatibility - **StatementSource:** e.g., `RBC_Chequing_2026-01` — identifies the source statement - **CategoryAudit:** `{"reviewed_at": "", "source": "statement_import"}` - **TransactionHash:** Uses raw description (not cleaned) for hash stability. When multiple identical transactions occur on the same day (same date, amount, description, type), an occurrence counter suffix (`|1`, `|2`, ...) is appended before re-hashing to produce distinct `TransactionHash` and `DateFileName` values. Occurrence 0 (the first) is unchanged for backward compatibility. Email-specific fields (`FromName`, `FromEmail`, `ToName`, `ToEmail`, `Subject`, `Body`, `FileName`) are omitted. ### SQLite persistence `src/finance/statement_store.py` provides a SQLite persistence layer (WAL mode) so users can resume partially-completed imports and view upload history. Three tables: `schema_version`, `statements`, `statement_transactions`. Default actions are assigned per reconciliation tier (new→import, matched→enrich/skip). Status is computed from action states: `pending_review` → `in_progress` → `complete`. ### API endpoints - `POST /api/v1/statements/upload` — Upload PDF, parse, reconcile, and persist to SQLite - `GET /api/v1/statements` — List all uploaded statements with status and counts - `GET /api/v1/statements/{statement_id}` — Retrieve statement detail with all transactions - `DELETE /api/v1/statements/{statement_id}` — Delete a statement and its transactions - `PATCH /api/v1/statements/{statement_id}/transactions/{tx_index}` — Update a single transaction (action, company, category) - `PATCH /api/v1/statements/{statement_id}/transactions` — Bulk-update multiple transactions - `POST /api/v1/statements/{statement_id}/reparse` — Re-parse PDF preserving user edits - `POST /api/v1/statements/import` — Execute import actions for reviewed transactions Uploaded PDFs are saved to `data/raw/statements//`. ## Attachments and receipts Any transaction can carry file attachments — a receipt photo, an invoice, a claim letter. Attachments are the evidence layer under the tax pack: a bank alert proves you *paid*; the receipt proves *what you bought*. ### Storage `src/finance/attachment_store.py` is a single SQLite store (`data/attachments.db`, WAL mode) on the StatementStore template — deliberately **not** a dual-backend pair, because attachment files live on the server's disk regardless of the transaction backend. Rows persist the transaction composite (`forwarded_to`, `date_file_name`); the `tx_id` surrogate exists only at the API boundary (`Depends(parse_tx_id)`). Attachment ids are deterministic (`att_` + 16 hex of `sha256(file_sha256|original_filename)`), so re-uploading the identical file upserts. Files land at `data/raw/attachments//_`; accepted uploads are JPEG/PNG/WebP/HEIC/PDF up to 10 MB, validated on both extension and declared content-type, with HEIC/HEIF converted to JPEG at upload (the stored content-type is always ours, never the client header). Demo mode uses its own empty `data/demo-attachments.db`. ### Receipt parsing (AI, opt-in) `src/finance/receipt_parser_ai.py` turns a receipt into `{merchant, date, total, line_items?}` using the provider already selected in Settings → Intelligence (`resolve_ai_statement_provider()` is reused, not forked). Text PDFs go through pdfplumber text; images are the codebase's first vision path — OpenAI gets one multimodal message with a base64 data URL, Codex gets `-i `, Claude Code gets the file path in the prompt plus `--allowedTools Read`; Gemini image input is deferred. Validation is fail-closed (`validate_receipt_parse`, pure): merchant/date/total are required, a text-PDF's total must appear verbatim in the extracted text, and line items are advisory — if they don't reconcile with the total within $0.05 they are dropped while the parse survives. Line items live only in the attachment row's `parse_json` and render only in the attachment view dialog; they are never aggregated. Parsing is gated by the `ai_receipt_parsing_enabled` consent (default off, never auto-enabled) and every successful parse carries a provenance stamp (`method`, `provider`, `model`, `parsed_at`, `schema_version`). ### Matching `src/finance/receipt_matcher.py` ranks the transactions a parsed receipt could explain — the same tier-ranked reconciliation shape as statement import, scoped to one receipt. T1 = exact amount (±$0.02) + same day + normalized-merchant equality; T2 = amount within tolerance + within a 3-day window; T3 = the restaurant-tip band (up to +20%) within the window. The matcher is pure (no storage imports); the candidates endpoint feeds it the receipt month's raw transactions (plus the adjacent month near a boundary). An unlinked receipt with exactly one T1 candidate auto-links; everything else is user-confirmed. Unmatched receipts never create transaction rows — they wait in the "Receipts to file" list on the Transactions page. ### API endpoints - `POST /api/v1/attachments` — multipart upload, optionally linked via a `tx_id` form field - `GET /api/v1/attachments?unlinked=true&kind=receipt` — list with optional filters - `GET /api/v1/transactions/{tx_id}/attachments` — a transaction's attachments - `GET /api/v1/attachments/{id}/file` — serve the file inline - `POST /api/v1/attachments/{id}/parse` — synchronous AI parse (422 when consent is off) - `GET /api/v1/attachments/{id}/candidates` — ranked match candidates; performs the auto-link - `POST /api/v1/attachments/{id}/link` — link to a transaction (`tx_id: null` unlinks) - `DELETE /api/v1/attachments/{id}` — remove the row and the file on disk ## Tax pack The `/tax` page ("Tax receipts") groups a calendar year's spending into claim lines with per-transaction evidence status, and exports the whole thing as a zip. ### How it works 1. **Mapping seed** — `src/finance/config/tax_line_mappings.json` maps seven lines (charitable, medical, childcare, moving, tuition, dues, instalments) to categories from the built-in taxonomy. It reads through `config_loader._config_path()`, so a personal copy in `data/config/` wins over the packaged seed; a category claimed by two lines fails at load. `note` is an informational string rendered verbatim. `cra_ref` (a country-specific line reference, e.g. Canada's "Line 34900") remains in the seed for self-hosters who want it, but is **no longer shown in the UI or the CSV export** — the labels are the country-neutral surface. 2. **Service** — `src/finance/tax_pack_service.py:TaxPackService.get_tax_pack(year)` walks the year's twelve months via `query_month`, keeps spending-type rows, and buckets by **lowercased** category (stored rows carry `"charitable giving"`, the seed says `"Charitable Giving"`). Evidence per transaction: `receipt` (a receipt-kind attachment links to the composite, bulk-probed in one query), `statement` (`StatementSource` present — no source email exists), or `email` (the source alert rides on the row). The pack computation reads transactions but is layered over a small override store (next point). 3. **Manual overrides** — membership is *(category-derived ∪ manually-included) − manually-excluded*. `src/finance/tax_override_store.py:TaxOverrideStore` is a single SQLite store (`data/tax_overrides.db`, WAL, on the attachment-store template — SQLite-only, not a dual-backend pair) keyed by the transaction composite, holding one row per override: `mode` (`include`|`exclude`) plus a target `line_key` for includes. `POST /api/v1/tax-pack/items` flags a transaction into a line (used by the "Flag as tax item" menu on the Journal and Transactions rows); `DELETE /api/v1/tax-pack/items/{tx_id}` clears an override (removes a manual flag or restores an excluded item). An included item whose category isn't mapped — or any include targeting `"other"` — lands in a synthetic **"Other claimable"** line that only appears when populated. Excluded derived members move to their line's `excluded_transactions` and stop counting toward totals/coverage. Both mutations are demo-gated. 4. **Export** — `GET /api/v1/tax-pack/export?year=` streams an in-memory zip: `summary.csv`, one `lines/.csv` per claim line, attachment files under `evidence//`, and the source email body as `.txt` for email-evidence rows. Disabled in demo mode. The page itself is the income-statement skeleton — year picker, expandable lines, evidence chips, a "Download tax pack" button — with a fixed footer: "Tidings organizes your records; it doesn't give tax advice." Each line's rows carry per-item actions: view the source email, view attached receipts, attach a new one, and remove the item (with a restorable "Removed" section per line). ## Agent activity ledger Every 2xx non-GET request under `/api/v1/*` (outside `/api/v1/auth/` and four write-verbed reads) is journaled to an append-only ledger. The pieces: - **Principal resolution** — `src/api/auth.py` resolves the caller on every request (`Principal`: token / session / tofu / dev-bypass) and stashes it on `request.state.principal`. Successful bearer auth also stamps the token's `last_used_at`, throttled to once per 15 minutes. - **Capture** — inside `bearer_auth_middleware` (deliberately not a second middleware: Starlette registration order would put it outside auth). After `call_next`, `src/api/activity.py` builds an envelope from the request scope only — bodies are never read, so multipart uploads and streaming responses pass through untouched — and dispatches the store write fire-and-forget (`asyncio.to_thread` task held in a strong-ref set; fail-open; zero response latency). - **Before/after images** — ten instrumented handlers (transaction patch/comment/fields/bulk, override put/delete, alias put/delete, budget config, groups) call `stage_before()` with the pre-mutation state and the value being written; everything else is captured envelope-only, `reversible: false`. Bulk images use a minimal per-row projection capped at 200 rows (a ledger entry is one DynamoDB item under the 400 KB limit). - **Store** — `ActivityStore` / `ActivityStoreLocal` (`src/finance/activity_store*.py`), the standard dual-backend pair (migration 004). Append-only API; 90-day retention on both backends (SQLite prune-on-write, DynamoDB TTL). - **Endpoints** — `GET /api/v1/whoami` (caller introspection), `GET /api/v1/activity` (filtered feed), `POST /api/v1/activity/{id}/revert` (`src/api/activity_revert.py` dispatch; stale-guard compares the entry's after-image against current state and 409s rather than clobber a newer edit; the revert is itself journaled and linked off the response path). - **UI** — Settings → Activity: the feed grouped by principal and 10-minute burst, expandable before/after, inline-confirm revert. ## Advanced: AWS serverless variant The AWS path uses the same parser, categorizer, and override system as the self-hosted path, but swaps the transport and storage layers: - **Transport.** SES → S3 → Lambda. Bank emails arrive via SES inbound, land as `.eml` objects in an S3 bucket, and trigger the Lambda function on each new object. See [Email-to-S3 setup](/self-hosting/email-to-s3/) for the SES + DNS + bucket policy walkthrough. - **Storage.** DynamoDB tables (`Transactions`, `BudgetConfig`, `CategoryConfig`) replace the SQLite file. Schema definitions are in the Data storage section above; both backends share the same field names. - **Notifications.** SNS publishes SMS messages to your phone number; the same notification message format is used as the self-hosted Ntfy/Twilio paths. - **Deployment.** A set of numbered shell scripts in `docker/email_parsing/` automate ECR image build/push, Lambda function creation, and EventBridge schedule for the monthly summary job. See [AWS deployment](/self-hosting/aws/) for the full sequence. Code volume favours the self-hosted path: `imap_poller.py` is ~575 lines with 50 unit tests; `lambda_function.py` is ~135 lines with 13 unit tests. The Lambda handler is intentionally thin because it delegates everything to shared `src/finance/` modules. ## Configuration | Setting | Location | Notes | |---------|----------|-------| | `OPENAI_API_KEY` | SSM Parameter Store (`/email-parser/openai-api-key`) | Loaded via `src/finance/secrets.py` tiered loader: SSM → env var → `data/.env` → project-root `.env`. Lambda returns HTTP 500 if missing | | AWS credentials | IAM role / env | S3, DynamoDB, SNS, Lambda access | | SNS topic ARN | `lambda_function.py` | Hardcoded; update when changing regions | | User mappings | `src/finance/user_mappings.csv` | Maps `ForwardedTo` → `UserId`; cached in memory | | OpenAI model | `lambda_function.py` | `DEFAULT_OPENAI_CHAT_MODEL` (`gpt-5.4-nano`, `src/finance/ai_cli.py`) is the fallback for every OpenAI-path feature; each feature's `*_model` config key overrides it (see [`configuration.md`](/configuration/)) | | Category overrides | `src/finance/config/category_overrides.json` | Company→category map; short-circuits OpenAI | | Categories enum | `src/finance/config/categories.json` | 39 predefined categories for OpenAI function call | | Blocked companies | `src/finance/config/blocked_companies.json` | Companies suppressed from SMS notifications | | Card name mappings | `src/finance/config/card_name_mappings.json` | Maps card last-4 digits → cardholder first name | | Budget config backup | `src/finance/config/budget_config.json` | Local JSON backup of BudgetConfig DynamoDB items | | Dismissed suggestions | `src/finance/config/dismissed_suggestions.json` | Override suggestions dismissed by user; prevents re-suggestion | | CI/CD | `.github/workflows/ci.yml` | Lint + test on push/PR to `main` | --- # Agent access Source: https://docs.gettidings.com/agent-access.md > Issue a bearer token so a local agent can call the Tidings API. Tidings exposes a JSON HTTP API at `/api/v1/*` that a local agent (Claude Code, Cursor, n8n, curl, a custom script) can call directly. Authentication is RFC-6750 bearer tokens — `Authorization: Bearer fin_…` on every request. This guide walks through issuing a token, using it, and revoking it. The bundled React dashboard does not need a token; it talks to `http://localhost:8000` over loopback and is unauthenticated by default. Agents that connect from anywhere else — a second device on your LAN, an agent running in another container, an automation pipeline — go through the bearer channel. ## Quickstart ```bash make agent-token LABEL='laptop-claude' ``` > Working in this checkout with Claude Code? The `headless-backend-bootstrap` skill runs this whole guide's setup in one invocation — it detects Docker-vs-dev-checkout, mints the token via the right path (including the restart the running process needs), verifies the auth contract, and prints the consumer snippets. Running the Docker stack rather than a dev checkout? `make agent-token` on the host writes to the checkout's `data/`, which the containers don't read — mint inside the container instead, so the hash lands in the named volume: ```bash docker compose exec finance python -c \ "from src.finance.agent_tokens import add_token; rec, raw = add_token(label='laptop-claude'); print(raw)" docker compose restart finance ``` The restart matters: the API loads tokens at startup, so a token minted via `exec` answers 401 until the `finance` container restarts. (Tokens minted from the app itself take effect immediately; only this out-of-band path needs the restart.) The command prints the token once. Save it now; only the sha256 hash is persisted to `data/config.json`, so the raw value is never recoverable. ``` Token: fin_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (illustrative — not a real token) id: 0123456789abcdef scope: read+write label: laptop-claude Save this token now — it will NOT be shown again. Pass it as `Authorization: Bearer ` on /api/v1/* requests. ``` Test the token against any read endpoint: ```bash TOKEN=fin_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx curl -sH "Authorization: Bearer $TOKEN" http://localhost:8000/api/v1/categories | jq . ``` A 200 with the category list means the token works. ## Try it without installing The demo journal is served read-only at a public endpoint, so an agent can explore the API before you clone anything. The base URL is `https://gettidings.com/demo/api/v1` — the same routes as a self-hosted install, backed by the fictional Mira Lin Chen fixtures. ```bash curl -s https://gettidings.com/demo/api/v1/health | jq .status curl -s 'https://gettidings.com/demo/api/v1/summary?month=2026-03' | jq . ``` The demo world covers `2025-05` through `2026-03`, with demo-today pinned to `2026-03-19`. All of it is fictional data. No token is needed — an `Authorization` header is accepted and ignored, so the demo never returns a 401. Writes are not part of the demo: any non-GET method returns a 405 pointing you at self-hosting for a writable journal. For how the endpoint is served and regenerated, see the [static hosted demo guide](https://github.com/tvhahn/tidings/blob/main/docs/guides/static-hosted-demo.md#demo-api). ## Token lifecycle Three Makefile targets cover the full lifecycle. Each wraps `scripts/agent/agent_token.py` directly. ```bash make agent-token LABEL='cursor-on-laptop' # issue, default scope read+write make agent-token LABEL='read-only-agent' SCOPE=read # issue with strict-read scope make agent-token-show # list ids, labels, scopes, last-used make agent-token-revoke ID=0123456789abcdef # delete by id ``` `agent-token-show` prints the table without raw values: ``` id label scope created_at last_used_at ---------------------------------------------------------------------------------- 0123456789abcdef laptop-claude read+write 2026-04-30T19:14:02 — 9ab21f0c7e... read-only-agent read 2026-04-30T19:15:10 2026-04-30T19:16:33 ``` Revocation is permanent — the row is removed from `data/config.json`. Any client still holding that token gets a 401 on its next request. `last_used_at` is stamped on successful bearer auth, throttled to once per token per 15 minutes (each stamp rewrites `data/config.json`), so it reads as "active recently", not per-request telemetry. For per-write accountability see [Every write is on the record](#every-write-is-on-the-record). ## Scopes Two scopes ship today; future scopes are a one-line addition in `src/api/auth.py`. | Scope | Allows | When to use | |---|---|---| | `read` | `GET /api/v1/*` | A third-party UI, a sketchy agent you want to box in, a read-only dashboard mirror. | | `read+write` | All methods on `/api/v1/*` | The default. Day-to-day agent work — categorize transactions, set overrides, import a statement. | The default at issuance is `read+write` because Tidings is single-user; gating writes behind a separate token rotation is friction without a security story. The `read` scope is the safer hand-out when the holder isn't fully trusted. ## Versioning The path prefix is `/api/v1/`. Breaking changes are allowed inside `v1`, lockstep with the `INSTALL.md` clone tag — no external consumer is depending on a stable contract today, so the prefix is hygiene rather than a guarantee. If you build against this API, pin to the `INSTALL.md` clone tag and use the committed `openapi.json` as your change log; `make verify-openapi` is the drift gate. The day a consumer needs a stability promise, `/api/v2/` ships alongside; until then there is one moving target. ## Public endpoints (no token required) These bypass auth so a fresh agent can discover the API before it has a token: - `GET /api/v1/health` — liveness + last-activity probe - `GET /openapi.json` — full OpenAPI schema - `GET /docs` — Swagger UI - `GET /redoc` — ReDoc UI - `GET /llms.txt` — a plain-text orientation file agents can read first For everything else under `/api/v1/*`: a request that presents an `Authorization` header is always validated against the configured tokens (invalid token → 401, insufficient scope → 403). A request with no header falls through to the browser channels — session cookie, or TOFU. That means anonymous access stays open until a **dashboard password** is set, even if tokens exist; setting a password (Settings → Password) is what closes it. With no password and no tokens — the state right after `docker compose up` — the API is fully open on loopback, preserving the zero-config quickstart. ## Worked examples ### curl ```bash TOKEN=fin_… BASE=http://localhost:8000/api/v1 # Read this month's spending curl -sH "Authorization: Bearer $TOKEN" "$BASE/summary?month=2026-04" | jq . # Search transactions curl -sH "Authorization: Bearer $TOKEN" \ "$BASE/transactions/search?merchant=tim+hortons" | jq '.transactions[] | {date, amount, category}' # Pin a category override (company name in the path, URL-encoded; category in the body) curl -sH "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -X PUT "$BASE/overrides/WHOLE%20FOODS" \ -d '{"category":"Groceries"}' # Search transactions by an array of merchants (POST sibling of the GET) curl -sH "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -X POST "$BASE/transactions/search-by-filter" \ -d '{ "from_month": "2026-01", "to_month": "2026-04", "merchant_in": ["whole foods", "trader joe", "costco"], "category_in": ["Groceries"], "min_amount": 10 }' | jq '.summary' ``` The POST `transactions/search-by-filter` endpoint is the agent-friendly companion to `GET /transactions/search` — same filter semantics, but `merchant_in` / `category_in` / `institution_in` / `type_in` accept arrays in the request body instead of needing each value URL-encoded as a separate query param. ### Python (httpx) ```python import httpx, os client = httpx.Client( base_url="http://localhost:8000/api/v1", headers={"Authorization": f"Bearer {os.environ['FINANCE_API_TOKEN']}"}, ) summary = client.get("/summary", params={"month": "2026-04"}).raise_for_status().json() print(summary["total_spent"]) ``` ### n8n Use the **HTTP Request** node: - URL: `http://localhost:8000/api/v1/summary?month={{ $now.format('yyyy-MM') }}` - Authentication: **Header Auth** (or **Generic Credential Type → Header Auth**) - Header name: `Authorization` - Header value: `Bearer fin_…` Store the token in n8n's credential vault, not in the workflow JSON. ### Custom HTTP-tool-using agents (Claude API tool use, ChatGPT actions, Cursor tools) Any agent surface that lets you define an HTTP tool can call this API. Configure the tool with: - **Base URL:** `http://localhost:8000` (or your reverse-proxy host) - **Authentication:** static header — `Authorization: Bearer fin_…` - **Spec:** point the agent at `http://localhost:8000/openapi.json` if it consumes OpenAPI; otherwise hand-write the tool schema for the endpoints you need. There is no native MCP server entry-point; use the HTTP path above — most agents handle bearer-auth REST cleanly. (Direction lives in [`ROADMAP.md`](https://github.com/tvhahn/tidings/blob/main/ROADMAP.md).) ## Errors The API returns the unified `{error, code, details}` shape on all 4xx/5xx responses. | Status | code | When | |---|---|---| | 401 | `UNAUTHORIZED` | Missing `Authorization` header, wrong scheme, or token not in `data/config.json`. | | 403 | `FORBIDDEN` | Token is valid but its scope does not permit this (method, path). For example, a `read` token attempting `PATCH /api/v1/transactions/bulk`. | Body for both: ```json { "error": "invalid token", "code": "UNAUTHORIZED", "details": null } ``` ## Every write is on the record Every mutating call to `/api/v1/*` is journaled in an append-only activity ledger: who made it (token id and label, browser session, or the pre-password device), which operation, when, and — for the invertible operations (transaction category/state/comment/field edits, bulk recategorization, overrides, merchant aliases, budget config, category groups) — a before/after image with one-call undo. Three endpoints and one page expose it: - `GET /api/v1/whoami` — the caller's resolved identity: kind, token id, label, scope, `last_used_at`. The quickest "is my token wired up?" check. - `GET /api/v1/activity` — the journal, newest first. Filters: `limit` (default 100, max 500), `since` (inclusive ISO timestamp), `operation`, and `principal` — a token id, or the literal `me` for "my own writes". An agent can read back its own mutations to self-verify they landed: ```bash curl -s -H "Authorization: Bearer $TIDINGS_TOKEN" \ "http://localhost:8000/api/v1/activity?principal=me&limit=10" ``` - `POST /api/v1/activity/{id}/revert` — undo an invertible entry. Refuses with `409 stale_revert` if the resource changed after the entry was written (pass `force=true` to override), 409 if already reverted or not reversible. The revert is itself journaled and the original entry is marked. - **Settings → Activity** — the same journal as a calm feed, grouped by principal and burst, with per-entry revert. Entries are retained for 90 days on both backends (SQLite prune-on-write, DynamoDB TTL). `last_used_at` on tokens is stamped at most once per 15 minutes per token — treat it as coarse presence, not an audit timestamp; the ledger is the audit trail. ## Security caveats Read this section before exposing the API on anything other than `localhost`. - **Tokens grant full data access at their scope.** A `read+write` token can delete every transaction. Treat them like database passwords. Rotate when an agent or device is decommissioned. Run `make agent-token-revoke ID=…` whenever a token leaks. The activity ledger (above) means a misbehaving token is at least *visible* and its instrumented edits are revertible — but permanent deletes and import commits are not, so the warning stands. - **Never commit a token.** They go in `.env`, in the agent's credential store, or in your password manager. The repo's `.gitignore` already excludes `data/config.json`, but the raw value never lives there anyway. - **Loopback is the assumed boundary.** Tidings has no rate limiting, no IP allowlisting, no origin allowlisting beyond CORS. Anything stronger is the responsibility of the network you put it on. - **TLS is the responsibility of the reverse proxy, not the app.** The FastAPI server speaks plaintext HTTP. Do not bind it to a public interface without a TLS terminator (Caddy, Traefik, nginx, Cloudflare Tunnel) in front. Bearer tokens travel in the `Authorization` header and are useless on the wire without TLS. - **Hash storage protects the disk, not the wire.** Tokens are sha256-hashed at rest, but the raw value travels with every request. If TLS is misconfigured, anyone on the path sees the token. ## Exposing the API on a LAN The defaults are tight on purpose. To expose the API to other devices (your phone over Tailnet, a household member's laptop, a separate container in the same compose project), three pieces have to line up. 1. **CORS.** The default allowlist is `http://localhost:5173`. Set `CORS_ALLOWED_ORIGINS` to the origin(s) that will call the API. A wildcard works for read-only uses but should be considered carefully. ```bash CORS_ALLOWED_ORIGINS=https://tidings.example.com,http://192.168.1.74:5173 # or, for development only CORS_ALLOWED_ORIGINS=* ``` 2. **Bind address.** Inside the devcontainer the FastAPI server binds to `0.0.0.0:8000` already; confirm the docker-compose port mapping forwards `8000` to the host. If you ran `uv run uvicorn` directly with `--host 127.0.0.1`, switch to `--host 0.0.0.0`. 3. **TLS.** Run a reverse proxy that terminates HTTPS. Caddy is the lowest-friction option: ```text tidings.example.com { reverse_proxy localhost:8000 } ``` Cloudflare Tunnel is a valid alternative if you want a stable public hostname without opening a port. Once those three are in place, every request from a non-loopback origin must carry a valid token. `make agent-token` is the right answer; do not poke holes in CORS to skip auth. ## How auth interacts with the bundled dashboard The dashboard has its own cookie-session channel, separate from bearer tokens. With no password set (`app_password_hash` absent from `data/config.json`), the app runs in TOFU mode: the dashboard works unauthenticated and prompts you to set a password under Settings → Password. Once a password is set, the dashboard authenticates via `POST /api/v1/auth/login` and rides a signed session cookie; bearer tokens remain the channel for agents and scripts. If you expose the API beyond loopback, set a password first — TOFU mode plus a reachable port is an open dashboard. ## Verification Three quick checks confirm the path end-to-end after issuing a token. Check 1 assumes a dashboard password has been set — in TOFU mode (no password yet) anonymous requests are allowed by design and it returns 200: ```bash TOKEN=fin_… # 1. No header → 401 (requires a dashboard password to be set; 200 in TOFU mode) curl -si http://localhost:8000/api/v1/categories | head -1 # HTTP/1.1 401 Unauthorized # 2. Valid token → 200 curl -si -H "Authorization: Bearer $TOKEN" \ http://localhost:8000/api/v1/categories | head -1 # HTTP/1.1 200 OK # 3. Read token on a write → 403 curl -si -H "Authorization: Bearer $READ_ONLY_TOKEN" \ -X DELETE http://localhost:8000/api/v1/transactions/foo/bar | head -1 # HTTP/1.1 403 Forbidden ``` If all three behave as shown, the auth layer is wired correctly. --- # Tidings for agents Source: https://docs.gettidings.com/for-agents.md > The machine-readable surfaces — llms.txt, raw-markdown docs, OpenAPI schemas, a public demo API, and bearer-token access to a real instance. Tidings treats agents as first-class users. The docs, the API, and the install path each have a machine-readable form; this page is the catalogue. ## Read the docs as markdown - [llms.txt](https://docs.gettidings.com/llms.txt) — a curated index of every docs page, one line each. - [llms-full.txt](https://docs.gettidings.com/llms-full.txt) — the entire documentation concatenated into one file. - Append `.md` to any page URL for its raw markdown — [/using/budgets.md](https://docs.gettidings.com/using/budgets.md), [/troubleshooting.md](https://docs.gettidings.com/troubleshooting.md), and so on. - Every page has a "Copy page" button in its header, for handing one page to an agent by hand. ## Try the API without installing The demo journal is served read-only at a public endpoint — eleven months of a fictional household's spending, no token required: ```bash curl 'https://gettidings.com/demo/api/v1/summary?month=2026-03' curl 'https://gettidings.com/demo/api/openapi.json' ``` The demo is a baked snapshot: GET only, exact query strings, months `2025-05` through `2026-03` — and a 404 lists the valid routes in its error body. The [agent guide](/agent-guide/) is a briefing written for agents: concepts, the demo API's rules, and a suggested tour shape. To hand your agent the tour: ```text Introduce me to Tidings. Read https://docs.gettidings.com/agent-guide.md, then use the live demo API to show me what a month of spending looks like. ``` ## Install by prompt [`INSTALL.md`](https://github.com/tvhahn/tidings/blob/main/INSTALL.md) is written as a prompt addressed to the agent, not a script — it covers OS detection, Docker versus AWS, failure handling, and verification. It is served at a stable URL, so the whole install fits in one line: ```text Help me install Tidings. Read https://docs.gettidings.com/install.md first, then walk me through it. ``` ## When something breaks The [troubleshooting guide](/troubleshooting/) is organized by symptom, most common first, and serves as raw markdown like every other page: ```text My Tidings install is misbehaving. Read https://docs.gettidings.com/troubleshooting.md, then help me diagnose it. ``` ## Work with a real instance Every route on a self-hosted instance is a versioned `/api/v1/` endpoint behind bearer-token auth. - Mint a token with `make agent-token LABEL='laptop-claude'`, or the Docker variant in the [agent access guide](/agent-access/) — which also covers scopes (`read`, `read+write`), revocation, worked examples for curl, Python, and n8n, and the LAN-exposure checklist. - The full OpenAPI schema is at [/openapi.json](https://docs.gettidings.com/openapi.json), and a running instance serves Swagger UI at `/docs`. --- # Agent guide Source: https://docs.gettidings.com/agent-guide.md > A briefing for AI agents — what Tidings is, how the demo API behaves, and how to give a human the tour. **You are an AI agent, and a human has asked you to introduce them to Tidings.** This page briefs you on what Tidings is, then hands you a live, read-only demo API so you can show rather than describe. Read it fully, then run the tour at the end. ## What Tidings is Tidings is a self-hosted personal finance journal built from the transaction alert emails a bank already sends. The user forwards those alerts to an inbox they control; an IMAP poller picks them up; deterministic parsers turn each one into a transaction; the result is a day-grouped journal of one household's spending. No bank credentials, no aggregator, no manual entry. Data lives in SQLite on the user's machine, and with no AI key configured, nothing leaves it. ## Core concepts - **Journal** — the default view: one month, read one day at a time, newest first. It shows actionable spending only; refunds, transfers, and ignored transactions live in the full ledger. - **Categories and overrides** — every transaction gets a category; an override pins a merchant pattern to a category from then on. - **Budgets** — annual targets per category, assessed by proration against the elapsed year ("pace"), plus a separate top-level spending ceiling. Categories are typed fixed, variable, or lumpy. - **Summary** — a month-scoped overview: stat cards, a Trend or Flow (Sankey) visualization, and a category table with six months of context. - **Insights** — the monthly briefing: computed momentum and anomaly cards, plus an opt-in AI-written narrative whose figures are checked against the underlying numbers. - **Merchants** — read-only merchant intelligence over the trailing six months: recurring detection, burn rate, price changes. - **Needs review** — quarantine for emails no parser could read; nothing is silently dropped. - **Statements** — PDF bank statement upload, for reconciling transactions that never arrived by email. ## The demo API Base URL: `https://gettidings.com/demo/api/v1` — schema at `https://gettidings.com/demo/api/openapi.json`. Three facts govern everything: - **No auth, GET only.** Every endpoint answers unauthenticated; an `Authorization` header is accepted and ignored; any non-GET method returns 405. - **Fictional data, fixed window.** The demo world covers `2025-05` through `2026-03`, with "today" pinned to `2026-03-19`. Only `2026-03` exercises current-month pace logic. - **Exact query strings only.** The demo is a baked snapshot, not a live query engine. `/transactions?month=2026-03` resolves; adding or reordering parameters does not. When a route returns 404, the error body lists the valid routes in `details.available` — read it and adjust instead of retrying. ## The tour Tidings copy is observational, not evaluative: state the numbers plainly, never congratulate or scold, no exclamation marks. Present the month the way a bank statement would, if a bank statement were pleasant to read. Fetch in this order, narrating as you go (all routes verified against the snapshot): 1. `GET /health` — confirm the demo is up and note that "today" is 2026-03-19. 2. `GET /summary?month=2026-03` — open with the month in one breath: total spent, deposits, the three largest categories. 3. `GET /journal?month=2026-03` — pick two or three days and read them like journal entries. 4. `GET /budget/status?year=2026` — the pace headline, and any category notably over or under. 5. `GET /summary/trend?end_month=2026-03&months=6` — how the month sits against the five before it. 6. Ask what the user wants next, and offer the pointers below. ## After the tour - The docs index for agents: `https://docs.gettidings.com/llms.txt` — and every docs page serves raw markdown if you append `.md` to its URL. - The machine-readable surfaces in one place: `https://docs.gettidings.com/for-agents.md` - If the user wants to install: read `https://docs.gettidings.com/install.md` and follow it — it is written as a prompt for you, and covers OS detection, Docker, verification, and handoff. - For API access to a real instance (bearer tokens, scopes): `https://docs.gettidings.com/agent-access.md` --- # API reference Source: https://docs.gettidings.com/api.md > Versioned /api/v1/ routes with a unified error shape. Generated from the live OpenAPI spec. # API reference > Versioned /api/v1/ routes with a unified error shape. Full machine-readable > schema: /openapi.json (OpenAPI 3.1). ## config - `GET /api/v1/config` — Get runtime app configuration (storage, demo mode, user_id, features) - `PUT /api/v1/config` — Update runtime app configuration - `POST /api/v1/config/test-openai` — Validate an OpenAI API key and persist it to data/.env on success - `POST /api/v1/config/test-s3-backup` — Verify an S3 bucket is a usable attachment-backup target ## ingestion - `POST /api/v1/transactions` — Add a transaction manually (bypasses email pipeline) - `POST /api/v1/transactions/upload-eml` — Upload a raw .eml file and parse via the email pipeline ## transactions - `GET /api/v1/transactions` — List active (non-deleted) transactions for a month - `GET /api/v1/transactions/all` — Combined active + attention + trash buckets for a month - `GET /api/v1/transactions/attention` — Transactions needing review (uncategorized, not ignored, not deleted) - `GET /api/v1/transactions/bulk` — Combined transactions across multiple months (max 12) - `PATCH /api/v1/transactions/bulk` — Update categories on many transactions in a single request - `GET /api/v1/transactions/latest` — Freshness probe — max DateFileName, optionally scoped to a month - `GET /api/v1/transactions/trash` — Soft-deleted transactions for a month - `DELETE /api/v1/transactions/{tx_id}` — Hard-delete a transaction (irreversible) - `PATCH /api/v1/transactions/{tx_id}` — Partial update — category, state (active/ignored/trashed), and/or reviewed - `PUT /api/v1/transactions/{tx_id}/comment` — Set or update the user comment on a transaction - `POST /api/v1/transactions/{tx_id}/delete` — Toggle the soft-delete flag (move to trash / restore) - `GET /api/v1/transactions/{tx_id}/detail` — Full transaction record including email body and headers - `PUT /api/v1/transactions/{tx_id}/fields` — Update editable transaction fields (company, amount, transaction_type) - `POST /api/v1/transactions/{tx_id}/ignore` — Toggle the ignored flag on a transaction - `POST /api/v1/transactions/{tx_id}/review` — Mark a transaction's category as manually reviewed ## categories - `GET /api/v1/categories` — List predefined categories ## summary - `GET /api/v1/summary` — Monthly spending summary with previous-month comparison - `GET /api/v1/summary/trend` — Multi-month spending trend (2-12 months) ## budget - `GET /api/v1/budget/config` — Get budget targets and groups for a year - `PUT /api/v1/budget/config` — Replace budget targets and groups for a year - `GET /api/v1/budget/historical-averages` — Per-category historical monthly averages over the last N months - `GET /api/v1/budget/status` — YTD budget pace status with per-category and per-group breakdown ## insights - `GET /api/v1/insights/context` — Raw context dict the AI briefing uses (for inspection / external automation) - `POST /api/v1/insights/generate` — Kick off background AI generation of a monthly spending briefing - `GET /api/v1/insights/saved` — List saved insight briefings for a month - `GET /api/v1/insights/saved/{insight_id}` — Get the markdown content of a saved insight briefing - `GET /api/v1/insights/status` — Background generation status (idle/running/error) ## overrides - `GET /api/v1/overrides` — List all category override rules - `POST /api/v1/overrides/consolidate` — Atomically replace member overrides with a single canonical override - `GET /api/v1/overrides/duplicates` — List override groups sharing a normalized merchant key - `GET /api/v1/overrides/match` — Preview the tiered category resolver against current overrides + aliases - `GET /api/v1/overrides/suggestions` — Suggest new override rules from manual category corrections - `POST /api/v1/overrides/suggestions/dismissed` — Dismiss an override suggestion (it stops surfacing until a newer correction) - `DELETE /api/v1/overrides/suggestions/dismissed/{key}` — Reverse a previous dismissal (suggestion may resurface) - `PUT /api/v1/overrides/{company}` — Pin a company → category override - `DELETE /api/v1/overrides/{company}` — Remove a company → category override ## ignore-rules - `GET /api/v1/ignore-rules` — List all merchant auto-ignore rules - `POST /api/v1/ignore-rules` — Add a merchant auto-ignore rule - `POST /api/v1/ignore-rules/apply` — Backfill Ignored on existing transactions matching a rule (or all rules) - `GET /api/v1/ignore-rules/suggestions` — Suggest ignore rules from merchants you habitually ignore by hand - `GET /api/v1/ignore-rules/suggestions/dismissed` — List dismissed ignore-rule suggestions, newest first - `POST /api/v1/ignore-rules/suggestions/dismissed` — Dismiss a suggested merchant so it stops being surfaced - `DELETE /api/v1/ignore-rules/suggestions/dismissed/{merchant}` — Reverse a suggestion dismissal so the merchant may resurface - `DELETE /api/v1/ignore-rules/{pattern}` — Remove a merchant auto-ignore rule ## category-management - `POST /api/v1/categories` — Add a category, optionally placing it in a budget group - `GET /api/v1/categories/icons` — Map of user-set category icon overrides - `PUT /api/v1/categories/icons` — Set or update a category's icon override - `DELETE /api/v1/categories/icons` — Remove a category's icon override (revert to default) - `GET /api/v1/categories/managed` — List categories with their group memberships and version - `DELETE /api/v1/categories/{name}` — Delete a category, optionally reassigning its transactions - `PUT /api/v1/categories/{name}/group` — Move a category to a different budget group, or remove from all groups - `GET /api/v1/categories/{name}/usage` — Counts of transactions, overrides, and budget membership for a category - `PUT /api/v1/categories/{old_name}` — Rename a category and cascade across overrides, budget, transactions ## groups - `GET /api/v1/groups` — Get budget category groups for a year - `PUT /api/v1/groups` — Replace budget category groups for a year ## search - `GET /api/v1/transactions/export` — Stream filtered transactions as CSV - `GET /api/v1/transactions/search` — Cross-month transaction search with filters - `POST /api/v1/transactions/search-by-filter` — Cross-month transaction search with array-shaped filters (POST body) ## statements - `POST /api/v1/statements/import` — Execute import/enrich/update actions for statement transactions - `POST /api/v1/statements/upload` — Upload a PDF statement, parse, and reconcile against transactions - `POST /api/v1/statements/{statement_id}/reparse` — Re-parse a statement PDF, preserving prior user edits ## statements-crud - `GET /api/v1/statements` — List all uploaded statements - `GET /api/v1/statements/{statement_id}` — Get a statement with all its parsed transactions - `DELETE /api/v1/statements/{statement_id}` — Delete a statement and its parsed transactions - `GET /api/v1/statements/{statement_id}/download` — Download the original PDF for a statement - `PATCH /api/v1/statements/{statement_id}/transactions` — Bulk-update statement transaction actions - `PATCH /api/v1/statements/{statement_id}/transactions/{row_id}` — Update a single statement transaction's action (auto-save), keyed by row_id ## attachments - `GET /api/v1/attachments` — List attachments, optionally filtered by link status and kind - `POST /api/v1/attachments` — Upload a receipt or document, optionally linked to a transaction - `DELETE /api/v1/attachments/{attachment_id}` — Delete an attachment and its file from disk - `GET /api/v1/attachments/{attachment_id}/candidates` — Rank the transactions a parsed receipt might explain - `GET /api/v1/attachments/{attachment_id}/file` — Download the attachment file, served inline with our stored content-type - `POST /api/v1/attachments/{attachment_id}/link` — Link an attachment to a transaction, or unlink it - `POST /api/v1/attachments/{attachment_id}/parse` — Parse a receipt attachment with the configured AI provider - `GET /api/v1/transactions/{tx_id}/attachments` — List attachments linked to a transaction ## income-statement - `GET /api/v1/income-statement` — Annual income vs. expenses view with projection and savings rate ## tax - `GET /api/v1/tax-pack` — Calendar-year tax pack: claim-line totals with per-transaction evidence - `GET /api/v1/tax-pack/export` — Download the year's tax pack as a zip of CSVs plus evidence files - `POST /api/v1/tax-pack/items` — Force a transaction into a claim line, or exclude it from one - `DELETE /api/v1/tax-pack/items/{tx_id}` — Clear a per-transaction tax override - `GET /api/v1/tax-pack/lines` — Selectable claim lines for the include-override picker ## merchants - `GET /api/v1/merchants/intelligence` — Recurring-charge detection, price-change alerts, committed burn rate ## journal - `GET /api/v1/journal` — Day-grouped transaction timeline for a month ## daily-summaries - `GET /api/v1/journal/summaries` — List saved AI day summaries for a month - `POST /api/v1/journal/summaries/generate` — Kick off background AI generation of day summaries for a month - `GET /api/v1/journal/summaries/status` — Background generation status (idle/running/error) ## merchant-aliases - `GET /api/v1/merchant-aliases` — List all merchant aliases (raw → canonical mappings) - `PUT /api/v1/merchant-aliases/{raw_name}` — Create or update a merchant alias (raw → canonical) - `DELETE /api/v1/merchant-aliases/{raw_name}` — Delete a merchant alias by raw name ## data - `POST /api/v1/data/export` — Stream a full-data backup zip (transactions + config) - `POST /api/v1/data/import/commit` — Apply a previously-previewed import using the chosen strategy - `POST /api/v1/data/import/preview` — Stage a backup upload and return a dry-run dedup summary - `GET /api/v1/data/s3-backup-status` — Read the S3 attachment-backup config and last-run state ## parse-failures - `GET /api/v1/parse-failures` — List quarantined parse failures (summaries only, no email body) - `POST /api/v1/parse-failures/retry-all` — Re-run the deterministic parsers across a whole institution's quarantined backlog - `GET /api/v1/parse-failures/{failure_id}` — Get a single quarantined parse failure, including the email body - `DELETE /api/v1/parse-failures/{failure_id}` — Dismiss a quarantined parse failure - `POST /api/v1/parse-failures/{failure_id}/resolve` — Record a hand-entered transaction for a quarantined email and mark it resolved - `POST /api/v1/parse-failures/{failure_id}/retry` — Re-run the deterministic parsers against a quarantined email - `POST /api/v1/parse-failures/{failure_id}/to-fixture` — Write a scrubbed test-fixture pair from a quarantined email (dev checkout only) ## auth - `POST /api/v1/auth/chatgpt/disconnect` — Disconnect a previously-linked ChatGPT account - `POST /api/v1/auth/chatgpt/start` — Begin the ChatGPT device-code sign-in via the Codex CLI - `GET /api/v1/auth/chatgpt/status` — Poll the ChatGPT sign-in state ## webapp-auth - `POST /api/v1/auth/login` — Issue a webapp session cookie - `POST /api/v1/auth/logout` — Clear the caller's webapp session cookie - `POST /api/v1/auth/set-password` — Set or change the webapp password (TOFU on first set) - `POST /api/v1/auth/sign-out-all` — Bump session_version, invalidating all existing cookies ## health - `GET /api/v1/health` — Liveness probe with last-activity snapshot ## coverage - `GET /api/v1/coverage` — Per-institution bank-alert cadence and passive email-capture rate ## activity - `GET /api/v1/activity` — List agent activity ledger entries, newest first - `POST /api/v1/activity/{entry_id}/revert` — Revert a single ledger entry, restoring the resource's prior state - `GET /api/v1/whoami` — Resolve the current caller's identity (token, session, TOFU, or dev-bypass) --- # More guides Source: https://docs.gettidings.com/guides.md > Contributor-facing guides that live in the repository and open on GitHub. These contributor-facing guides live in the repository's `docs/guides/` directory and open on GitHub. They cover operational and tooling topics for people working on Tidings itself, so most self-hosters won't need them. - [Environment management](https://github.com/tvhahn/tidings/blob/main/docs/guides/environment-management.md) — uv setup, Python environment, and dependency management. - [Dev surfaces](https://github.com/tvhahn/tidings/blob/main/docs/guides/dev-surfaces.md) — The frontend port map: the five dev surfaces, port-override flags, and BrowserRouter basename rules. - [DevContainer startup](https://github.com/tvhahn/tidings/blob/main/docs/guides/devcontainer-startup.md) — DevContainer lifecycle and environment priming. - [Slash commands](https://github.com/tvhahn/tidings/blob/main/docs/guides/slash-commands.md) — Category management, test review, spending insights, and doc review slash commands. - [Static-hosted demo](https://github.com/tvhahn/tidings/blob/main/docs/guides/static-hosted-demo.md) — Building, previewing, and regenerating fixtures for the static Cloudflare Pages demo. - [DynamoDB cost analysis](https://github.com/tvhahn/tidings/blob/main/docs/guides/dynamodb-cost-analysis.md) — The DynamoDB cost model and sizing assumptions. - [Releases](https://github.com/tvhahn/tidings/blob/main/docs/guides/releases.md) — Versioning policy, branching model, and the step-by-step release ritual.