Jobs Add asks a period first; systemd unit split, Docker log driver, install docs, movers fixes

.github/workflows/ci.yml

@@ -126,19 +126,19 @@ jobs:
           wait_for 'cache: flushed'
           docker rm -f finbot
       - if: steps.changes.outputs.docs_only != 'true'
-        name: A mistake in the startup file stops the container before the bot starts
+        name: A mistake in the startup env stops the container before the bot starts
         run: |
           printf 'telegramBotTokn=x\n' > "$RUNNER_TEMP/bad.env"
           if docker run --rm --read-only --tmpfs /tmp -e DATA_DIR=/tmp \
               -v "$RUNNER_TEMP/bad.env:/run/finbot.env:ro" finbot > "$RUNNER_TEMP/out" 2>&1; then
             cat "$RUNNER_TEMP/out"
-            echo "::error::the container started with a mistake in its startup file"
+            echo "::error::the container started with a mistake in its startup env"
             exit 1
           fi
           cat "$RUNNER_TEMP/out"
           grep -q 'unknown key telegramBotTokn' "$RUNNER_TEMP/out"
           if grep -q 'finbot is running' "$RUNNER_TEMP/out"; then
-            echo "::error::the bot started with a mistake in its startup file"
+            echo "::error::the bot started with a mistake in its startup env"
             exit 1
           fi
 

Dockerfile

@@ -1,5 +1,5 @@
 # finbot in a container, on Debian with Debian's own python3 packages (no pip), as on a Debian host. Every optional
-# library is in, so the image runs every network in either mode. The startup file and DATA_DIR stay outside the image:
+# library is in, so the image runs every network in either mode. The startup env and DATA_DIR stay outside the image:
 # see compose.yaml and doc/docker.md.
 FROM debian:trixie-slim
 
@@ -23,5 +23,5 @@ ARG GIT_COMMIT=unknown
 ENV FINBOT_COMMIT=$GIT_COMMIT PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 DATA_DIR=/data HOME=/tmp
 USER finbot
 
-# each start: copy the defaults missing from /data, check the startup file, and only then start the bot
+# each start: copy the defaults missing from /data, check the startup env, and only then start the bot
 CMD ["sh", "-c", "util/sync_defaults.py --apply && util/check_config.py /run/finbot.env && exec ./bot.py"]

README.md

@@ -22,34 +22,33 @@ Run finbot with Docker or with Python.
 
 ### With Docker
 On a Linux host with Docker:
-1. `git clone https://github.com/robdevops/finbot /opt/finbot`
-2. `sudo /opt/finbot/util/docker_setup.sh`
-3. Set the keys of each network you use in `/opt/finbot.env` (below)
+1. `sudo git clone https://github.com/robdevops/finbot /opt/finbot`
+2. `sudo /opt/finbot/util/docker_setup.sh`, which makes `/etc/finbot.env` from its template
+3. Set the keys of each network you use in `/etc/finbot.env` (below)
 4. `cd /opt/finbot && sudo docker compose up -d --build`
-5. `journalctl CONTAINER_NAME=finbot -f`
+5. `sudo docker compose logs -f`
 
 ### With Python
-On Linux, macOS or WSL, with Python 3.12 or later:
-1. `git clone https://github.com/robdevops/finbot ~/finbot && cd ~/finbot`
-2. `python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-optional.txt` (on Debian and
-   Ubuntu, `sudo apt install python3-venv` first)
-3. `cp defaults/finbot.env ~/finbot.env && chmod 600 ~/finbot.env`, then set the keys of each network you use in it
-   (below)
-4. `.venv/bin/python util/check_config.py ~/finbot.env`
-5. `(set -a && . ~/finbot.env && exec .venv/bin/python bot.py)`, which logs to the terminal until Ctrl-C
+On Linux, macOS or WSL, with Python 3.12 or later and its `venv` module:
+1. `sudo git clone https://github.com/robdevops/finbot /opt/finbot && cd /opt/finbot`
+2. `sudo python3 -m venv .venv && sudo .venv/bin/pip install -r requirements.txt -r requirements-optional.txt`
+3. `sudo cp defaults/finbot.env /etc/finbot.env && sudo chmod 600 /etc/finbot.env`, then set the keys of each network
+   you use in it (below)
+4. `sudo .venv/bin/python util/check_config.py /etc/finbot.env`
+5. `sudo sh -c 'set -a && . /etc/finbot.env && exec .venv/bin/python bot.py'`, which logs to the terminal until Ctrl-C
 
 To run it as a service, under systemd on a Debian host, see [doc/install.md](doc/install.md).
 
 ### The chat networks
 By default, finbot fetches its updates from each network itself (pull mode), which is what this quick start guide covers.
 Alternatively, if you prefer to run an HTTPS gateway (a reverse proxy on a domain with a valid certificate), push mode
-subscribes the bot to the updates each network pushes to it (see each guide's _push mode_).
-* **Telegram:** create the bot with [@BotFather](https://telegram.me/BotFather), set `telegramBotToken`, restart
+subscribes the bot to the updates each network pushes to it.
+* **Telegram:** create the bot with [@BotFather](https://telegram.me/BotFather), set `telegramBotToken` in the env file, restart
   finbot ([guide](doc/telegram.md)).
 * **Slack:** create the app from the manifest that `util/slack_manifest.py` prints, add an app-level token, set
-  `slackBotToken` and `slackAppToken`, restart finbot ([guide](doc/slack.md)).
-* **Discord:** create the application and its bot, turn on its Message Content intent, set `discordBotToken`, restart
-  finbot ([guide](doc/discord.md)).
+  `slackBotToken` and `slackAppToken` in the env file, restart finbot ([guide](doc/slack.md)).
+* **Discord:** create the application and its bot, turn on its Message Content intent, set `discordBotToken` in the env file,
+  restart finbot ([guide](doc/discord.md)).
 
 Then send `/help` in a chat with the bot. Commands work as plain messages too (`.help`), which are quicker to type on a
 phone ([Commands](doc/commands.md#supported-commands)). From its first command, each chat has the default scheduled

compose.yaml

@@ -1,6 +1,6 @@
-# finbot with Docker Compose (doc/docker.md). The configuration lives beside the checkout, outside the image and the
-# container: ../finbot.env, the startup file, and ../finbot-data, DATA_DIR (settings.conf, state and cache), both
-# made by util/docker_setup.sh. Build and start: docker compose up -d --build
+# finbot with Docker Compose (doc/docker.md). The configuration lives outside the image and the container:
+# /etc/finbot.env, the startup env, and ../finbot-data beside the checkout, DATA_DIR (settings.conf, state and cache),
+# both made by util/docker_setup.sh. Build and start: docker compose up -d --build
 services:
   finbot:
     build:
@@ -10,17 +10,17 @@ services:
     image: finbot
     container_name: finbot
     restart: unless-stopped
-    env_file: ../finbot.env
+    env_file: /etc/finbot.env
     environment:
       DATA_DIR: /data
       ip: 0.0.0.0 # push mode listens inside the container: publish its port below
     volumes:
       - ../finbot-data:/data
-      - ../finbot.env:/run/finbot.env:ro # checked before the bot starts
+      - /etc/finbot.env:/run/finbot.env:ro # checked before the bot starts
     # ports: # push mode only, on the host's loopback, behind your reverse proxy
     #   - 127.0.0.1:5000:5000
     logging:
-      driver: ${FINBOT_LOG_DRIVER:-journald} # json-file where the host has no journald
+      driver: ${FINBOT_LOG_DRIVER:-local} # journald where util/docker_setup.sh found one (it writes .env); else local
     # hardening: the bot needs none of what these take away
     user: "10001:10001"
     read_only: true

defaults/finbot.env

@@ -1,15 +1,15 @@
-# finbot.env: THE STARTUP FILE. Where finbot runs, and who it talks to.
+# finbot.env: THE STARTUP ENV. Where finbot runs, and who it talks to.
 # The bot never reads this file. Your launcher loads it into the bot's environment, and a change needs a RESTART:
-#   systemd:  EnvironmentFile=/opt/finbot.env             (in finbot.service)
-#   docker:   --env-file /opt/finbot.env
-#   init.d:   set -a; . /opt/finbot.env; set +a; ./bot.py
+#   systemd:  EnvironmentFile=/etc/finbot.env             (in finbot.service)
+#   docker:   --env-file /etc/finbot.env
+#   init.d:   set -a; . /etc/finbot.env; set +a; ./bot.py
 # Keep it private: chmod 600, owned by root:root. It holds the credentials (check_config.py warns if not).
 # Flat KEY=value lines: no spaces around "=", no quotes (docker keeps quotes literally), no trailing comments; comments
 # on their own line.
 # Everything else (thresholds, windows, display, caching, ...) is in settings.conf, which the bot reloads without a
 # restart.
 # A setting key in this file is an error: finbot refuses to start. Lint both files with:
-#   util/check_config.py /opt/finbot.env
+#   util/check_config.py /etc/finbot.env
 
 # Where state and cache live. Leave unset under systemd with StateDirectory=finbot (it uses $STATE_DIRECTORY,
 # /var/lib/finbot, the only directory the shipped unit can write).

doc/commands.md

@@ -38,7 +38,7 @@ Where:
 /holdings [portfolio]
 /jobs [upcoming|system]
 /jobs enable|disable|delete|run [number]
-/jobs add report schedule
+/jobs add report [argument] [schedule]
 /jobs add reminder schedule "text"
 /marketcap [symbol|top|bottom]
 /movers [percent|top|bottom|session|premarket] [period]
@@ -156,7 +156,7 @@ demote_volatile=True
 
 Job (see [Scheduling](scheduling.md)), such as `/jobs add movers close weekdays at 4:30pm New York`:
 ```
-movers premarket|midsession|interday|close|DAYS
+movers premarket|midsession|interday|close|PERIOD
 ```
 When scheduled, the mode is its argument:
 * If `premarket` is passed, it only reports on pre/post market price movements.
@@ -204,7 +204,7 @@ past_days=28
 
 Job (see [Scheduling](scheduling.md)), such as `/jobs add performance weekly`:
 ```
-performance [DAYS]
+performance [PERIOD]
 ```
 
 Command:

doc/configuration.md

@@ -4,14 +4,14 @@ Configuration is split into two files -- finbot.env for startup environment vari
 
 Copy the shipped template:
 ```
-sudo cp /opt/finbot/defaults/finbot.env /opt/finbot.env && sudo chmod 600 /opt/finbot.env && sudo chown root:root /opt/finbot.env
+sudo cp /opt/finbot/defaults/finbot.env /etc/finbot.env && sudo chmod 600 /etc/finbot.env && sudo chown root:root /etc/finbot.env
 ```
 
 ## `finbot.env`
 **Load `finbot.env` with your launcher:**
-* **systemd:** `EnvironmentFile=/opt/finbot.env` in `finbot.service`, plus `ExecStartPre=+.../util/check_config.py /opt/finbot.env` for pre-flight lint.
+* **systemd:** `EnvironmentFile=/etc/finbot.env` in `finbot.service`, plus `ExecStartPre=+.../util/check_config.py /etc/finbot.env` for pre-flight lint.
 * **Docker:** `compose.yaml` loads it, and checks it before the bot starts (see [docker.md](docker.md)). Write the values without quotes.
-* **init.d / shell:** `util/check_config.py /opt/finbot.env && set -a && . /opt/finbot.env && set +a && ./bot.py` (`set -a` is what exports the values to the bot).
+* **init.d / shell:** `util/check_config.py /etc/finbot.env && set -a && . /etc/finbot.env && set +a && ./bot.py` (`set -a` is what exports the values to the bot).
 * **Scheduled jobs:** they are run by `bot.py` (see [Scheduling](scheduling.md)), so they run on the same settings. For a one-off run, send `/jobs run` in the chat.
 
 The startup keys, read by your launcher. A change needs a restart. Blank means unset. Each network's guide says which
@@ -86,7 +86,7 @@ The administrators of the chat a person is in are admins there with no setting:
 
 A key in the wrong file, or one that is not listed, stops finbot starting; on a reload it is reported and nothing is applied. A key finbot no longer reads (`alliterate`, `chunk_maxlines`) is ignored, and the log at start says to delete it.
 
-edit `settings.conf`, then reload if running. The service log shows what changed (`config: future_days 30 -> 45`). If the file has a mistake, the reload reports it (`ERROR config: ...`), applies nothing and leaves finbot running on its previous settings. Check an edit before reloading with `sudo util/check_config.py /opt/finbot.env` (under systemd it reads the service's `settings.conf`). Edit `finbot.env`, then restart: a running bot cannot see it change.
+edit `settings.conf`, then reload if running. The service log shows what changed (`config: future_days 30 -> 45`). If the file has a mistake, the reload reports it (`ERROR config: ...`), applies nothing and leaves finbot running on its previous settings. Check an edit before reloading with `sudo util/check_config.py /etc/finbot.env` (under systemd it reads the service's `settings.conf`). Edit `finbot.env`, then restart: a running bot cannot see it change.
 
 **Upgrading from a version before 16 Oct 2026?** Start commit `ff52f2b` once first, as [Scheduling](scheduling.md#from-an-earlier-version) says: it splits one old `.env` (`util/split_env.py`, which it asks for) and renames `preferences.conf` to `settings.conf`, which later versions no longer do.
 
@@ -95,7 +95,7 @@ All state and cache lives in one data directory: `DATA_DIR` if set, else systemd
 Reference data (`finbot_adr.json`, `finbot_sws_*.json`) ships in `defaults/`. A file of the same name in the data directory takes precedence. Defaults are never copied or overwritten automatically; to see or pull in updates run:
 ```
 util/sync_defaults.py          # dry run
-util/sync_defaults.py --apply  # copy missing files (including a starter settings.conf)
+util/sync_defaults.py --apply  # copy missing files (including the settings.conf template)
 util/sync_defaults.py --force  # also replace changed files, keeping .bak copies
 util/sync_defaults.py --dest /srv/finbot --apply  # target another directory instead of the data directory
 util/sync_defaults.py --dest /srv/finbot --create --apply  # and make it (mode 750) if it does not exist yet

doc/docker.md

@@ -2,28 +2,28 @@
 
 finbot runs in a container as it runs on a Debian host: the image is `debian:trixie-slim` with Debian's own `python3-*`
 packages, every optional library included, so it runs every network in either mode. The image holds only the code.
-The configuration lives on the host, beside the checkout, so a rebuild or an update never touches it.
+The configuration lives on the host, outside the image (the startup env in `/etc`, the data beside the Git checkout), so a rebuild or an update never touches it.
 
 ## Quick start
 
 1. `git clone https://github.com/robdevops/finbot /opt/finbot`
 2. `sudo /opt/finbot/util/docker_setup.sh`
-3. Set `telegramBotToken` in `/opt/finbot.env` (or another network's keys: see the [Telegram](telegram.md),
-   [Slack](slack.md) and [Discord](discord.md) guides).
+3. Set your chatbot's API key in `/etc/finbot.env` (`telegramBotToken` for Telegram; see the
+   [Telegram](telegram.md), [Slack](slack.md) and [Discord](discord.md) guides).
 4. `cd /opt/finbot && sudo docker compose up -d --build`
-5. `journalctl CONTAINER_NAME=finbot -f`
+5. `sudo docker compose logs -f`
 
 ## The layout on the host
 
 | Path | What | In the container |
 |---|---|---|
-| `/opt/finbot` | the checkout, with `Dockerfile` and `compose.yaml` | the image's `/opt/finbot`, copied at build |
-| `/opt/finbot.env` | the startup file: the tokens, and the mode of each network | the environment, and `/run/finbot.env`, read-only, for the check |
+| `/opt/finbot` | the Git checkout, with `Dockerfile` and `compose.yaml` | the image's `/opt/finbot`, copied at build |
+| `/etc/finbot.env` | the startup env: the tokens, and the mode of each network | the environment, and `/run/finbot.env`, read-only, for the check |
 | `/opt/finbot-data` | `DATA_DIR`: `settings.conf`, each chat's jobs and watchlist, the records, the cache | `/data` |
 
-`compose.yaml` finds the other two beside the checkout (`../finbot.env`, `../finbot-data`): clone it elsewhere and
-they move with it. The startup file stays out of the image because it holds the secrets, and the data dir because it
-must outlast every container.
+`compose.yaml` reads the startup env at `/etc/finbot.env` and finds the data dir beside the Git checkout
+(`../finbot-data`): clone it elsewhere and the data dir moves with it. The startup env stays out of the image because
+it holds the secrets, and the data dir because it must outlast every container..
 
 ## What `util/docker_setup.sh` does
 
@@ -32,18 +32,18 @@ Run it once, as root; it is safe to run again. It:
    the shipped defaults into it: `settings.conf` and the data files. It never replaces a file you
    have changed. The script needs only the standard library, so any `python3` on the host runs it.
 2. gives the data dir to the container's user: `chown -R 10001:10001 /opt/finbot-data`.
-3. copies `defaults/finbot.env` to `/opt/finbot.env` if there is none, owned by `root:10001`, mode 640: root and the
+3. copies `defaults/finbot.env` to `/etc/finbot.env` if there is none, owned by `root:10001`, mode 640: root and the
    container can read it, nobody else.
 
 By hand, that is:
 ```
 sudo python3 /opt/finbot/util/sync_defaults.py --create --apply --dest /opt/finbot-data
 sudo chown -R 10001:10001 /opt/finbot-data
-sudo cp /opt/finbot/defaults/finbot.env /opt/finbot.env
-sudo chown root:10001 /opt/finbot.env && sudo chmod 640 /opt/finbot.env
+sudo cp /opt/finbot/defaults/finbot.env /etc/finbot.env
+sudo chown root:10001 /etc/finbot.env && sudo chmod 640 /etc/finbot.env
 ```
 
-The startup file is `KEY=value` lines, as for systemd; write the values without quotes.
+The startup env is `KEY=value` lines, as for systemd; write the values without quotes.
 
 ## Build and start
 
@@ -55,18 +55,26 @@ cd /opt/finbot && sudo GIT_COMMIT=$(git rev-parse HEAD) docker compose up -d --b
 
 Each start of the container, in order:
 1. `util/sync_defaults.py --apply` copies into `/data` any default it lacks, such as a data file added by an update;
-2. `util/check_config.py /run/finbot.env` checks the startup file and `settings.conf`. A mistake stops the start
+2. `util/check_config.py /run/finbot.env` checks the startup env and `settings.conf`. A mistake stops the start
    here, and the log says which key and why;
 3. the bot starts.
 
 ## Logs
 
-The log goes to the host's journal:
+```
+sudo docker compose logs -f
+```
+
+Where journald runs, `util/docker_setup.sh` sends the log to the host's journal, which keeps it when you rebuild. Read
+it there too:
 ```
 journalctl CONTAINER_NAME=finbot -f
 ```
-On a host without journald, set `FINBOT_LOG_DRIVER=json-file` (in the environment of `docker compose`, or in a `.env`
-file beside `compose.yaml`), and read it with `sudo docker compose logs -f`.
+
+Elsewhere, Docker keeps the log itself (its `local` driver, which rotates it), and `docker compose logs` is the way to
+read it. That log is lost when the container is rebuilt.
+
+To choose, set `FINBOT_LOG_DRIVER` (`journald`, `local` or `json-file`) in a `.env` file beside `compose.yaml`.
 
 ## Reload, flush, update
 
@@ -76,7 +84,7 @@ sudo docker compose kill -s USR1 finbot    # flush the cache, and fill it again
 sudo docker compose exec finbot util/flush.py --after 24h    # flush the files of the last day (see caching.md)
 cd /opt/finbot && sudo git pull && sudo GIT_COMMIT=$(git rev-parse HEAD) docker compose up -d --build
 ```
-A change to `/opt/finbot.env` needs a restart: `sudo docker compose up -d`.
+A change to `/etc/finbot.env` needs a restart: `sudo docker compose up -d`.
 
 ## Push mode
 
@@ -113,14 +121,15 @@ sudo systemctl disable --now finbot
 sudo cp -a /var/lib/finbot/. /opt/finbot-data/    # or /opt/finbot/var/. for the default DATA_DIR
 sudo chown -R 10001:10001 /opt/finbot-data
 ```
-then `util/docker_setup.sh` (it keeps your files) and start as above. The startup file can stay at `/opt/finbot.env`.
+then `util/docker_setup.sh` (it keeps your files) and start as above.
 
 ## Troubleshooting
 
 * **The container stops at once and keeps restarting:** the log says why. `ERROR config: ... unknown key ...` is a
-  mistake in `/opt/finbot.env` or `settings.conf`; fix it and `sudo docker compose up -d`.
+  mistake in `/etc/finbot.env` or `settings.conf`; fix it and `sudo docker compose up -d`.
 * **`PermissionError` on `/data`:** the data dir is not the container user's: `sudo chown -R 10001:10001
   /opt/finbot-data`.
 * **`/data does not exist: --create makes it`:** the data dir is not mounted: `compose.yaml` mounts
   `../finbot-data`; run `util/docker_setup.sh` to make it.
-* **No log in the journal:** the host has no journald; use `FINBOT_LOG_DRIVER=json-file` (see [Logs](#logs)).
+* **No log in `journalctl`:** the container is not using journald; read it with `sudo docker compose logs -f` (see
+  [Logs](#logs)).