Skip to content

Split replies by the length Telegram counts, not the raw HTML - #51

Open
robdevops wants to merge 44 commits into
mainfrom
claude/compassionate-heisenberg-pz23nq
Open

Split replies by the length Telegram counts, not the raw HTML#51
robdevops wants to merge 44 commits into
mainfrom
claude/compassionate-heisenberg-pz23nq

Conversation

@robdevops

@robdevops robdevops commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Replies full of links (DeepSeek's cited answers especially) were split into two messages even when they were well under Telegram's limit.

Why

Telegram's 4096-character message limit (1024 for captions) counts the text after the HTML is parsed: a link counts only its label. split_html counted the raw HTML, so every ticker link and [n] citation URL (often 100–250 characters) used up room. A reply of ~2,500 visible characters with 15 citations and 15 ticker links went over 3,904 raw characters and was cut at the last newline.

What

  • textfmt.visible_len: length as Telegram counts it. Tags count nothing; < > & " and numeric entities count as their character, while any other &name; (which Telegram shows as typed) counts in full; emoji count as two UTF-16 units. split_html now cuts by it, keeping its newline, tag and entity rules. The default size is MAX_TG_MESSAGE (4000).
  • split_html never cuts inside a tag. A cut stops before tags that follow the last character that fits, so they open the next message. A raw & in an href's query string no longer counts as an entity to back away from.
  • plain_text(text, limit): the plain-text fallback, used when Telegram rejects the HTML, drops the URLs when "label (url)" would push a message or caption over Telegram's limit. Used in send.py, commands.edit_in_place and the chart caption swaps.
  • filings.build: the "fits in one message" check uses visible_len, so /events isn't sent to the model to shorten when it already fits.
  • Tests: an autouse fixture restores quote_links.US_EXCHANGES. The known-tickers tests leaked it, and the Google Finance link test failed about one run in ten under xdist.
  • Docs: docs/how-it-works.md, CLAUDE.md map.

The -s ours merge commit brings in the squash-merged #50 history (identical tree), so the branch could be pushed without a force-push.

Tests

make check (Python 3.13.5): 567 passed; 25 repeated runs had no failures. New tests:

  • a citation-heavy reply over 4096 raw but under it visibly stays one message;
  • emoji count as two;
  • long-URL links still split at the visible limit with balanced tags;
  • no cut inside a link whose href has a raw & (sizes swept across the tag boundary);
  • undecoded entities count in full;
  • plain_text drops URLs past the limit.

Not checked against live Telegram: there's no access from the sandbox.

🤖 Generated with Claude Code

https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw

claude added 30 commits November 2, 2026 23:43
@claude
Charts: draw superscripts and other glyphs the built-in font lacks as…
85ceba3 
… plain stand-ins

Pillow's default font has no ² ³ − → € etc., so they drew as boxes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Pie legend: whole-number values instead of percentages (the donut sho…
2956087 
…ws the share)

Falls back to shares when a value would round to zero.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Pie: unit in legend and total from the title's bracket; subtitle matc…
7cb0ac4 
…hes value legend

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Pie: larger donut, donut and legend centred together
b6280b8 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Merge main into branch after the squash-merge of #48
bb03d55
@claude
OpenRouter: end a chat answer with its web-search sources when it lin…
c3b375b 
…ks none

The url_citation annotations were only counted; now they are kept on the Step and Answer
and ask() appends a Sources line (max 3) to chat answers that carry no link of their own.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
OpenRouter: cite sources as [n] links after the cited sentence
4307dc8 
Positions from the url_citation annotations are applied to the raw reply before any
formatting; the Sources line stays as the fallback when no marker could be placed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Config: drop LOG_WIDTH, rename BOT_TZ/HISTORY_LIMIT/TELEGRAM_PRELOAD_…
3cc0e2b 
…CHAT; log user input as two words

- LOG_WIDTH removed: tool-name lists are logged in full.
- BOT_TZ -> TZ, HISTORY_LIMIT -> HISTORY_MAX (default 10, and a real maximum: the
  stepped window now holds max/2+1 .. max messages), TELEGRAM_PRELOAD_CHAT ->
  TELEGRAM_IMAGE_PRELOAD_CHATID. The old names are not read any more.
- User input in the log is cut to its first two words and one ellipsis character; the
  other log-line truncations (error snippets, failed HTML) are gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Docs: ADMIN_ONLY says what on and off do; configuration tables get a …
02f8cbb 
…Default column

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
OpenRouter default model: deepseek/deepseek-v4.1-flash-1017
cfd93f8 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
SEARCH_MODEL defaults to MODEL with :online added
1412dc1 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
OpenRouter: debug-log what the citations carry and how many [n] marke…
07a0624 
…rs were placed

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Cite web-search sources as [n] after the sentence each snippet fits best
f23ca7f 
OpenRouter's url_citation positions are 0 and the search runs in an earlier round than the
answer, so the position-based marker is replaced by matching the answer's sentences to each
source's snippet (figures weigh most, words every snippet shares count for less).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Ask the model for inline [n] source links; drop the Sources list
4806c05 
The chat system prompt (OpenRouter only) tells the model to cite a sentence that rests on a search
result as [<a href=URL>n</a>]. Its links are kept only for pages the search returned and renumbered;
without them the snippet-overlap markers apply. The trailing Sources line is gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Cite: debug-log why no [n] markers were added and which model links w…
dbfe333 
…ere dropped

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
NOT_TICKERS env var: extra words never linked as tickers
be66672 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Docs: configuration.md ends after the Advanced and optional table
b371c23 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Cite: add [n] for the pages the model did not cite, then number all i…
6a4b748 
…n order

The model cited 3 of ~10 pages the search returned; the overlap pass now runs for the rest even when
the model wrote markers of its own.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Ticker links: only marked forms and known tickers, never bare words
9347579 
Bare capitalised words (GAAP, EBITDA, AFR, GM, UPS) kept getting Yahoo links and the stoplist never
ended; ASX:NXT linked to the US NXT. Replies now link only $NVDA, NXT.AX, ASX:NXT (mapped to
Yahoo's suffix), Name (TICKER), Sharesight's NXT (ASX), ^GSPC and BTC-USD, plus bare words in a
known set: the S&P 500 and NASDAQ-100 (from Wikipedia) and the US Sharesight holdings, refreshed
daily in the background and kept in kv, never fetched while answering. A bare code in brackets
links only if known, since it names no exchange. Holding news links the holdings' codes and a
movers explanation the listed codes with the list's exchange. The prompt asks for these forms.
The NOT_TICKERS env var is gone; the built-in list now only serves the tool gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Ticker links: fixes from an adversarial review
3e75889 
- An exchange suffix links on its own: ALL.AX (Aristocrat), ASX.AX, AI.PA were left plain because
  the code is also on the acronym list.
- A bracket that isn't a ticker ("NVDA (Q3 FY26)") no longer swallows the ticker before it.
- Share classes: BRK.B as the model writes it links to Yahoo's BRK-B.
- "AT&amp;T (T)" and "Procter &amp; Gamble (PG)" are bolded whole.
- Numeric holding codes (700 on HKEX) never link bare numbers, and keep their unsubscribe button.
- Index tickers that are also finance or tech acronyms (GM, UPS, IR, HD, USB, RF...) link only in a
  marked form, as single letters already did.
- A refresh part that fails (a page down, Sharesight not up yet) is tried again in 30 minutes,
  not 24 hours; a footnote in the table header no longer hides the ticker column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Known tickers: Wikipedia's action API with a contact User-Agent, and …
c361600 
…the ASX 200

Wikipedia answered 403 to the page fetches: its policy refuses programs whose User-Agent names no
contact. The lists now come through the action API (action=parse) with the repo URL in the
User-Agent, and a refusal logs Wikimedia's reason. The S&P/ASX 200 is added: its codes link bare
with .AX. Holdings on any known exchange count (NXT -> NXT.AX). A code in a US index and in the
ASX 200 isn't linked bare unless a holding settles it. ASX codes that are everyday words or
acronyms (GPT, WOW, ZIP, CAR...) need a marked form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Test the ASX 200 page layout (Code column, number codes like 360)
7891c66 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Known tickers: read the NASDAQ-100 from "List of NASDAQ-100 companies"
9810d08 
The "Nasdaq-100" article has no constituents table any more (0 tickers in production).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
FINANCE_LINK_PROVIDER: ticker links to Yahoo Finance (default) or Goo…
2aaf21f 
…gle Finance

Google takes CODE:EXCHANGE (NXT.AX -> NXT:ASX, 0700.HK -> 700:HKG); a US ticker names no exchange
(NASDAQ or NYSE), so it opens a Google Finance search. History compaction and the holding-news
mute codes read either provider's links back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
FINANCE_LINK_TARGET (was _PROVIDER); Google links name the US exchange
d7d9874 
Google Finance quotes are CODE:EXCHANGE (JPM:NYSE, SPCX:NASDAQ, BRK.A:NYSE). A US ticker's exchange
now comes from the text (NYSE:JPM, JPM (NYSE)), the Sharesight holdings, the NASDAQ-100 (all
NASDAQ) or the S&P 500 list, whose ticker links point at nyse.com or nasdaq.com; kept in kv with
the lists. Only a ticker none of these place opens a Google Finance search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
FINANCE_LINK_TARGET=tradingview; quote URLs move to lib/quote_links.py
ac16638 
TradingView pages are EXCHANGE-CODE (ASX-NXT, NYSE-BRK.A, NASDAQ-SPCX), with the US exchange from
the same sources as Google's; a code it can't place opens TradingView's page for the bare code.
The three sites' URL builders leave tickers.py (near its size limit) for their own module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
/usage: requests and cost per kind under each period
5ea6e16 
Shows what the daily holding-news check (and movers, DM buttons, chat commands, filings) costs
next to chat; every request was already recorded with its kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Menu lists /usage (was /tokens); docs name /usage first
59a5b5d 
/tokens and /credits still work. Telegram shows the menu commands in autocomplete, so admins now
see /usage there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Holding news: one daily search for everyone's holdings, split per person
8b46fb3 
Each recipient used to get a separate searched request, so a stock two people hold was searched
twice and every request paid its own overhead (holding news was the biggest cost in /usage).
The daily check now asks once for the union of everyone's unmuted holdings, with everyone's recent
items as the don't-repeat list, then gives each person the items about their own holdings; an item
naming no holding goes to everyone. /holdingnews still checks just the asker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Holding news: skip days, a window back to the last check, no repeated…
7a0962d 
… articles

SHARESIGHT_HOLDING_NEWS_SKIP_DAYS (e.g. sun,mon) skips the daily check on those weekdays. Each
check now searches back to the last one that got an answer (kv; at least 24 h, at most 7 days),
so skipped days, a busy provider or an outage leave no gap; the prompt and the message header
say how far (past 72h). Repeats: the don't-repeat list always reaches past the last check, and
an item citing an article already sent is dropped whatever the model says. The feature becomes a
package (holding_news/ with text.py) to stay under the module size limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
claude added 14 commits November 5, 2026 09:40
@claude
Rename ADMIN_CHAT_IDS to TELEGRAM_ADMIN_CHAT_IDS
86954d0 
The old name is no longer read: rename it in the service's environment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Portfolio names and holdings from Sharesight itself: drop PORTFOLIO_N…
344af40 
…AMES

The daily known-ticker refresh now reads every portfolio list_portfolios returns (skipping
consolidated views, which repeat the others): their holdings feed the known tickers and their names
route portfolio questions in the gate. Both are kept in kv, so a restart has them at once. The
PORTFOLIO_NAMES setting is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Merge the squash-merged history of #49 (identical tree to main)
51941d9
@claude
/future TICKER: analysts' low/average/high revenue and EPS as line ch…
10da1e8 
…arts

Two charts from Yahoo's get_analyst_estimates (revenue, EPS), each with Low, Average and High lines
fanning out from last fiscal year's actual through this and next fiscal year; the axis is labelled
with real fiscal years from the latest annual statement. Drawn with the existing make_chart, no
model. History uses the estimates' own year-ago figures, so EPS stays on the analysts' (adjusted)
basis rather than mixing in GAAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Holding news: a wider search allowance, no ETFs, no notes about the s…
3c18c23 
…earch

The combined daily check searches every holding in one request, and OpenRouter's search tool capped
the whole request at 20 results, so it ran out part-way (and said so in a "Note:" bullet sent to
everyone). Requests can now ask for a larger allowance (Request.search_results; OpenRouter's
max_total_results) and the daily check asks for 60. ETFs and funds, found by name, are left out
of the search. The prompt asks for news items only, and lines about the search are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Move to the yahoo-finance2 MCP server
98337f4 
The Yahoo server is now yahoo-finance2's own (npx -y -p yahoo-finance2 yahoo-finance-mcp): generic tools (chart,
quote, quoteSummary, fundamentalsTimeSeries, search, ...) returning raw yahoo-finance2 JSON. The old tool names are
gone, so what called them by name moves over:

- Ticker charts and their period buttons fetch prices with `chart` (period1 from the period; times in the exchange's
  own time zone; 1D and 5D keep the last trading days, so a weekend isn't empty). The fetch passes `_raw`, a new
  internal flag (no slimming, no size cap, stripped from the model's calls like `_keep_sales`), so a day of 5-minute
  bars isn't thinned to 60 points.
- /future reads quoteSummary's earningsTrend; the fiscal-year labels come from its end dates (a June year is right
  for ASX names), so the statements call is gone.
- Quotes are trimmed to the fields a model uses (about 870 characters each instead of 3,700), so a ten-stock quote
  fits one result instead of being cut after three.
- mcp_servers.json: the new command, an allowed_tools list (quote, quoteSummary, chart, fundamentalsTimeSeries,
  search: names that don't look read-only, and the rest left out), moduleOptions hidden, and a description naming
  the queryOptions the model needs. The Yahoo part of the prompt drops from about 1,120 to 725 tokens.
- Fixtures are real output from the new server; tools/capture.py and the docs follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
MCP: a server that can't be reached or started is one log line
d4e2a88 
A wrong URL, a refused, failed or unauthorised connection, a missing command or an MCP error is logged as
"MCP server yahoo failed: HTTP 401 Unauthorized from http://127.0.0.1:3000/mcp", without the SDK's nested
ExceptionGroup traceback (httpx's "For more information" line and a local server's stderr are folded into the same
line). Anything else still gets its traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
DARKMODE takes the dark hours: 18:00-6:00, 18-6 or 6pm-6am
91280f5 
Besides off, on and auto (9 pm to 7 am), DARKMODE can now be a range in TZ, in 24-hour or am/pm form, with or
without minutes; it may run past midnight and ends just before its last time. Settings.darkmode is now that range as
minutes of the day (None = off), and the chart checks the minute, not just the hour. A value that isn't a range
(18, 25-6, 6-6) stops startup with the accepted forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
DARKMODE: drop auto
e3a1ef5 
The dark hours are now a range you set (21-7 is what auto was); auto is refused at startup with the accepted forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
docs: shorten the DARKMODE row
81a6089 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Split replies by the length Telegram counts, not the raw HTML
6df9b8a 
Telegram's 4096-character message limit (and 1024 for captions) counts the
text after the HTML is parsed: a link counts only its label. split_html
counted the raw HTML, so every ticker and [n] citation URL used up room, and
a reply of ~2,500 visible characters with many citations was split in two.

- textfmt.visible_len: tags count nothing, an entity counts as its
  character, emoji as two UTF-16 units; split_html cuts by it.
- plain_text(text, limit): the plain-text fallback drops the URLs when
  "label (url)" would push a message or caption over Telegram's limit.
- filings: the "fits in one message" check uses visible_len too, so
  /events isn't sent to the model to shorten when it already fits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Merge the squash-merged history of #50 (identical tree to main)
7ee6a65 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
split_html: never cut inside a link; count undecoded entities in full
359668f 
Found by review of the previous commit:
- A cut that landed just after an <a href="...?a=1&b=2"> tag found the
  href's raw "&" and moved back inside the tag, leaving both messages with
  broken HTML. The cut now stops before tags that follow the last character
  that fits (they open the next message), an "&" moves the cut only when it
  starts a real entity that the cut would split, and the tag check runs last.
- Telegram decodes only &lt; &gt; &amp; &quot; and numeric entities; any
  other "&name;" is shown as typed, so visible_len counts it in full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
@claude
Tests: restore quote_links.US_EXCHANGES after each test
552b315 
The known-tickers refresh tests fill it and leave it set, so a later test
on the same xdist worker found NVDA placed on NASDAQ and the Google Finance
link test failed about one run in ten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014k4kywyN5dAo39izUPvxbw
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@robdevops @claude