Skip to content

Agents: screenshot of any tab, typing into embedded PuTTY windows - #7

Merged
Modding merged 1 commit into
mainfrom
agent-screenshot-and-putty-typing
Nov 14, 2026
Merged

Modding merged 1 commit into
mainfrom
agent-screenshot-and-putty-typing

Conversation

@Modding

@Modding Modding commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Agents could open an embedded PuTTY, Remote Desktop or VNC tab but neither see nor use it: read only returned the tab's label and typing was refused.

Changes

  • screenshot (CLI window screenshot [index] [file.png], MCP window_screenshot, which answers with an image): a PNG of what a tab shows, embedded windows included. Taken from the window itself, so a window lying on top is neither in the way nor in the picture. The file name must end in .png.
  • Typing into embedded PuTTY windows: characters are posted to the window, the keyboard focus stays where it is. Enter goes as a key press (PuTTY's own line editing shows a posted carriage return as ^M). Same switch as typing into terminals. Remote Desktop and VNC windows answer with an error that points to the screenshot.
  • The setting's label, the user guides (five languages) and the README say so.
  • --open embedtest:… takes a program with arguments.

Tested

  • Windows: fmt, clippy -D warnings, 39 tests.
  • A window with a fresh data folder and a real PuTTY (-raw to a local test server) embedded in a tab, driven through the control channel: the screenshot shows the PuTTY window with the server's greeting; three typed lines, one with umlauts, arrived at the server byte for byte with their line ends; typing is refused while the switch is off; a file name not ending in .png is refused; the home tab gives a picture too.

Not tested

  • Typing into a PuTTY SSH session on a real server (the test used a raw connection).
  • The screenshot of a Remote Desktop or VNC window, and with the window minimised.
  • Linux and macOS (own drawing only, no embedding there): left to CI for the build.

- New control command "screenshot" (CLI: window screenshot, MCP: window_screenshot, answered
  as an image): a PNG of what a tab shows, including a PuTTY, Remote Desktop or VNC window
  embedded in it, where reading the tab has no text. The picture is taken from the window
  itself (PrintWindow), so a window lying on top is neither in the way nor in the picture.
  The file name must end in .png.
- Typing reaches embedded PuTTY windows too: characters are posted to the window, the
  keyboard focus stays where it is. Enter goes as a key press, because PuTTY's own line
  editing shows a posted carriage return as ^M. Behind the same switch as typing into
  terminals; other embedded windows answer with an error that points to the screenshot.
- The embedding check (--open embedtest:…) takes a program with arguments.
- Roadmap: what Remote Desktop and VNC would need.
@Modding
Modding merged commit 33d5781 into main Nov 14, 2026
4 checks passed
@Modding
Modding deleted the agent-screenshot-and-putty-typing branch November 14, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant