Skip to content

Python support: transpiled Python on every target, Python servers, Pillow and font-from-data - #6128

Merged
shai-almog merged 146 commits into
masterfrom
python-support
Nov 5, 2026
Merged

shai-almog merged 146 commits into
masterfrom
python-support

Conversation

@shai-almog

Copy link
Copy Markdown
Collaborator

Python support for Codename One: Python sources are transpiled to Java and run on every target, client and server.

What is in it

  • maven/python-transpiler and maven/python-runtime: the transpiler, the runtime and the bundled stdlib and site packages, with ~125 conformance programs whose expected output is real CPython output.
  • Build integration in build-engine (Maven goal, Gradle task, project importer, PyPI resolution, C-extension stubs) and the python.virtualThreadStack build hint / @VirtualThreadStack.
  • Python servers on the backend: FastAPI/Starlette/Flask, WebSockets and streaming responses, BackgroundTasks after the response, SQLAlchemy on SQLite, PostgreSQL and MySQL (CI runs it against both), a GIL that cooperates with virtual threads.
  • Pillow: on servers over a new backend image API (com.codename1.backend.image: raster, JPEG/PNG, TrueType text); on clients over Codename One's Image/ImageIO.
  • Core: Font.createTrueTypeFontFromData / isTrueTypeStreamSupported on JavaSE, Android, iOS/Mac, JavaScript, Windows and Linux; Database column types.
  • Backend: MySQL/PostgreSQL sslmode=verify-ca with a CA file (plus the MySQL spellings).
  • Fix: the core inflater (io/gzip/InfCodes) rejected valid deflate streams whose last code ended a few bits into the final byte (about half of all gzip files, depending on the header mtime). Fuzzed against java.util.zip with 160k streams fed in chunks.
  • Developer guide chapters for all of the above.

Verification

  • python-transpiler/python-runtime tests (145), core-unittests, backend tests; SpotBugs/PMD/Checkstyle clean on the touched modules; cast-semantics, native-signature, C++ linkage, control-character and build-hint gates.
  • End-to-end server runs on the JVM and native, and SQLAlchemy against PostgreSQL 16 and MySQL 8.4.
  • Not verified locally: the JavaScript font path in a browser (TrueTypeFontFromDataTest in the JS suite covers it here), the Windows/Linux C font changes (no toolchain locally) and the iOS private-API scan.

🤖 Generated with Claude Code

shai-almog and others added 30 commits October 27, 2026 17:01
Python 3 sources transpile to Java 17 (maven/python-transpiler) and run on
maven/python-runtime on every Codename One target. Generators, async,
match, dataclasses, the builtins and a stdlib wave are covered by a
conformance suite diffed against CPython on the JVM and natively through
ParparVM. Constructs a translated program cannot run (eval, exec,
compile, __import__, frame hooks) are build errors naming file and line.

ParparVM: a frameless method returning a value after several try blocks
that end on one label ran a single END_TRY and left the other blocks on
the thread's try stack, so a later throw longjmp'd into a returned frame.
The folded return paths now unwind the stack (FramelessTryReturnIntegrationTest).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
from java.<package> import <Class> resolves at build time against the
compile class path; a generated JavaInterop bridge dispatches with
instanceof and direct calls (no reflection), converts values both ways,
adapts Python callables to functional interfaces and exposes Java
collections to len/in/iteration/indexing.

cn1:import-python-project and cn1:transpile-python (Maven), the
transpilePython task (Gradle), and the archetype's python profile build
src/main/python. runMain returns the exit status, honouring SystemExit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vice library

from java.java.* resolves against the class path's own class library
(java-runtime, JavaAPI) when it carries one, so an API the device lacks is
a build error rather than a bridge that fails to link.

python-project-test.sh imports scripts/python-samples/hello-ui (a
Codename One UI written in Python), builds it with Maven and Gradle and
stages every target; python-project.yml runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyHttp is the client the packages are written over: ConnectionRequest in
an application, installable like PyFileSystem. requests (verbs, Session,
auth, params/json/form bodies, cookies, exceptions) and httpx (verbs,
Client, AsyncClient, URL, Headers, HTTPStatusError) are our own Python
over it; http_requests and http_httpx match the real packages' output
against the same test server.

BaseException.__new__ ignores keyword arguments, as CPython's does, so an
exception subclass whose __init__ takes keywords can be raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_type_hints

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lice.indices

The HTTP client is installed by the generated registry only when the
program bundles requests or httpx, so other programs do not link the
network stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ule __dict__

Fixes found while porting attrs and pydantic: an annotation built by a call
(Annotated[int, Field(gt=0)], conint(ge=2)) was stored as None; a class
__delattr__ was bypassed by del; classes had no __hash__ attribute; and
type(name, bases, ns) gave __module__ 'builtins'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_cn1server.serve(dispatch, host, port) bridges a Python dispatch callable
to the backend HttpServer when the backend is on the transpile class path;
scripts/python-backend/build.sh builds such an app as a native binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clean-room site packages flask, werkzeug.exceptions, fastapi, starlette
(responses, requests, testclient, status) and uvicorn. Every app exposes
dispatch(method, path, query, headers, body) -> (status, headers, body);
app.run / uvicorn.run hand it to _cn1server.serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions

inspect.isgeneratorfunction/iscoroutinefunction/isasyncgenfunction now
answer from __code__ as CPython's do; FastAPI's yield dependencies use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper returns

CommonWorkloads ported to Python (scripts/python-bench/common) with a
Java hot-spot variant, and a comparison across CPython, PyPy, GraalPy and
Codename One with peak RSS per workload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mance chapter

The developer guide's Python chapter compares CPython, PyPy, GraalPy and
Codename One on startup, kernels, Flask and the CommonWorkloads suite, with
peak memory in every table, two charts, and a section on moving a measured
hot spot into Java.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IntRanges runs interval analysis over each typed function: loop counters
are bounded by their range, accumulators by trip count times their
per-pass growth, and masked values by the mask. A local whose every
value fits in 64 bits becomes a Java long, and an int expression whose
result fits is computed with wrapping long arithmetic, which equals the
exact result however large the intermediates are. Expression helpers
such as i32/i64 are inlined, module int constants are folded, and >> of
a value masked into [0, 2^64) becomes >>>.

Builtins without a typed lowering (print, sorted, ...) are now called
generically from typed code instead of leaving the whole function
untyped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd //, lowered loop tests

Locals that only ever hold a freshly constructed object whose __init__
copies its arguments into fields, and are only read through fields or
getters, become one local per field. while tests lower their arithmetic
and compares like any statement. for c in str walks code points, with
ord(c) the code point itself. l.extend([v] * n) fills in place. Lone
surrogates index and repr as CPython does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped builtins over BOT, dict clear/len intrinsics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… int results held bare, taken in the slow path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t stores per instance, try-free dict key hashing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-top arithmetic over impure calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct clear; direct dict len

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… box the list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shai-almog and others added 12 commits October 31, 2026 08:48
…ll dicts scan

A virtual thread's call-stack, try-block and pending-allocation buffers now come
from one packed mmap that is unmapped when the retired state is freed, and each
retirement charges the BiBOP trigger so thousands of dead states cannot pile up
between cycles (Flask under load held ~4,000 of them, ~300 MB resident).

PyDict allocates nothing until first insert and keeps no index table at or below
eight entries, scanning keys linearly instead. Words.next keeps its loop
conditions free of stack juggling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The semantic analyzer's merge compared values with SimdAllocaValue.equals,
which refuses ASM's own BasicValue constants, so a long pushed by LCONST_1
meeting the checker's long at a join fell to BasicInterpreter.merge and became
a one-slot UNINITIALIZED_VALUE; the next DUP2 failed "Illegal use of DUP2" on
generated Python that ASM's plain analyzer accepts. Merge now compares types.

The typed float path for ** emitted Math.pow, which CLDC lacks; more stdlib
functions are typed now, and the demos' compliance check caught 30 such calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pinning the GC

JFR on the JVM build of the Flask sample (escape analysis off) put the
request path at 17.7 KB allocated per request; it is now 12.7 KB on the
JVM, and the native binary's peak RSS under ab -c 8 fell from 765-816 MB
to 219-220 MB with no GC tuning.

- Flask shim: Headers keeps folded names beside the pairs; routing skips a
  rule whose literal prefix does not match before running its regex, and
  makes the 405 set only when needed; prebuilt JSON encoders; lazy flask.g;
  no copies of the header list or the blueprint chain per request.
- json: unescaped strings quote without a char list; sort_keys sorts pairs
  without a lambda; the circular check is an identity stack, not an id()
  dict.
- runtime: regex scratch (text, registers, stacks) reused per program via
  an atomic hand-off; str.startswith/endswith intrinsics and in-place
  affix; str < str compares without wrapping; str iteration and indexing
  share Latin-1 one-char strings; dict literals presized, first growth 4.
- backend: a connection's deferred-JSON sink and target memo table are
  made on first need, so a one-request connection allocates neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…66% survival

The collector sat at its peak footprint for 30 seconds after load stopped, because
only allocation or the long idle starts a cycle. The first quiet pass after a busy
one now idles 1s, so a quiet cycle runs the major sweep, page trim and malloc relief.

The backend build grows the trigger only at 66% survival instead of 25%, so the heap
settles near 1.5x the live set rather than 4x. Flask sample, 3 interleaved runs:
footprint after load 137-191MB -> 75-98MB, 5s later 101-139MB -> 68-70MB, at
27.7-29.2k -> 25.7-25.9k req/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…duty cycle

A fixed 66% cost the Flask server 9% of its throughput. The threshold now moves
from CN1_BIBOP_TIGHT_SURVIVAL_PERCENT (backend: 66) when the collector is idle for
90% of wall time or more, to the VM's 25% when it is busy 40% or more. Default
builds set both equal and are unchanged. Full speed: 29.1-29.7k req/s vs
29.6-30.1k default and 26.0-26.9k fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cales with a small budget

Under a per-process budget the survival threshold is the tight one regardless of
the collector's duty cycle. The 64MB admission margin becomes a quarter of the
budget below 256MB: under 128MB the Flask server went from 478 to 16.3k req/s at
a 75MB footprint. Budgets of 256MB or more are admitted as before.

compare.py runs every server for the same 30s, reports physical footprint after
load, 5s later and at peak, and runs Codename One both unconstrained and under a
--budget-mb budget (default 128). render_charts.py adds the throughput-against-
memory chart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd under a budget

The comparison now runs every Codename One binary twice: as shipped, where the
adaptive collector spends spare memory for throughput, and under a 128 MB
per-process budget, where it keeps the heap tight. Memory is physical footprint
for every runtime; Flask gets 30 s of load and a settle read. Adds the
throughput-vs-footprint chart.

Also routes the transpiler's float pow through Py.powd (Math.pow): MathUtil.pow
had made nbody 4x slower.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on Windows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-thread pacing claims lived in __thread storage, and a virtual thread
waiting for budget can resume on another host, so claims leaked until no
allocation was admitted and a budgeted server crawled. The claim is now one
global value stamped with the GC epoch, expiring after two cycles. Retired
virtual-thread stacks are released under a budget.

The comparison runs the 128 MB budget only: Flask under 64 MB cannot hold
the cap, since its unpaced virtual-thread stacks and malloc swing past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/developer-guide/languagetool-accept.txt
@shai-almog

shai-almog commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Compared 193 screenshots: 193 matched.
✅ Native iOS Metal screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 2400 seconds

Build and Run Timing

Metric Duration
Simulator Boot 55000 ms
Simulator Boot (Run) 1000 ms
App Install 17000 ms
App Launch 2000 ms
Test Execution 463000 ms

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 337ms / native 4ms = 84.2x speedup
SIMD float-mul (64K x300) java 300ms / native 3ms = 100.0x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 97.000 ms
Base64 CN1 decode 68.000 ms
Base64 native encode 1321.000 ms
Base64 encode ratio (CN1/native) 0.073x (92.7% faster)
Base64 native decode 1141.000 ms
Base64 decode ratio (CN1/native) 0.060x (94.0% faster)
Base64 SIMD encode 61.000 ms
Base64 encode ratio (SIMD/CN1) 0.629x (37.1% faster)
Base64 SIMD decode 60.000 ms
Base64 decode ratio (SIMD/CN1) 0.882x (11.8% faster)
Base64 encode ratio (SIMD/native) 0.046x (95.4% faster)
Base64 decode ratio (SIMD/native) 0.053x (94.7% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 3.000 ms
Image createMask ratio (SIMD on/off) 0.429x (57.1% faster)
Image applyMask (SIMD off) 367.000 ms
Image applyMask (SIMD on) 38.000 ms
Image applyMask ratio (SIMD on/off) 0.104x (89.6% faster)
Image modifyAlpha (SIMD off) 312.000 ms
Image modifyAlpha (SIMD on) 57.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.183x (81.7% faster)
Image modifyAlpha removeColor (SIMD off) 55.000 ms
Image modifyAlpha removeColor (SIMD on) 46.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.836x (16.4% faster)

shai-almog and others added 12 commits November 2, 2026 10:06
…docs: Vale

test_virtual_thread.c links cn1_virtual_thread.c without cn1_globals.m, so
the new cn1ProcessHasMemoryBudget() reference failed the link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
viewSafeAreaInsetsDidChange calls it on Mac Catalyst and tvOS too, but it was
defined only in the iPhone layout branch, so the Metal Catalyst build failed
with -Wobjc-method-access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	Ports/iOSPort/nativeSources/CodenameOne_GLViewController.m
#	scripts/developer-guide/check-guide-api-names.py
…e fairly

CPython and PyPy served the same ~690 requests a second under app.run()
although PyPy runs Flask twice as fast: Werkzeug's development server drains
the socket for 10 ms after every response, and ab counts a request done when
the socket closes. The guide keeps that chart, marks it as suspicious, tells
how the cause was found, and adds the fair comparison: compare.py now also
serves each interpreter behind gunicorn with 8 workers, footprint summed over
every process. flask_cost.py times Flask alone, without a server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ide's accept list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base row is a single run; the branch's VM runtime measures identical to
master's on the self-hosted translator (interleaved A/B, reason in the file).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture

The multi-window default scenario was captured before its first paint (skin
and form solid black) because the menu bar alone passes the flat-image check;
single-window mode already waits that out, multi mode had no equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The iOS accessibility thread contract (on IOSNative.m changes) and the
Xcode leg of the Gradle/Maven parity check queued on hosted macos-15 for
every matching pull request, and were dropped unrun when that pool was
short. Both now run nightly, plus on a pull request touching their own tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shai-almog

shai-almog commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Compared 193 screenshots: 193 matched.
✅ Native iOS Metal screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 2675 seconds

Build and Run Timing

Metric Duration
Simulator Boot 97000 ms
Simulator Boot (Run) 1000 ms
App Install 14000 ms
App Launch 6000 ms
Test Execution 449000 ms

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 66ms / native 3ms = 22.0x speedup
SIMD float-mul (64K x300) java 65ms / native 4ms = 16.2x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 521.000 ms
Base64 CN1 decode 158.000 ms
Base64 native encode 859.000 ms
Base64 encode ratio (CN1/native) 0.607x (39.3% faster)
Base64 native decode 941.000 ms
Base64 decode ratio (CN1/native) 0.168x (83.2% faster)
Base64 SIMD encode 244.000 ms
Base64 encode ratio (SIMD/CN1) 0.468x (53.2% faster)
Base64 SIMD decode 111.000 ms
Base64 decode ratio (SIMD/CN1) 0.703x (29.7% faster)
Base64 encode ratio (SIMD/native) 0.284x (71.6% faster)
Base64 decode ratio (SIMD/native) 0.118x (88.2% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 2.000 ms
Image createMask ratio (SIMD on/off) 0.286x (71.4% faster)
Image applyMask (SIMD off) 237.000 ms
Image applyMask (SIMD on) 57.000 ms
Image applyMask ratio (SIMD on/off) 0.241x (75.9% faster)
Image modifyAlpha (SIMD off) 50.000 ms
Image modifyAlpha (SIMD on) 54.000 ms
Image modifyAlpha ratio (SIMD on/off) 1.080x (8.0% slower)
Image modifyAlpha removeColor (SIMD off) 138.000 ms
Image modifyAlpha removeColor (SIMD on) 254.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 1.841x (84.1% slower)

…nutes

The x64 suite allocated a fixed 12G swap file and died 'No space left on
device' two minutes into the C build on a runner with little free disk; the
swap now leaves 40G for the build. build-ios-tv measured 15-22 min of build
plus 28-33 min of tests and was cancelled at its 60 minute limit mid-run on
this and other branches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shai-almog
shai-almog merged commit 5f96a6b into master Nov 5, 2026
100 of 107 checks passed
@shai-almog
shai-almog deleted the python-support branch November 5, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants