BUILDING

*************************************************
*** COMPILING and INSTALLING QORE FROM SOURCE ***
*************************************************

see README-GIT to learn how to build qore from git sources
see README-WINDOWS for info specific to the windows builds

Source revision metadata
------------------------
CMake records the current commit when building a Git checkout, including linked
worktrees, and refreshes it on the next build after HEAD changes. Unchanged
metadata does not cause recompilation. Archives created with `git archive`
carry their original commit in .git-archive-revision, so their revision metadata
does not depend on Git being installed or on an enclosing checkout. For example:

    git archive --format=tar --prefix=qore-source/ HEAD > qore-source.tar

Keep .git-archive-revision when repacking an exported archive. An unversioned
source copy reports "unknown"; an invalid archive marker or broken Git checkout
fails instead of silently retaining a stale revision.

Build Requirements
------------------
*) a C++ compiler with C++17 support

*) flex 2.5.31 (or greater -- 2.5.4 or before will NOT work, sorry.  flex 2.5.37 or greater is recommended)
qore requires flex 2.5.31+ to build a reentrant parser; flex 2.5.37 or greater is recommended.
CMake will locate flex automatically; to use a non-default flex, set -DFLEX_EXECUTABLE=/path/to/flex when configuring.

*) bison 1.85 (or better, 2.* versions are fine)
qore requires bison 1.85+ to be able to interface properly with the scanner produced by flex 2.5.3*+

*) POSIX threads
OS-level POSIX thread support is required to build qore.

*) PCRE2 (libpcre2-8)
for Perl-Compatible Regular Expressions. Install libpcre2-dev (Debian/Ubuntu), pcre2-devel (RHEL/Fedora), or pcre2-dev (Alpine).
	https://github.com/PCRE2Project/pcre2

*) OpenSSL (different minimum versions depending on features):
   - Core/Qore: OpenSSL 0.9.7 or higher (3.0 or better recommended)
   - QUIC/HTTP/3 support: OpenSSL 3.5 or higher (required for QUIC TLS API)
	http://www.openssl.org
The configuration fails, naming both libraries, when a shared library linked into the Qore library, or a
library of a binary module that links OpenSSL itself, loads another OpenSSL than the one that Qore found:
two OpenSSL libraries in one process crash when an object of one is used by the other.  Set
OPENSSL_ROOT_DIR, or install a build of the library that uses the same OpenSSL.

*) bzlib 1.0.4 or higher

*) LibYAML
Required for the bundled yaml module. Install libyaml-dev (Debian/Ubuntu), libyaml-devel (RHEL/Fedora),
yaml-dev (Alpine), or libyaml (Homebrew).

*) zlib 1.1.3 or higher

*) mpfr 2.4.1 or higher - version 3.1+ recommended + note that the gmp library is also required to build mpfr
   	http://www.mpfr.org/
	http://gmplib.org/
This is for the arbitrary-precision numeric type introduced with Qore 0.8.6

*) brotli
	https://github.com/google/brotli
Required for Brotli compression support. Install libbrotli-dev (Debian/Ubuntu), brotli-devel (RHEL/Fedora), or brotli-dev (Alpine).

*) zstd (Zstandard) 1.4 or higher
	https://facebook.github.io/zstd/
Required for Zstandard compression support. Install libzstd-dev (Debian/Ubuntu), libzstd-devel (RHEL/Fedora), or zstd-dev (Alpine).

*) lz4
	https://lz4.github.io/lz4/
Required for LZ4 compression support. Install liblz4-dev (Debian/Ubuntu), lz4-devel (RHEL/Fedora), or lz4-dev (Alpine).

*) nghttp2 1.70.0 or higher
	https://nghttp2.org/
Required for HTTP/2 support. Install libnghttp2-dev (Debian/Ubuntu), nghttp2-devel (RHEL/Fedora), or nghttp2-dev (Alpine).
When no adequate system installation is found, the pinned release is downloaded and linked statically
(see "Downloaded Dependency Sources" below); this also applies to c-ares, ngtcp2 and nghttp3.  Installing
the system packages gives faster builds and distribution security updates.

*) c-ares 1.34.8 or higher
	https://c-ares.org/
Required for asynchronous DNS resolution, so that host name lookups never block the threads of the async
I/O controller.  1.34.8 is the minimum because of CVE-2026-33630, CVE-2026-69184 and CVE-2026-69186.
c-ares 1.34.7 and 1.34.8 can lose a query (c-ares issue 1280): the statically linked c-ares is patched,
and Qore starts a lookup again whose query a system c-ares lost.
Install libc-ares-dev (Debian/Ubuntu), c-ares-devel (RHEL/Fedora), or c-ares-dev (Alpine).

*) Kerberos 5 / GSSAPI
	https://web.mit.edu/kerberos/
Required for the builtin krb5 module and HTTP Negotiate/SPNEGO integration support.
Install libkrb5-dev (Debian/Ubuntu), krb5-devel (RHEL/Fedora), or krb5-dev (Alpine).
On macOS, install kerberos5 with MacPorts or krb5 with Homebrew when the system
Kerberos headers are not sufficient for building the GSSAPI extension support.

*) ngtcp2 1.25.0 or higher + ngtcp2_crypto_ossl + nghttp3 1.18.0 or higher
	https://github.com/ngtcp2/ngtcp2
	https://github.com/ngtcp2/nghttp3
Required for QUIC/HTTP/3 support. The ngtcp2_crypto_ossl backend requires OpenSSL 3.5+ (the QUIC TLS API).
Install libngtcp2-dev and libngtcp2-crypto-ossl-dev and libnghttp3-dev (Debian/Ubuntu),
ngtcp2-devel and ngtcp2-crypto-ossl-devel and nghttp3-devel (RHEL/Fedora),
or ngtcp2-dev and ngtcp2-crypto-ossl-dev and nghttp3-dev (Alpine).

Note: System packages on Ubuntu 24.04 (OpenSSL 3.0) and macOS Homebrew (OpenSSL 3.x) do not provide
the OpenSSL QUIC TLS backend. On these platforms, build ngtcp2 and nghttp3 from source with OpenSSL 3.5+:

    # First, build and install OpenSSL 3.5+ if not already available
    # Then build ngtcp2 with the OpenSSL backend:
    git clone https://github.com/ngtcp2/ngtcp2.git && cd ngtcp2
    git submodule update --init
    cmake -B build -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DENABLE_OPENSSL=ON -DENABLE_GNUTLS=OFF -DBUILD_TESTING=OFF
    cmake --build build && sudo cmake --install build

    git clone https://github.com/ngtcp2/nghttp3.git && cd nghttp3
    git submodule update --init
    cmake -B build -DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_TESTING=OFF
    cmake --build build && sudo cmake --install build

If OpenSSL 3.5+ is installed in a non-standard location (e.g. /opt/openssl-3.5), pass its
location to both the ngtcp2 and Qore builds:

    # ngtcp2 build with custom OpenSSL:
    cmake -B build -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DENABLE_OPENSSL=ON -DENABLE_GNUTLS=OFF -DBUILD_TESTING=OFF \
      -DOPENSSL_ROOT_DIR=/opt/openssl-3.5

    # Qore build with custom OpenSSL:
    cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DOPENSSL_ROOT_DIR=/opt/openssl-3.5

*) Eigen3 3.4 or higher (Eigen 5.x is also accepted)
	https://eigen.tuxfamily.org/
Required for the linear algebra of the builtin ml module.  Install libeigen3-dev (Debian/Ubuntu),
eigen3-devel (RHEL/Fedora), eigen-dev (Alpine), eigen (Homebrew), or eigen3 (MacPorts).

*) ICU
	https://icu.unicode.org/
Required for the Unicode CLDR locale data of the builtin i18n module.  Install libicu-dev (Debian/Ubuntu),
libicu-devel (RHEL/Fedora), icu-dev and icu-data-full (Alpine), icu4c (Homebrew), or icu (MacPorts).

*) libprotobuf
	https://protobuf.dev/
Required for the builtin protobuf module.  Install libprotobuf-dev (Debian/Ubuntu), protobuf-devel
(RHEL/Fedora), protobuf-dev (Alpine), protobuf (Homebrew), or protobuf3-cpp (MacPorts).

*) ONNX Runtime (optional)
	https://onnxruntime.ai/
Enables the OnnxModel class of the ml module, which runs pre-trained ONNX models; without it, OnnxModel
raises MISSING-FEATURE-ERROR and all native ML algorithms are still available.  Install onnxruntime-dev
(Alpine) or onnxruntime (Homebrew), or a release from https://github.com/microsoft/onnxruntime/releases.
-DQORE_WITH_ONNXRUNTIME=OFF disables it, and -DQORE_REQUIRE_ONNXRUNTIME=ON fails the configuration when
it cannot be enabled.

On macOS, CMake searches the MacPorts (/opt/local) and Homebrew prefixes for these dependencies.

*) Rust toolchain with cargo (optional)
	https://rustup.rs/
Optional; only needed for installing the tree-sitter CLI tool when developing the astparser grammar.
Install rust and cargo: rust cargo (RHEL/Fedora), cargo (Debian/Ubuntu), cargo rust (Alpine).
Then install the tree-sitter CLI: cargo install tree-sitter-cli@0.26.5

*) tree-sitter CLI (optional; for astparser grammar development)
	https://tree-sitter.github.io/tree-sitter/
Optional; only needed when regenerating the Qore grammar parser sources for the astparser module.
Pre-generated parser sources are included in the source tree and are used by the CMake build by default.
The tree-sitter C runtime library is downloaded by CMake at build time when it is not installed
(see "Downloaded Dependency Sources" below).
Install with: cargo install tree-sitter-cli@0.26.5

CMAKE Build
-----------

Example from the root source directory:

    mkdir build; cd build
    cmake .. -DCMAKE_INSTALL_PREFIX=/usr
    make && make docs && sudo make install

When reconfiguring an existing build, Qore checks CMake's temporary package redirects before
dependency discovery. These files describe targets from one configure run and must be recreated
on the next run. If CMake could not remove old redirects (for example because another user owns
the directory), Qore moves the directory to CMakeFiles/pkgRedirects.stale-<suffix>, prints that
path, and creates an empty redirect directory. This prevents false package matches such as
Boost being reported as found while Arrow cannot link Boost::headers. If the directory cannot be
moved, because its parent directory is not writable or, on systems that need write permission on
a directory to rename it, such as macOS 15, the directory itself is not writable, configuration
stops with a permissions diagnostic.


Downloaded Dependency Sources
-----------------------------

When a dependency is not installed (or is older than the required version), the build downloads
its source archive and builds it: c-ares, nghttp2, ngtcp2 and nghttp3, the tree-sitter runtime,
Apache Arrow for the dataframe module, and jemalloc on musl targets.  Every archive is listed
once, with its version, upstream URL and SHA-256, in cmake/third-party-sources.cmake; the
SHA-256 is verified whichever source is used.  That file also describes how to add an archive
or bump its version.  Use -DQORE_REQUIRE_SYSTEM_DEPENDENCIES=ON to fail instead of downloading.

-DQORE_THIRD_PARTY_MIRROR=<base URL> (or the QORE_THIRD_PARTY_MIRROR environment variable)
makes the build download each archive from <base URL>/<name>/<version>/<file> first, and from
upstream only when the mirror cannot provide it.  Qore's CI uses the package registry of its
GitLab project; there the job token is sent to the mirror, and QORE_THIRD_PARTY_MIRROR_TOKEN
can provide a GitLab access token elsewhere.  The token is sent only to the mirror.

cmake -P cmake/QoreCheckThirdPartySources.cmake checks that every download in the build takes
its source from cmake/third-party-sources.cmake; with -DQORE_THIRD_PARTY_MIRROR=<base URL>, it
also checks that the mirror has every archive with its pinned SHA-256.


Reproducible Builds
-------------------

The qpp and qcc tools take --file-prefix-map=<old>=<new> options, which map source paths in generated
metadata, debug information and exception locations.  The CMake build passes each -ffile-prefix-map option
of the C++ compiler flags (CMAKE_CXX_FLAGS and those of the build type) to both, for example:

    cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_CXX_FLAGS="-ffile-prefix-map=$PWD/..=/usr/src/qore"


Documentation Without Java Bindings
-----------------------------------

Module documentation generates Java bindings by default and requires the JNI
module for that output. To build only the HTML API references, configure with:

    cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release \
        -DQORE_GENERATE_JAVA_BINDINGS=OFF
    cmake --build build --target docs

This option applies to both bundled and external module documentation targets.


AOT Consumer Link Configuration
-------------------------------

CMake generates aot-link.conf for qcc and installs it under libdir/qore.
It resolves imported CMake and pkg-config library targets, including transitive
dependencies, into linker arguments. Package-build flags in
CMAKE_SHARED_LINKER_FLAGS are not copied into this installed configuration:
compiler wrappers and RPM package notes may require an environment that users
do not have when compiling their own programs.

Use QORE_AOT_LINK_FLAGS for additional flags that should apply to both dynamic
and static AOT consumers, for example:

    cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release \
        -DQORE_AOT_LINK_FLAGS="-Wl,-z,relro -Wl,-z,now"

These flags must be usable on the target system without the package builder's
environment. Sanitizer flags and the macOS SDK sysroot are handled separately.


Two-Pass Build (for environments installing external binary modules between passes)
-----------------------------------------------------------------------------------

The qore qlib contains user modules that use `%try-module` / `%ifdef
No<Name>` to branch on the availability of external binary modules
(json, yaml, xml, uuid, ...).  Those branch outcomes are baked into
the AOT-compiled `.qmod` files at qcc compile time.  If qore is built
before the external modules are installed, qlib `.qmod`s end up with
no-<module> stubs that throw at runtime even after the module is
installed.

For CI and packaging flows that build qore alongside external binary
modules, use the two-pass build:

    # Pass 1: build and install libqore, headers, and tools only.
    # AOT of qlib qmods is skipped, so external module presence does
    # not affect qlib output.
    mkdir build; cd build
    cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DQORE_BUILD_AOT_MODULES=OFF
    make && sudo make install

    # Pass 2: build and install all external binary modules
    # (module-xml, module-uuid, etc.; json and yaml are part of Qore)
    # against the qore installed above.

    # Pass 3: re-cmake qore with AOT re-enabled; the cmake-time probe
    # records which external modules are now loadable and the qmod
    # rules depend on its result, so all qlib qmods are AOT-compiled
    # with the correct %ifdef branches.
    cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DQORE_BUILD_AOT_MODULES=ON
    make && sudo make install

Single-source-tree re-builds: the AOT rules track which external
binary modules were loadable at cmake configure time (via the file
`qore-aot-probe.stamp` in the build directory).  Re-running cmake
after installing or removing an external binary module forces a
re-AOT of the affected qlib qmods on the next make invocation; no
clean rebuild needed.


OS-Specific Issues
------------------
*) Linux:
there are no particular issues on Linux, this is one of the main development platforms.
Various distributions have been tested: FC3-8, Gentoo, Ubuntu, ARCH, etc

*) FreeBSD:
building from source - for FreeBSD up to 9.1: flex has to be installed from ports
as long as all the prerequisite libraries are installed, cmake should set up Qore for building and installing on FreeBSD without the need for any particular options or environment variables
note that there is a freebsd port of Qore available (http://www.freebsd.org/cgi/cvsweb.cgi/ports/lang/qore)

*) Darwin - OS/X
One of the main development platforms for Qore.  No particular issues with recent versions of OS/X.  Note that there is a port available for Qore in macports (https://trac.macports.org/browser/trunk/dports/lang/qore/Portfile); there are also ports for the development version and for many binary modules

*) Solaris:
Supported with g++ with c++17 support, Oracle Studio CC not tested recently

*) HP-UX
Qore supported HP-UX in the past, but it has not been tested for a few years.
Itanium should be supported better than PA-RISC.

*) Windows
A fully functional version of Qore can now be build for Windows; see README-WINDOWS for more information

CPU Support
-----------
*) gcc with i386, x86_64, ppc, sparc32, itanium, aCC with itanium, CC (SunPro or Sun Studio CC) with i386, x86_64, sparc32: fast inline assembly atomic operations are supported for reference counting, as well as a SMP cache invalidation optimization for temporary objects (temporary objects do not require a cache invalidation);
*) CPU stack guard is working on all above combinations and with g++ and aCC with PA-RISC as well
*) all others: a pthread mutex is used to ensure atomicity for reference counting.
The cache invalidation optimization is not safe on platforms without an atomic reference counting implementation, therefore is not implemented for these platforms

Modules
-------
Note that binary modules are delivered separately from the qore library, see the file README-MODULES for more information.

Test timeout diagnostics
------------------------

The test runner aborts timed-out tests with GNU timeout and uses gdb, when
available, to report thread stacks. It recognizes absolute Linux core-file
patterns with fixed directory paths, including OBS's /.build/cores/%p. Only
core files newer than the current test are considered. Piped core handlers
remain the responsibility of the host crash collector.

Run the isolated runner regression tests with:

    python3 -B -W error examples/test/runner/test_timeout_diagnostics.py -v