Skip to content

Fix crash by opening driver libraries with RTLD_NODELETE - #65

Open
ximion wants to merge 1 commit into
open-ephys:mainfrom
ximion:wip/lusb-thread-crash-fix
Open

ximion wants to merge 1 commit into
open-ephys:mainfrom
ximion:wip/lusb-thread-crash-fix

Conversation

@ximion

@ximion ximion commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Hi!

I was investigating a nasty crash in Syntalos and traced it back to an issue in liboni: When the driver library is unloaded, a bunch of threads from the FT600 library, notably libusb threads, stick around and don't quit. This is expected behavior - and annoyingly, that static library embeds libusb (all something to be solved on another day).

Then, if any USB event comes in and wakes up the background thread from the unloaded library, it tries to execute code that is gone missing and jumps into uninitialized memory (or to a random spot).

So, unloading this is inherently unsafe. Therefore, I changed the dlopen() call to use RTLD_NODELETE which will prevent the symbols from going missing. Repeated calls to dlopen() will just recycle the already loaded symbols, so this is safe, and when the program terminates, everything should be unloaded cleanly.

The only alternative to this that I see would be making absolutely sure that all background threads of the driver have joined, but due to how libusb works (and who knows what else a driver may run), we can't do that. It would also be kinda brittle to rely on this.

Let me know if this solution makes sense to you!
Best,
Matthias

The drivers may launch own background threads, which will not be torn
down cleanly before dlclose(), as the libraries were not designed with
that usercase in mind.
We therefore ensure that their symbols stick around, by opening with
RTLD_NODELETE. This prevents crashes when e.g. reloading a board.
ximion added a commit to syntalos/syntalos that referenced this pull request Aug 27, 2026
This is a workaround until open-ephys/liboni#65
has been merged or another solution has been found.
Ideally, we would also stop mixing libusb versions, which isn't even
hard to achieve, but is a change for later (and would require upstream
changes).
@aacuevas

Copy link
Copy Markdown
Collaborator

Hello @ximion
Thanks for the report!
Sorry we took so long to reply to this one, it flew under our radar.

Regarding this patch, would it be possible to check if building libonidriver_ft600.so using dynamic linking to libftd3xx.so instead of static would solve this issue?
There is nothing preventing a dynamic link. We chose to use static linking so it was easier to distribute the files through the Open Ephys GUI integrated plugin installer architecture, but dynamic linking should work as well, and that might solve this issue.

We are hesitant to implement this fix since similar situation with loaded symbols have caused trouble in the past while updating the acquisition board plugin through the GUI.

If dynamic linking solved the issue, we might just make it the default linking and make a custom flag when building it for our GUI.

@ximion

ximion commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Regarding this patch, would it be possible to check if building libonidriver_ft600.so using dynamic linking to libftd3xx.so instead of static would solve this issue?

That wouldn't fix this, because we would still have a thread lifetime issue... What would solve this completely though is if we would build everything from sources properly. The FT600 sources are available, and libusb is a very stable library. So we could make the libraries build against the system libusb (optionally link it statically). That way, the entire process can use the same libusb library, and if we dynamically link that library (and don't dlopen() a second copy) its symbols will stay around and no thread should run into trouble upon unload.

I could make that change as a PR, but it would be quite substantial. The upside is that you could use CI easier and build for any OS and architecture, because there would be no precompiled binary blobs anymore (at least not for the FT600 path).

You probably would want the buildsystem to produce a static version / dependency-minimized of this library still for the Open Ephys GUI. Those are a bit easier to deploy on Linux if you don't know the target system.

As for this PR: I don't foresee it causing issues on its own, but using it does mean that a bunch of stuff stays in memory even if it isn't technically needed any longer - that includes some code that listens to hardware events. I don't know if that causes issues for Open Ephys GUI (for Syntalos, I am a bit surprised that it can host libusb twice without any issues, but that may be due to its plugin system).

@jonnew

jonnew commented Oct 13, 2026

Copy link
Copy Markdown
Member

Hi @ximion , thanks for tracking this down and for the detailed follow-up.

My hesitation with the PR as written is that it changes loader behavior for every driver, while the thread leak comes only from the FT600 driver and its embedded libusb. I'd rather limit the fix to that driver. Could you try marking the driver itself as non-unloadable at link time instead? On Linux that's one line in drivers/ft600/Makefile:

LDFLAGS += -Wl,-z,nodelete

That sets DF_1_NODELETE on libonidriver_ft600.so, so dlclose() never unmaps it no matter how it was opened, and onidriverloader.c can stay as it is. It also protects any host that loads the driver some other way. If you can confirm it fixes the Syntalos crash, I'm happy to merge that version. Have you seen the same crash on macOS? ld64 has no equivalent flag, so we'd need a different approach there.

Building from source

On building from source: I wasn't aware the D3XX sources were public. As far as I know, FTDI only ships the prebuilt libftd3xx binaries. Can you point me to them?

On the root problem

The root cause is the D3XX library itself (libftd3xx). It starts libusb's hotplug thread with libusb_init, provides no call that shuts it down, and apparently never calls libusb_exit on its own. As a result, nothing that loads it can safely unload it. Every application that loads D3XX as a plugin is exposed to this, not just liboni, so the proper fix belongs with FTDI. Since you've done the debugging and reproduced it with Syntalos, would you be willing to report it to FTDI support? Roughly, in order of preference, I'd ask them for:

  • A library-level shutdown function (e.g. FT_Cleanup()) that calls libusb_exit and joins its threads. Even better, the library could do this itself when the last device handle is closed.
  • Failing that, a shared libftd3xx.so linked with -z nodelete, so the library can't be unloaded while its threads are running.
  • Whether they'd publish the D3XX sources, or a build that links against the system libusb. That would also settle the build-from-source question.

A small standalone program that reproduces it would probably help them most: load the driver, open and close a device, unload it, then show the thread still running in /proc/self/task and the crash after a USB event. Please link the FTDI ticket here if you open one. If they ship a fix, we'll switch to it and drop the workaround.

FWIW @aacuevas has put his time in with FTDI for similar issues. It appears to be a couple guys, a coffee machine, and 2000s dell pc, and some very bright ideas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants