Repository navigation
Conversation
The drivers may launch own background threads, which will not be torn down cleanly before dlclose(), as the libraries were not designed with that usercase in mind. We therefore ensure that their symbols stick around, by opening with RTLD_NODELETE. This prevents crashes when e.g. reloading a board.
This is a workaround until open-ephys/liboni#65 has been merged or another solution has been found. Ideally, we would also stop mixing libusb versions, which isn't even hard to achieve, but is a change for later (and would require upstream changes).
|
Hello @ximion Regarding this patch, would it be possible to check if building We are hesitant to implement this fix since similar situation with loaded symbols have caused trouble in the past while updating the acquisition board plugin through the GUI. If dynamic linking solved the issue, we might just make it the default linking and make a custom flag when building it for our GUI. |
That wouldn't fix this, because we would still have a thread lifetime issue... What would solve this completely though is if we would build everything from sources properly. The FT600 sources are available, and libusb is a very stable library. So we could make the libraries build against the system libusb (optionally link it statically). That way, the entire process can use the same libusb library, and if we dynamically link that library (and don't dlopen() a second copy) its symbols will stay around and no thread should run into trouble upon unload. I could make that change as a PR, but it would be quite substantial. The upside is that you could use CI easier and build for any OS and architecture, because there would be no precompiled binary blobs anymore (at least not for the FT600 path). You probably would want the buildsystem to produce a static version / dependency-minimized of this library still for the Open Ephys GUI. Those are a bit easier to deploy on Linux if you don't know the target system. As for this PR: I don't foresee it causing issues on its own, but using it does mean that a bunch of stuff stays in memory even if it isn't technically needed any longer - that includes some code that listens to hardware events. I don't know if that causes issues for Open Ephys GUI (for Syntalos, I am a bit surprised that it can host libusb twice without any issues, but that may be due to its plugin system). |
|
Hi @ximion , thanks for tracking this down and for the detailed follow-up. My hesitation with the PR as written is that it changes loader behavior for every driver, while the thread leak comes only from the FT600 driver and its embedded That sets Building from sourceOn building from source: I wasn't aware the D3XX sources were public. As far as I know, FTDI only ships the prebuilt libftd3xx binaries. Can you point me to them? On the root problemThe root cause is the D3XX library itself (libftd3xx). It starts libusb's hotplug thread with
A small standalone program that reproduces it would probably help them most: load the driver, open and close a device, unload it, then show the thread still running in FWIW @aacuevas has put his time in with FTDI for similar issues. It appears to be a couple guys, a coffee machine, and 2000s dell pc, and some very bright ideas |
Hi!
I was investigating a nasty crash in Syntalos and traced it back to an issue in liboni: When the driver library is unloaded, a bunch of threads from the FT600 library, notably libusb threads, stick around and don't quit. This is expected behavior - and annoyingly, that static library embeds libusb (all something to be solved on another day).
Then, if any USB event comes in and wakes up the background thread from the unloaded library, it tries to execute code that is gone missing and jumps into uninitialized memory (or to a random spot).
So, unloading this is inherently unsafe. Therefore, I changed the
dlopen()call to useRTLD_NODELETEwhich will prevent the symbols from going missing. Repeated calls todlopen()will just recycle the already loaded symbols, so this is safe, and when the program terminates, everything should be unloaded cleanly.The only alternative to this that I see would be making absolutely sure that all background threads of the driver have joined, but due to how libusb works (and who knows what else a driver may run), we can't do that. It would also be kinda brittle to rely on this.
Let me know if this solution makes sense to you!
Best,
Matthias