Repository navigation
Closed
Copy link
Description
Describe the bug
Jamulus 4.0.0 beta1 seems to hang on macOS while fetching the server list. This might be a deadlock
To Reproduce
- Connect to a server
- Open connect dialog via File->connect
- Sometimes observe that Jamulus hangs. This might occur if one switches directories often in the UI
Expected behavior
No hang
Trace
It seems as if the hang might occur in CLMessReadyForSending:
12 QCoreApplication::sendEvent(QObject*, QEvent*) + 172 (QtCore + 625164) [0x107004a0c] 1-12
12 QApplication::notify(QObject*, QEvent*) + 3084 (QtWidgets + 70888) [0x105c494e8] 1-12
12 QApplicationPrivate::notify_helper(QObject*, QEvent*) + 332 (QtWidgets + 64136) [0x105c47a88] 1-12
12 QWidget::event(QEvent*) + 3200 (QtWidgets + 390252) [0x105c9746c] 1-12
12 QObject::event(QEvent*) + 672 (QtCore + 929364) [0x10704ee54] 1-12
12 ??? (Jamulus + 997264) [0x104beb790] 1-12
12 CProtocol::CreateCLPingWithNumClientsMes(CHostAddress const&, int, int) + 116 (Jamulus + 155652) [0x104b1e004] 1-12
12 CProtocol::CreateAndImmSendConLessMessage(int, CVector<unsigned char> const&, CHostAddress const&) + 152 (Jamulus + 134384) [0x104b18cf0] 1-12
12 CProtocol::CLMessReadyForSending(CHostAddress, CVector<unsigned char>) + 56 (Jamulus + 1406932) [0x104c4f7d4] 1-12
12 ??? (QtCore + 963772) [0x1070574bc] 1-12
12 ??? (Jamulus + 623968) [0x104b90560] 1-12
12 ??? (Jamulus + 624200) [0x104b90648] 1-12
12 __sendmsg + 8 (libsystem_kernel.dylib + 27008) [0x18228b980] 1-12
*12 ??? (kernel.release.t6041 + 24220) [0xfffffe000b739e9c] 1-12
*12 ??? (kernel.release.t6041 + 1923588) [0xfffffe000b909a04] 1-12
*12 ??? (kernel.release.t6041 + 7742628) [0xfffffe000be964a4] 1-12
*12 ??? (kernel.release.t6041 + 6768432) [0xfffffe000bda8730] 1-12
*12 ??? (kernel.release.t6041 + 6767100) [0xfffffe000bda81fc] 1-12
*12 sosend + 1044 (kernel.release.t6041 + 6696948) [0xfffffe000bd96ff4] 1-12
*12 ??? (kernel.release.t6041 + 6695552) [0xfffffe000bd96a80] 1-12
*12 ??? (kernel.release.t6041 + 6744736) [0xfffffe000bda2aa0] 1-12
*12 ??? (kernel.release.t6041 + 6334440) [0xfffffe000bd3e7e8] 1-12
*12 lck_mtx_sleep + 212 (kernel.release.t6041 + 481724) [0xfffffe000b7a99bc] 1-12
*12 ??? (kernel.release.t6041 + 590028) [0xfffffe000b7c40cc] 1-12
*12 ??? (kernel.release.t6041 + 595068) [0xfffffe000b7c547c] 1-12
Operating system
macOS 26.5.1
Version of Jamulus
Jamulus 4.0.0 beta1
Additional context
Might be some multithreading bug? Cannot reproduce it consistently for now.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
- StatusShow more project fieldsDone
Activity
ann0see commentedon Jun 28, 2026
Could this be related to some corrupt packet or packet loss (I doubt this though)? I switched between Rock/Jazz directories repeatedly and could sometimes cause the hang. If I let the debugger continue, it recovers.
it might hang during sending
PROTMESSID_CLM_PING_MS_WITHNUMCLIENTS. It may or may not be related to the recent change to BSD/macOS in send_to_ipv4_with_tos.pljones commentedon Jun 29, 2026
Is this new with 4.0.0 beta1 or is it reproducible on 3.12.2?
ann0see commentedon Jun 29, 2026
I could not reproduce it in 3.12.2, just 4.0.0beta1. Let me check what happens if I disable ipv6.
ann0see commentedon Jun 29, 2026
Yes, I think it does not occur if I disable ipv6. And it does seem to be reproducible in 3.12.2,3.11.0 with -6.
3.8.1 I could not reproduce for now. But it's an intel build.
softins commentedon Jun 29, 2026
In
socket.cpp, find these lines:and add
&& 0to the#iftemporarily and recompile. That will use standardsendto()always, although TOS on IPv4 will not work when IPv6 is enabled. See if the hang no longer occurs.ann0see commentedon Jun 29, 2026
I'll try it. However, 3.10.0 (with -6) definitely hangs also on the Jazz genre. I think it doesn't occur before as we read a different config file.
softins commentedon Jun 29, 2026
Interesting, since 3.10.0 long predates the use of
sendmsg()ann0see commentedon Jun 29, 2026
Yes. Also with the && 0 it does still hang
softins commentedon Jun 29, 2026
I wonder if a Wireshark capture will offer any insight?
ann0see commentedon Jun 29, 2026
Will try.
ann0see commentedon Jun 29, 2026
Jazz -> classical hang: until no. 502 (filtered for Jamulus protocol).
pause thread -> code is at _sendto with your change appended.
continue in Xcode, works again, change to Jazz -> hangs again.
Now seems to hang forever.
Will send the pcap to team mail.
ann0see commentedon Jun 29, 2026
Last UDP payload: 0000f303001300021000332e31322e312d4a414d46414e2d3130138e
23 remaining items
ann0see commentedon Jul 2, 2026
GPT did some simulation and believes that it's priority invasion by the audio thread (but I wasn't always connected to a server - so I'm unsure if this actually holds!).
softins commentedon Jul 3, 2026
I'm still suspicious of the global lock Gemini said macOS uses when managing its dual-stack address mapping.
I plan to implement separate v4 and v6 sockets within
CSocketin the next few days, as time permits. Let's see if that helps.There will be other advantages to having separate sockets, such as being able correctly to handle
--serverbindipand a new--serverbindip6, and not having to map addresses within Jamulus. It also opens the possibility of adding--noipv4for a v6-only server in the future.ann0see commentedon Jul 3, 2026
I had some local AI analysis (there's not soo much new, but it gives some examples with debugging output)
Metadata
Summary
Jamulus hangs intermittently in the connect dialog on macOS when fetching the server list. The hang occurs inside
sendto()when the client attempts to send a packet to a server whose IPv4 address is sent over an IPv6 dual-stack socket using a V4Mapped address (::ffff:x.x.x.x).The
sendto()syscall blocks in the kernel when the destination is unreachable. While blocked, the socket's send buffer fills and the kernel sleeps waiting for space. The thread holds aQMutexduring the entiresendto()call, so any other thread attempting to send blocks on the mutex, creating a deadlock.Steps to Reproduce
File → Connect).::ffff:a.b.c.d),sendto()may block indefinitely.Note: The hang seems to only only occur when IPv6 is enabled (either via
-6flag or by default). With IPv4-only, the hang does not occur.Root Cause
1. IPv6 dual-stack socket + V4Mapped addresses
The hang occurs when IPv6 is enabled (either via
-6flag or by default). With IPv4-only, the hang does not occur. This confirms the issue is specific to the V4Mapped address path: whenbIPv6Availableis true, IPv4 destinations are sent via an IPv6 socket using a V4Mapped address (::ffff:x.x.x.x).When
sendto()is called with a V4Mapped IPv6 address on macOS, and the destination is unreachable (no route, host down, etc.), the kernel does not returnEHOSTUNREACHimmediately. Instead, the socket is placed in a sleeping state inside the kernel (lck_mtx_sleepviasosend), and the syscall blocks until the send buffer has space or a route is found. This can take minutes or never return.2. Mutex held during blocking call
In
CSocket::SendPacket()(socket.cpp:378), aQMutexLockeracquires&Mutexbefore callingsendto(). The mutex is held for the entire duration of the blocking syscall:3. Retry loop amplifies the problem
Note: This might be an invalid AI finding.
The existing retry loop (socket.cpp:389) retries the send up to 2 times when
sendto()fails. If the firstsendto()blocks, the retry never executes because the thread is stuck in the kernel.4. Stack trace confirmation
From the original issue trace:
5.
sendto()stops atb.loassembly instructionTe hang occurs inside
sendto()at theb.lo(branch-on-lower/unordered) assembly instruction — indicating the kernel is maybe waiting for a route to be found (not "unreachable" in the ICMP sense).redundant information removed
9. Xcode status type warning
Xcode complains
statusshould bessize_tnotint, but this does not change anything.10. Reproducible with
-6flagann0eeconfirmed the hang is reproducible in 3.10.0 (with-6) on the Jazz genre, and not reproducible in 3.12.2 without-6.Evidence
Code path (socket.cpp:391-430)
Observed 50-second gap (Xcode debugger pause, and continue)
Debug logging reveals a 50-second gap (which is roughly the time the app is paused in Xcode) where no log lines are produced:
Key observations from the reproduction:
O_NONBLOCK= NOT SET— the socket is in blocking mode, sosendtoblocks in the kernelerrno= 0— macOS does not return a proper error code (likeEHOSTUNREACH) for unreachable V4Mapped destinations. Instead, the kernel sleeps and returns-1witherrno=0sendtoreturns in milliseconds, proving the kernel was not responsible for the delayExecution trace at the point of the hang
The following log output represents the complete state of the program right before
sendto()is called:This is the last log output before the hang. The very next line of code is:
At this point:
status = 0(unchanged —sendtohas not returned yet)Mutexis locked byQMutexLockerUdpSocketAddris fully constructed with V4MAPPED addressiVecSizeOut = 1026bIPv6Available = truefcntlnot called to setO_NONBLOCK)The next log line (
>> sendto (V4MAPPED) returned: status= -1) appears 50+ seconds later, after the process was unpaused from the Xcode debugger. The kernel did not sleep for 50 seconds — the process was suspended by the debugger.When the process resumes (unpaused),
sendtoreturns in milliseconds (4ms for the retry), proving the kernel was not responsible for the delay. The 50-second gap was entirely due to the debugger pause.pljones commentedon Jul 4, 2026
O_NONBLOCK= NOT SET-- What does this do on BSD / macOS for UDP?This is noise from the LLM, right? It keeps repeating it but it's not actually useful. Why did the debugger pause?
ann0see commentedon Jul 4, 2026
Yes it's noise from the LLM. I paused it manually via Xcode.
ann0see commentedon Jul 4, 2026
https://stackoverflow.com/questions/2889901/is-o-nonblock-being-set-a-property-of-the-file-descriptor-or-underlying-file
I think it's a parameter for blocking/non blocking io.
pljones commentedon Jul 4, 2026
Mmm. I guess it's a bit like a disk write: "stick the content of this buffer onto the outbound network and don't come back until you've finished" (with O_NONBLOCK not set). So if there's something wrong with the descriptor of the buffer (like the header) and something breaks, you find out. With O_NONBLOCK true, you'd need some other way to know whether the network call succeeded.
So I guess my question is "how do you handle a call when O_NONBLOCK is true compared to when O_NONBLOCK is false for a UDP packet sendto() on macOS/BSD?"
Note
Claude says
UDP sendto() behavior with O_NONBLOCK on macOS/BSD
Source: Claude
Date: 04/07/2026, 18:13:12
Link: https://claude.ai/chat/85aa6eac-8036-4f47-b392-8fd0bda06322
Model: claude-sonnet-5
Prompt:
how do you handle a call when O_NONBLOCK is true compared to when O_NONBLOCK is false for a UDP packet sendto() on macOS/BSD?
Response:
Blocking (default, O_NONBLOCK not set)
If the socket send buffer doesn't have room for the packet,
sendto()blocks the calling thread until space becomes available (or until a signal interrupts it). For UDP this is rare — it mostly happens when you're sending faster than the interface/driver can drain the socket buffer (SO_SNDBUF), not because of a remote receiver (UDP has no flow control back to the sender).Non-blocking (O_NONBLOCK set)
If the send buffer can't accept the packet immediately,
sendto()returns -1 immediately and setserrno = EWOULDBLOCK(==EAGAINon macOS/BSD). It never blocks the thread.Practical edge cases to test/handle
Oversized datagram: If the packet exceeds what the socket/route can send (larger than the max UDP payload for the path, e.g. >65507 bytes, or exceeding
IP_MAXPACKETconstraints), you getEMSGSIZE— this happens regardless of O_NONBLOCK, immediately, not a blocking condition at all.No local buffer space:
ENOBUFScan occur on BSD/macOS when the interface queue is full — this is distinct fromEWOULDBLOCK. It can happen even on a blocking socket in some driver-queue-full scenarios (particularly with high packet rates), so don't assume blocking sockets never return an error for buffer issues.ICMP-derived errors on "connected" UDP sockets: If you called
connect()on the UDP socket, a prior ICMP "port unreachable" can cause a subsequentsendto()/send()to returnECONNREFUSED— this is asynchronous and independent of blocking mode.Signal interruption: A blocking
sendto()can return-1/EINTRif interrupted by a signal before space is available — always check for and retry onEINTRin blocking code.Partial sends don't apply to UDP: unlike TCP, a successful
sendto()return value is always the full datagram size or an error — no partial-write handling needed.Testing this yourself
Run the same loop without
O_NONBLOCKand you should observe the loop's throughput throttle down (blocking) instead of returning errors — a good way to confirm the behavioral difference empirically on your system.One caveat worth flagging: exact
errnobehavior forENOBUFSvsEWOULDBLOCKunder buffer pressure can vary slightly between macOS versions and other BSDs (FreeBSD/OpenBSD), so if this is for production code, checkman 2 sendon the specific target OS rather than assuming Linux-style semantics apply.ann0see commentedon Jul 10, 2026
I might be related to the SRV lookup - not sure. I had a slight hang (short) while trying over a ipv4 only + ipv6 over vpn connection. I'll try what happens with the #3774 version.
IPv6/IPv4 dual-stack socket created
qt.multimedia.ffmpeg: Using Qt multimedia with FFmpeg version 7.1.2 LGPL version 2.1 or later
resolved choral.jamulus.io to a single SRV record: choral.jamulus.io:22724
resolved choral.jamulus.io to a single SRV record: choral.jamulus.io:22724
softins commentedon Jul 10, 2026
That doesn't look normal. Looking up the standard directories should not use SRV at all, as they have port numbers embedded in the app.
The only reason SRV would be used in your example would be if choral.jamulus.io was given without port number as a custom directory.
ann0see commentedon Jul 11, 2026
Correct. That could be.
ann0see commentedon Jul 15, 2026
(will reopen if the bug comes up again - not in the respective network where this bug occurred for now)