Repository navigation
Description
We had some type of downtime on Spotbit on Sunday:
daemon.log:
Dec 7 11:05:05 localhost systemd[1]: spotbit.service: Main process exited, code=killed, status=9/KILL
Dec 7 11:05:05 localhost systemd[1]: spotbit.service: Failed with result 'signal'.
SIGKILL indicates to me that was user generated, but there was no one logged in at the time, so I assume some process choose to kill spotbit for some reason.
The user.log shows active usage up until shortly before that:
user.log:
Dec 7 11:04:26 localhost python3[431]: 127.0.0.1 - - [07/Dec/2020 11:04:26] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
Dec 7 11:04:26 localhost python3[431]: INFO:werkzeug:127.0.0.1 - - [07/Dec/2020 11:04:26] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
Dec 7 11:04:27 localhost python3[431]: INFO:spotbit:[2020-12-07 11:04:27.185816] | inserted into okcoin USDT 40000 times
Dec 7 11:04:51 localhost python3[431]: 127.0.0.1 - - [07/Dec/2020 11:04:51] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
Dec 7 11:04:51 localhost python3[431]: INFO:werkzeug:127.0.0.1 - - [07/Dec/2020 11:04:51] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
Dec 7 11:04:51 localhost python3[431]: 127.0.0.1 - - [07/Dec/2020 11:04:51] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
Dec 7 11:04:51 localhost python3[431]: INFO:werkzeug:127.0.0.1 - - [07/Dec/2020 11:04:51] "#033[37mGET /now/usd/kraken HTTP/1.1#033[0m" 200 -
There's a confusing mess of spotbit.err and spotbit.log logs in various places. The only error file, /root/spotbit.err, is three days old (e.g., a day before the crash) and the most up-to-date log file, /home/spotbit/.spotbit/spotbit.log, shows very little between 10-06 and 10-09. (It's also hard to read because most lines don't have a date.)
The system was in a very bad state when I found it. spotbit was not running. I restarted it, then I restarted it and tor, and still nothing was being passed to spotbit. I rebooted the system and this time tor didn't start right, and I rebooted the system a second time before everything finally came back into sync. I have to assume that's all due to something not shutting down cleanly, most likely the database, which was throwing locked messages after the first reboot.
So, I know this isn't much info, here's a list of suggested things that might help:
- Make sure
spotbitis logging an error file - See if we can improve
spotbit.logto at least log date-time on all lines - Investigate if anything in
tororspotbitmight issue aSIGKILL - See if there is any way to better restart
spotbitcleanly after a messySIGKILL - Monitor our services, including spotbit (I'm going to probably take this one on with Christopher next week)
Activity
shannona commentedon Dec 9, 2020
@javiervargas found the answer in the system logs: an out of memory error. That answers point #3 above, which was the reason for this particular crash. spotbit is on a tiny 1GB-memory machine, and it might be that's not sufficient memory for large lookups. I've tried to expand things by pushing the swap from 512MB to the recommend 1GB. If that's insufficient, the machine probably needs to be pushed to a larger size.
I leave the following possible issues:
Investigate if anything in tor or spotbit might issue a SIGKILLjaviervargas commentedon Dec 9, 2020
This is the log:
nochiel commentedon Nov 25, 2022
This is fixed in the latest refactor in master to improve stability and make Spotbit more lightweight. If Spotbit crashes now, we should be able to pinpoint specific issues easily.