ep_user_pad crashes every 8 hours

[quenenni]

Every 8 hours precisely, this plugin crashes and crashes etherpad.

[ERROR] console - Error: Connection lost: The server closed the connection.
    at Protocol.end (/home/etherpad/node_modules/ep_user_pad/node_modules/mysql/lib/protocol/Protocol.js:73:13)
    at Socket.onend (stream.js:66:10)
    at Socket.EventEmitter.emit (events.js:123:20)
    at TCP.onread (net.js:417:51)

I thought at the beginning that was server related, like a cron doing something ep_user_pad didn't like, but no.

When I stop etherpad and restart it, the plugin crashes again every 8 hours, but at differents times.

If I remove ep_user_pad (but keep ep_user_pad_frontend), no crashes happen.

Do you have any idea what can cause this? Like a timer to clean something every x hours?

[youhost]

Same issue here.

[quenenni]

I removed the ep_user_pad to only keep ep_user_pad_frontend active and this exact same error still occurs, but on the frontend part:

[ERROR] console - Error: Connection lost: The server closed the connection.
    at Protocol.end (/home/etherpad/node_modules/ep_user_pad_frontend/node_modules/mysql/lib/protocol/Protocol.js:73:13)
    at Socket.onend (stream.js:66:10)
    at Socket.EventEmitter.emit (events.js:123:20)
    at TCP.onread (net.js:417:51)

Maybe it comes from the version of the mysql extension you are using in this plugin:

"version": "2.0.0-alpha9",

I installed v2.0.0-rc2 in ep_user_pad_frontend.

I'll see if it works this way.

I tried with the last version (v2.2) of mysql, but lots of new dependencies to manage and I blocked at some point.

[youhost]

Hello,
I am happy to see that you are working on this.
I am sorry I can not help with the tecnical part on module as we are not developers.
But if the module shall work properly we have a plan to redesign the UI and we would be happy to share it.

[quenenni]

Hello,

Well I'm trying, but I must say that this bug seems quite difficult to resolve.
I have no idea at the moment where the problem can come from.

The new version of mysql didn't change anything... 8 hours later.. Bang..

I'm trying first to make the plugin 'ep_tables' works. Then I'll try to find some times for this one.

In the meantime, I resolved this problem by using supervisord (http://supervisord.org/) to restart etherpad each time it crashes.. It's ugly, but it works :)

I'm also happy to see your interest in redesigning the UI 👍
I opened a ticket with things I've done:
aoberegg/ep_user_pad_frontend#6

[aoberegg]

Hi All!
I'm really happy to see that my plugin is used by someone but I'm sorry to tell you that I don't have time to enhance the plugin at the moment. I also really don't know where this bug could be located.

[youhost]

Hi all,
I will be following this and as soon as it works I confirm I will work on the UI.
I will follow your progress and sorry again for beeing unable to help you on the dev side.
I've checked supervisord.org but was unable to understand anything.
Hope to ear from you soon...

[youhost]

Hi quenenni,
Did you found some time to check the code?
I've seen there is a variant of this plugin, but I'm afraid that changes are only on the front end...
https://github.com/ckubu/ep_frontend_community

[quenenni]

It's still on my todo list, but I'm still working on the ep_tables plugin to fix it (nearly done).
I don't have much time to develop these plugins, that's why I'm so slow.

But I still plan to do it.

@aoberegg thanks for giving me access. As soon as I finished ep_table, I'll check here. But I found a variant of your plugin where the dev seems quite active. He already fusioned the frontend & admin part, did a lot of cleaning, fix stuffs,.. so I'll go check there also. Maybe his version doesn't have this problem anymore (I really don't where to look for to resolve it)
https://github.com/vltugraz/ep_user_pads

@youhost the variant you found doesn't seem to be very active. He didn't do a lot in 2 months, so I think we can forget it for the moment.

[quenenni]

@youhost Is your server a Linux Debian server?
If yes, there is a supervisor package: apt-get install supervisor

After that, create the file /etc/supervisor/conf.d/etherpad.conf (and adapt to your system):

[program:etherpad]
command=/home/etherpad/bin/run.sh
process_name=%(program_name)s
numprocs=1
directory=/tmp
umask=022
priority=999
autostart=true
autorestart=unexpected
startsecs=15
startretries=10
exitcodes=2
stopsignal=TERM
stopwaitsecs=10
user=etherpad
redirect_stderr=true
stdout_logfile=/var/log/etherpad/etherpad.log
stdout_logfile_maxbytes=5MB
stdout_logfile_backups=10
serverurl=AUTO

Check also /etc/supervisor/supervisord.conf (few things to change)

If set to daemon mode (nodaemon = false), restart supervisor (/etc/init.d/supervisor restart), if not, start supervisor in a console (command: supervisord)

[youhost]

@quenenni thank you very much for the supervisor indication setup.
our etherpad is running on a debian system and I appreciate your indications.
Time is never enough and I understand that much time is necessary to do everything.
In the meanwhile I will check https://github.com/vltugraz/ep_user_pads too.

[Greyshaun33]

Has this issue been resolved as of yet? (I know i can just make something to restart upon crash but its handy to have as little running as possible)

[loadavg]

Any fix for this yet ? I am up and running and loving it but this is a pretty major issue! I am on Centos 6 so the debian tool above doesnt work for me...

[quenenni]

I just found the source of the problem.
It's the mysql directive wait_timeout.
("The number of seconds the server waits for activity on an interactive connection before closing it.")

By default, it's set to 28800 (seconds) = 8h.

I still don't know how to fix it, but if anyone has a suggestion, I'm all ear.
I guess we should find the way the plugin manage the mysql connexions. There must be an option to not keep the mysql connexions actives, but connect and disconnect as we need them.

@aoberegg, maybe you can check this out.

[quenenni]

Well, I found something that seems promising, but I'm not sure yet what consequences it can have.

In hooks.js, lots of mysql connections are made and maybe try to stay active.
Nowhere the connections are closed after being used, so I did that.

I added at the end of "exports.expressCreateServer" (line 1662 in hooks.js):

    if (connection) {
      connection.end(cb);
      connection2.end(cb);
    }

    return cb();

And my first tests show that etherpad doesn't crash anymore and everything I tested on a pad seems to work fine.

If someone else could test also that and report if he sees something wrong with the change..

[quenenni]

Well, it's not working.
There are problems when accessing the plugin pages (the pads are working well).

The problem seems to be the use of the lib mysql directly instead of using the DB / ueberDB libs integrated in etherpad that manage the opening and closing of the DB connections.

I have to search for another way to fix that.

[Greyshaun33]

Hi, Just wondering where you found the "It's the mysql directive wait_timeout." so i could change the default value from 28800 to something like 12000000 So as it takes forever to time out because having to reset it once in a blue moon will be better then resetting it every 8 Hours.

[quenenni]

I don't knonw if any of you tested the alternative, the one here https://github.com/vltugraz/ep_user_pads.
I did and, unfortunately, the problem is still there for me...

Can anyone confirm he has the same thing?

I think I'm gonna wait for the project that framapad started. Thy did a (successfull) crowfunding campaign in order to create mypads, a plugin like this one.
http://www.ulule.com/etherpad-framapad/
It (hopefully) should be ready for the end of the year.

@Greyshaun33
In your mysql conf.
On a debian linux server, it's in /etc/mysql/my.cnf.
It's possible that directive is not present. you can just add it and restart the mysql server.

[BlackyFox]

Hi all !
Has anyone found a way to bypass that bug ?
I mean, the problem could be solved if we had a something like a ping to keep the connexion alive and not idle.