Using upstart to detect node.js app crashes
Here at Gitter we love writing node.js applications, but like any real-world application, things occassionally go wrong. Nine times out of ten, this is down to an unhandled exception in node.js (although we're seeing a lot less of these since we started using promises extensively).
We wanted a way of detecting process crashes and reporting the state of the server immediately after the process terminated.
respawn option immediately restarts our services on termination, but it doesn't have an easy way of executing a script when the process terminated abnormally. And although the
post-stop script executes, its not possible to tell whether the shutdown is due to a planned service shutdown or a crash.
Upstart can execute scripts before and after a process starts and stops. It has hooks for:
When a service is shutdown correctly, the
post-stop scripts execute. However, when a service crashes, upstart will execute the
post-stop script, but not the
pre-stop script. We can use this to detect a crash.
For each of the node upstart services we add the following stanzas to the upstart configuration:
# # Crash reporting # pre-start exec /opt/upstart-monitor/bin/pre-start pre-stop exec /opt/upstart-monitor/bin/pre-stop post-stop exec /opt/upstart-monitor/bin/post-stop
The three scripts referenced are tiny. Here's how they look:
This script creates a file when the service starts.
#!/bin/sh touch /var/run/$UPSTART_JOB.shutdown
This script removes the file on proper shutdown.
#!/bin/sh rm /var/run/$UPSTART_JOB.shutdown
This script checks to see whether proper shutdown has
been performed and if it hasn't notifies devops.
#!/bin/sh SHUTDOWN_FILE=/var/run/$UPSTART_JOB.shutdown if [ -f $SHUTDOWN_FILE ]; then rm -f $SHUTDOWN_FILE # The service did not shutdown correctly, # perform your diagnostics or notify your # administrators over here fi
Do you have a better way of doing this? Pop in to Gitter and let us know!