Stuff a Dev Googles: Keep a Unix Process Running after SSH Connection Times Out

As a software developer, I would say my job is 60% googling stuff, 25% talking to people, and 15% writing actual code. Needless to say, “google-fu” is a critical part of my job. This article is a part of a series I intend to maintain on “Stuff a Dev Googles.” The point of the series is to share not only what I’ve learned but the process by which I’ve learned it.

A while ago I had to run a script that took about 30 hours to run. Its purpose was to check hundreds of thousands of urls to see if what was supposed to exist at that url actually existed. Essentially, it checked that requesting a url returned a 200, successful, response. There wasn’t much I could do to make this code run any faster at the time other than using a HEAD request rather than a GET request. Therefore, I was stuck just running the slow script.

Note: I started writing this almost a year ago, and I’m just now finally deciding to publish it. I’ve learned some things since then, and I do have some ideas for a better way to do this. I could have written the script in a way that would have so that I could perform the operations in batches, then I could have run the scripts in discrete jobs and queued them or something like that. However, the issue I was trying to fix was also time sensitive. There wasn’t enough time to figure out an elegant way to solve the problem. So with that in mind, let’s continue with my story.

When I tried to run the script for the first time, I ran into an issue. I sshed into the server, started the script in the afternoon, went to bed that night, and checked on it in the morning. To my dismay I was greeted with this:

Failed: Broken pipe

In the middle of running my script the ssh connection was severed and my process died.

I mentioned this to one of my coworkers and he told me that I could run the process in the background to fix this issue.

So I googled…

Google Search for “running command in the background linux”

and I selected this result

my_command > output.log 2>&1 &

But this command is not quite what I wanted. That’s because even though the command runs in the background, it will still stop executing if the ssh connection is broken. I asked a coworker about my problem and he said that what I actually wanted to do is prevent the ssh connection from timing out due to the runtime of the script. He mentioned that something called nohup could do this.

I googled

I found this result that looked very promising.

I could run nohup my_command to keep the process running and prevent timeout of the ssh connection.

The Answer

nohup my_command

But what did nohup actually do? To find that out I googled nohup linux

And got this as my first search result:

This site explained that nohup executes another program while ignoring all SIGHUP (hangup) signals. ASIGHUP signal gets sent when the terminal that is executing the process is closed. When a ssh connection times out, it closes all terminals, and normally any process that was running is terminated because aSIGHUP is sent, telling the process to kill itself. Therefore whennohup ignores the SIGHUP it prevents the process from being killed. From the article:

The nohup command executes another program specified as its argument and ignores all SIGHUP (hangup) signals. SIGHUP is a signal that is sent to a process when its controlling terminal is closed.

Usually, when you run a program over SSH, if your connection drops or you log out, the session is terminated, and all the processes executed from the terminal will stop. This is where the nohup command comes handy. It ignores all hangup signals, and the process will continue to run.

Low and behold, I had found the answer I needed. As you might have noticed, this time the key to solving the issue this time was to ask a coworker, so actually my google-fu skills didn’t quite get me to solve the problem. However, I did get to learn about how to run a linux command in the background in the process. I also learned why the Broken Pipe was happening — namely that the ssh connection timed out and killed the process.

I realized that the better thing to google would have been:

That search gives the correct answer in the second search result.

So, there you have it. Want to run something that will normally get killed when a ssh connection times out? Use nohup !