Jacob Paris
← Back to all content

Offload work to a background process in Node JS

Javascript is a single threaded language, so the moment you start running some intensive task, it will block the event loop and prevent the rest of your application from working.

while (10 > 5) {
"This will never end and will cause the browser tab to freeze or the Node.js process to stall indefinitely.",

While most applications won't have any forever-tasks that stall it completely, functions that take a long time to run can degrade the user experience elsewhere in the app.

This can cause dropped network requests, slow page loads, and other issues.

If it's javascript's bottleneck that a single javascript process can only run one thread at a time, then the solution must be to use multiple javascript processes to run different parts of our application.

By offloading the work to a separate process, we can keep our main application running and responsive while the background tasks run.

The simplest setup is to use a singleton module to spawn a new Node.js process that runs another javascript file.

import { spawn } from "child_process"
import { join } from "path"
let subprocess: ReturnType<typeof spawn> | null = null
declare global {
var subprocess: ReturnType<typeof spawn> | null
if (!global.subprocess) {
global.subprocess = spawn(
join(__dirname, "subprocess.js"),
stdio: "inherit",
shell: true,
detached: true,
subprocess = global.subprocess
export default subprocess

This will successfully offload the work to a separate process, but to be a practical solution for most developers, there are many more things to consider.

  • How do we communicate between the two processes?
  • How do we handle errors?
  • How do we run a typescript file?
  • How do we handle the subprocess crashing?

Rather than solving these one by one, we can use a battle-tested library like BullMQ that supports background processes natively to handle all of these concerns.

Professional headshot

Hey there! I'm a developer, designer, and digital nomad building cool things with Remix, and I'm also writing Moulton, the Remix Community Newsletter

About once per month, I send an email with:

  • New guides and tutorials
  • Upcoming talks, meetups, and events
  • Cool new libraries and packages
  • What's new in the latest versions of Remix

Stay up to date with everything in the Remix community by entering your email below.

Unsubscribe at any time.