Best way to queue or thread CPU intensive tasks so Node server doesn't crash?
I wrote code that forks and queues up and stops child processes but the processes themselves use up quite a bit of CPU.
Is there a library that's particularly helpful with building these types of async/cron-like Node services? Wasn't finding anything in my search.
Should I consider offloading that to another server or is it just time to up the CPU of my virtual machine?
Although I have never used it or anything, I've heard about forking child processes. I think this talk covers it - https://www.youtube.com/watch?v=K8spO4hHMhg
Not sure if it'll help though. You should also try asking in the subreddit /r/nodejs
You should try using a queue eg. Bull queue.
You can run the workers on the same server or somewhere else (it's backed by redis).
Notre sure if it 100% fits your use case since as far as I understand you seem to want to wait for the job to complete before sending a response back or something? In which case you should wait on the output of the queued job (that's writes to a db or something)
Bull looks like exactly what I need. I don't have redis setup for my project but shouldn't be too tough. Thanks dude!!! 🙌 Will let you know how it goes.
There's cool UIs to monitor the queues as well, I've got an example of a working setup at https://github.com/HugoDF/express-bull-es6 (post explaining what's going on at https://codewithhugo.com/bring-redux-to-your-queue-logic-an-express-setup-with-es6-and-bull-queue/).
Bull is the only queue with a modern interface (ie Promises) that I found when digging around.
Let me know if you get stuck anywhere.