Concurrent Web Crawling in Ruby with Async | Los Angeles AI Apps

Los Angeles AI Apps is a digital product agency that builds custom Web applications with AI Integration. Launch an AI MVP in as little as one week.Of all the languages one could use to build a web crawler, Ruby is not one that immediately stands out. Web crawling is I/O heavy; it involves downloading dozens, hundreds, or even thousands of web pages. The proven solution to I/O heavy workloads is an event-driven, non-blocking architecture, something usually associated with languages like Go, JavaScript or Elixir. I’d like to put forth Ruby as a great language for handling I/O bound workloads, especially when paired with the Async library. I’m sure Go, JavaScript and Elixir are fine languages; however, in my opinion, they don’t quite match Ruby in terms of readability and expressiveness. Let’s first build a web crawler without concurrency, downloading pages one at a time. Then, I will show how to add concurrency in a lightweight manner and with minimal changes to the code.