Speed Up Your Data Processing

Parallel and Asynchronous Programming in Data Science

Chin Hwee Ong

ASYNC / Concurreny Data Data Science Multi-Processing Multi-Threading

See in schedule Download/View Slides

In a data science project, one of the biggest bottlenecks (in terms of time) is the constant wait for the data processing code to finish executing. Slow code, as well as connectivity issues, affect every step of a typical data science workflow — be it for network I/O operations or computation-driven workloads. In this talk, I will be sharing about common bottlenecks in data processing within a typical data science workflow, and exploring the use of parallel and asynchronous programming using concurrent.futures module in Python to speed up your data processing pipelines so that you could focus more on getting value out of your data. Through real-life analogies, you will learn about:

1. Sequential vs parallel processing,
2. Synchronous vs asynchronous execution,
3. Network I/O operations vs computation-driven workloads in a data science workflow,
4. When is parallelism and asynchronous programming a good idea,
5. How to implement parallel and asynchronous programming using concurrent.futures module to speed up your data processing pipelines

This talk assumes basic understanding of data pipelines and data science workflows. While the main target audience are data scientists and engineers building data pipelines, the talk is designed such that anyone with a basic understanding of the Python language would be able to understand the illustrated concepts and use cases.

Type: Talk (30 mins); Python level: Intermediate; Domain level: Beginner