Speed Up Your Data Processing

Parallel and Asynchronous Programming in Data Science

Chin Hwee Ong

ASYNC / Concurreny Data Data Science Multi-Processing Multi-Threading

See in schedule

In a data science project, one of the biggest bottlenecks (in terms of time) is the constant wait for the data processing code to finish executing. Slow code, as well as connectivity issues, affect every step of a typical data science workflow — be it for network I/O operations or computation-driven workloads. In this talk, I will be sharing about common bottlenecks in data processing within a typical data science workflow, and exploring the use of parallel and asynchronous programming using concurrent.futures module in Python to speed up your data processing pipelines so that you could focus more on getting value out of your data. Through real-life analogies, you will learn about:

1. Sequential vs parallel processing,
2. Synchronous vs asynchronous execution,
3. Network I/O operations vs computation-driven workloads in a data science workflow,
4. When is parallelism and asynchronous programming a good idea,
5. How to implement parallel and asynchronous programming using concurrent.futures module to speed up your data processing pipelines

This talk assumes basic understanding of data pipelines and data science workflows. While the main target audience are data scientists and engineers building data pipelines, the talk is designed such that anyone with a basic understanding of the Python language would be able to understand the illustrated concepts and use cases.

Type: Talk (30 mins); Python level: Intermediate; Domain level: Beginner


Chin Hwee Ong

ST Engineering

Ong Chin Hwee is a data engineer, aspiring polymath and Industry 4.0 enthusiast who happens to have a background in aerospace engineering and computational modelling. As an avid user of the pandas library and a beneficiary of open source, Chin Hwee is a contributor to the documentation for pandas 1.0 and has a not-so-secret wish for data processing codes to run faster.