Talk: Speed Up Your Data Processing: Parallel and Asynchronous Programming in Data Science (PyCon TW 2020 Edition)

Written on September 9, 2020
Speaking at PyCon Taiwan has been one of my key priorities in year 2020 even before the pandemic (another of my key prioriies was to speak at a conference outside of Asia). The fact that there will be an offline audience at the other end of the remote call is also a "pull" factor in my decision to speak at PyCon Taiwan this year - I really miss being able to see live audience responses at a physical conference!
Read More

Talk: Speed Up Your Data Processing (EuroPython 2020 Edition)

Written on July 24, 2020
I originally designed this talk to be interactive and audience-driven, with the pace of the talk driven by casual "coffee shop" banters with the audience. With the COVID-19 pandemic showing no signs of abating, my original plan of making my European speaking debut at Ljubljana was disrupted. I still wanted to make my European speaking debut with the same talk somehow; hence, I submitted the talk proposal for EuroPython 2020 and hoped for the best.
Read More

Lightning Talk: Just-in-Time with Numba - 7-minute PyLadies version

Written on March 29, 2020
One fine day in the morning of 28 March 2020 Singapore time, I came across a tweet from PyLadies calling for lightning talk submissions for their International Women's Month Lightning Talks Zoom call. Just a day ago, my data scientist friend mentioned about Numba on Facebook, and I happened to have spoken quite a bit about Numba in my first conference talk last year.
Read More

Talk: Speed Up Your Data Processing: Parallel and Asynchronous Programming in Python

Written on March 21, 2020
Year 2020 started out bad. Real bad. The COVID-19 pandemic led to a string of cancellations for tech events (including PyCon) and travel restrictions. As a result of travel restrictions and Business Continuity Plans, many speakers were not able to deliver their talks in person at the eleventh hour, participation was greatly reduced by around 90%, and even the FOSSASIA organizers could not make it to the venue in person due to COVID-19 restrictions. Still, the organizers made the decision to proceed with the event with a mix of offline and online talks with live streaming and chats.
Read More

TIL: DataFrame reshaping in Pandas - melt, unstack

Written on February 20, 2020
As a data engineer, part of my daily work involves performing data processing and manipulation on raw data into data that is ready for analysis. As my development team primarily uses Python for our data science workflow, we often use Pandas to perform operations and transformations on datasets before analysing the data. While we primarily use Pandas for data cleaning and engineering as part of the data science process, sometimes we also have to perform complex data transformations to obtain actionable insights that business users can leverage on to improve their processes.
Read More

Contributing to pandas documentation for the first time - and lessons learnt

Written on November 8, 2019
To mark my first year as a data engineer, I started thinking "How can I contribute back to the community that enables my work for the past year?" I came across an open issue on documentation for pandas, a popular open-source Python library for data analysis and manipulation, and decided to give it a try. Here's a work-in-progress developer log on the lessons learnt through contributing documentation to an open-source Python library for the first time - and how contributing to open-source projects is not as scary as it might seem to be.
Read More

Understanding Python Dependency Management using pideptree

Written on October 13, 2019
Dependency management is important, as packages depend on versions of other core packages in order to run as intended. Typically in a Python project, dependencies are downloaded using a requirements.txt file, which lists the packages and their dependencies as a flat file. While the package versions are included in the requirements.txt file, the dependency relationships are not explicitly stated.
Read More

TIL: Migrating Git repositories manually from GitLab to Azure DevOps (TFS)

Written on October 12, 2019
My development team has been using GitLab on-premise to manage their code repositories. As we are moving development work to our new on-premise development cloud with expanded processing capabilities, we also need to migrate our code repositories to the new development cloud which uses Azure DevOps Team Foundation Server (TFS) for Git workflows. To support the chief architect in the migration, I prepared a quick migration guide for the team's move to Azure DevOps TFS.
Read More

Musings about Remote Development with Visual Studio Code

Written on August 22, 2019
Keeping codes and configuration files in sync between client machine and remote server used to be a drawn-out exercise in personal responsibility via SFTP/SCP. VS Code Remote Development looks to change that - for the better. Here's my notes on VS Code Remote Development.
Read More

Accelerating Batch Processing of Images in Python — with gsutil, numba and concurrent.futures

Written on May 27, 2019
In a data science project, one of the biggest bottlenecks (in terms of time) is the constant wait for the data processing code to finish executing. Sometimes, the gigantic execution times even end up making the project infeasible and often forces a data scientist to work with only a subset of the entire dataset, depriving the data scientist of insights and performance improvements that could be obtained with a larger dataset.
Read More