Effective Pandas: Patterns for Data Manipulation (Treading on Python)

Product Information

Profiling let you make the most pragmatic decisions for the least overall effort: Code run “fast enough” and “lean enough” This looks ok, and you have probably seen code like this. You’ve probably written code like this if you are reading this post. Static trie using Cython bindings to an external library -> it cannot be modified after construction Scikit-learn’s DictVectorizer and FeatureHasher NumExpr breaks the long vectors into shorter, cache-friendly chunks and processes each in series, so local chunks of results are calculated in a cache-friendly way

Provide a suite of parallelization solutions that scales from a single core on a laptop to multicore machines to thousands of cores in a cluster. Lets us share higher-level Python objects between processes as managed shared objects; the lower-level objects are wrapped in proxy objects I feel Matt Harrison knows its stuff, but he also cares to teach it. For example: giving real world examples makes it easy to relate to the problem at hand. Short chapters are rightly sized knowledge pills. The summary and exercises at the end of each help to make sure one understands.

The book goes beyond explaining the data structures and methods that underpin Pandas, but he also provides a ton of practical advice regarding best practices in data manipulation and transformations. Task requires mainly linear algebra and matrix manipulations (multiplication, addition, Fourier transforms) That operation was made much easier by this addition in 2014, which lets you slice arbitrary levels of a MultiIndex.. This book contains best practices with Pandas, essential for anyone who wants to improve their data manipulation skills. JIT (just in time): Numba, PyPy -> you don’t have to do much work up front, but you have a “cold start” problem -> impressive speedups with little manual intervention

Builds on Dask to provide three parallelized options with very simple calls: apply, resample and rolling Vaex Get started In my opinion, one of the most underutilized methods in Pandas is the .assign method. My take (which may be extreme) is that .assign is the one true way to create a new column or update an existing column. For those who don't want to read the whole article, here are my reasons for preferring .assign: Python 3.x, all strings are Unicode by default, and if you want to deal in bytes, you’ll explicitly create a byte sequence Chaining is writing a series of operations to a dataframe to manipulate it. Each operation works on the result of the previous operation.In fact, writing chains forces you to think about each step that you will perform on the data. It is a constraint that I find allows me to write better code. (Again, you might think I’m spewing crazy talk. Try it out and see how it helps with your code.) I've been using Pandas for about 10 years, and I still improved my Pandas skill working through the Effective Pandas... Matt Harrison is ready to drop some knowledge on you and have you riffing your own data manipulation solos like you're Slash in "November Rain", or Prince in "Purple Rain"... delayed decorator: wraps our target function so it can be applied to the instantiated Parallel object via an iterator Not a good tool for tasks that require exceedingly large amounts of data, many conditional manipulations of the data, or changing data