Mini Shell

==================================
A Guide to Masked Arrays in NumPy
==================================

.. Contents::

See http://www.scipy.org/scipy/numpy/wiki/MaskedArray (dead link)
for updates of this document.


History
-------

As a regular user of MaskedArray, I (Pierre G.F. Gerard-Marchant) became
increasingly frustrated with the subclassing of masked arrays (even if
I can only blame my inexperience). I needed to develop a class of arrays
that could store some additional information along with numerical values,
while keeping the possibility for missing data (picture storing a series
of dates along with measurements, what would later become the `TimeSeries
Scikit <http://projects.scipy.org/scipy/scikits/wiki/TimeSeries>`__
(dead link).

I started to implement such a class, but then quickly realized that
any additional information disappeared when processing these subarrays
(for example, adding a constant value to a subarray would erase its
dates). I ended up writing the equivalent of *numpy.core.ma* for my
particular class, ufuncs included. Everything went fine until I needed to
subclass my new class, when more problems showed up: some attributes of
the new subclass were lost during processing. I identified the culprit as
MaskedArray, which returns masked ndarrays when I expected masked
arrays of my class. I was preparing myself to rewrite *numpy.core.ma*
when I forced myself to learn how to subclass ndarrays. As I became more
familiar with the *__new__* and *__array_finalize__* methods,
I started to wonder why masked arrays were objects, and not ndarrays,
and whether it wouldn't be more convenient for subclassing if they did
behave like regular ndarrays.

The new *maskedarray* is what I eventually come up with. The
main differences with the initial *numpy.core.ma* package are
that MaskedArray is now a subclass of *ndarray* and that the
*_data* section can now be any subclass of *ndarray*. Apart from a
couple of issues listed below, the behavior of the new MaskedArray
class reproduces the old one. Initially the *maskedarray*
implementation was marginally slower than *numpy.ma* in some areas,
but work is underway to speed it up; the expectation is that it can be
made substantially faster than the present *numpy.ma*.


Note that if the subclass has some special methods and
attributes, they are not propagated to the masked version:
this would require a modification of the *__getattribute__*
method (first trying *ndarray.__getattribute__*, then trying
*self._data.__getattribute__* if an exception is raised in the first
place), which really slows things down.

Main differences
----------------

 * The *_data* part of the masked array can be any subclass of ndarray (but not recarray, cf below).
 * *fill_value* is now a property, not a function.
 * in the majority of cases, the mask is forced to *nomask* when no value is actually masked. A notable exception is when a masked array (with no masked values) has just been unpickled.
 * I got rid of the *share_mask* flag, I never understood its purpose.
 * *put*, *putmask* and *take* now mimic the ndarray methods, to avoid unpleasant surprises. Moreover, *put* and *putmask* both update the mask when needed.  * if *a* is a masked array, *bool(a)* raises a *ValueError*, as it does with ndarrays.
 * in the same way, the comparison of two masked arrays is a masked array, not a boolean
 * *filled(a)* returns an array of the same subclass as *a._data*, and no test is performed on whether it is contiguous or not.
 * the mask is always printed, even if it's *nomask*, which makes things easy (for me at least) to remember that a masked array is used.
 * *cumsum* works as if the *_data* array was filled with 0. The mask is preserved, but not updated.
 * *cumprod* works as if the *_data* array was filled with 1. The mask is preserved, but not updated.

New features
------------

This list is non-exhaustive...

 * the *mr_* function mimics *r_* for masked arrays.
 * the *anom* method returns the anomalies (deviations from the average)

Using the new package with numpy.core.ma
----------------------------------------

I tried to make sure that the new package can understand old masked
arrays. Unfortunately, there's no upward compatibility.

For example:

>>> import numpy.core.ma as old_ma
>>> import maskedarray as new_ma
>>> x = old_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> x
array(data =
 [     1      2 999999      4      5],
      mask =
 [False False True False False],
      fill_value=999999)
>>> y = new_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> y
array(data = [1 2 -- 4 5],
      mask = [False False True False False],
      fill_value=999999)
>>> x==y
array(data =
 [True True True True True],
      mask =
 [False False True False False],
      fill_value=?)
>>> old_ma.getmask(x) == new_ma.getmask(x)
array([True, True, True, True, True])
>>> old_ma.getmask(y) == new_ma.getmask(y)
array([True, True, False, True, True])
>>> old_ma.getmask(y)
False


Using maskedarray with matplotlib
---------------------------------

Starting with matplotlib 0.91.2, the masked array importing will work with
the maskedarray branch) as well as with earlier versions.

By default matplotlib still uses numpy.ma, but there is an rcParams setting
that you can use to select maskedarray instead.  In the matplotlibrc file
you will find::

  #maskedarray : False       # True to use external maskedarray module
                             # instead of numpy.ma; this is a temporary #
                             setting for testing maskedarray.


Uncomment and set to True to select maskedarray everywhere.
Alternatively, you can test a script with maskedarray by using a
command-line option, e.g.::

  python simple_plot.py --maskedarray


Masked records
--------------

Like *numpy.core.ma*, the *ndarray*-based implementation
of MaskedArray is limited when working with records: you can
mask any record of the array, but not a field in a record. If you
need this feature, you may want to give the *mrecords* package
a try (available in the *maskedarray* directory in the scipy
sandbox). This module defines a new class, *MaskedRecord*. An
instance of this class accepts a *recarray* as data, and uses two
masks: the *fieldmask* has as many entries as records in the array,
each entry with the same fields as a record, but of boolean types:
they indicate whether the field is masked or not; a record entry
is flagged as masked in the *mask* array if all the fields are
masked. A few examples in the file should give you an idea of what
can be done. Note that *mrecords* is still experimental...

Optimizing maskedarray
----------------------

Should masked arrays be filled before processing or not?
--------------------------------------------------------

In the current implementation, most operations on masked arrays involve
the following steps:

 * the input arrays are filled
 * the operation is performed on the filled arrays
 * the mask is set for the results, from the combination of the input masks and the mask corresponding to the domain of the operation.

For example, consider the division of two masked arrays::

  import numpy
  import maskedarray as ma
  x = ma.array([1,2,3,4],mask=[1,0,0,0], dtype=numpy.float_)
  y = ma.array([-1,0,1,2], mask=[0,0,0,1], dtype=numpy.float_)

The division of x by y is then computed as::

  d1 = x.filled(0) # d1 = array([0., 2., 3., 4.])
  d2 = y.filled(1) # array([-1.,  0.,  1.,  1.])
  m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
  array([True,False,False,True])
  dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
  result = (d1/d2).view(MaskedArray) # masked_array([-0. inf, 3., 4.])
  result._mask = logical_or(m, dm)

Note that a division by zero takes place. To avoid it, we can consider
to fill the input arrays, taking the domain mask into account, so that::

  d1 = x._data.copy() # d1 = array([1., 2., 3., 4.])
  d2 = y._data.copy() # array([-1.,  0.,  1.,  2.])
  dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
  numpy.putmask(d2, dm, 1) # d2 = array([-1.,  1.,  1.,  2.])
  m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
  array([True,False,False,True])
  result = (d1/d2).view(MaskedArray) # masked_array([-1. 0., 3., 2.])
  result._mask = logical_or(m, dm)

Note that the *.copy()* is required to avoid updating the inputs with
*putmask*.  The *.filled()* method also involves a *.copy()*.

A third possibility consists in avoid filling the arrays::

  d1 = x._data # d1 = array([1., 2., 3., 4.])
  d2 = y._data # array([-1.,  0.,  1.,  2.])
  dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
  m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
  array([True,False,False,True])
  result = (d1/d2).view(MaskedArray) # masked_array([-1. inf, 3., 2.])
  result._mask = logical_or(m, dm)

Note that here again the division by zero takes place.

A quick benchmark gives the following results:

 * *numpy.ma.divide*  : 2.69 ms per loop
 * classical division     : 2.21 ms per loop
 * division w/ prefilling : 2.34 ms per loop
 * division w/o filling   : 1.55 ms per loop

So, is it worth filling the arrays beforehand ? Yes, if we are interested
in avoiding floating-point exceptions that may fill the result with infs
and nans. No, if we are only interested into speed...


Thanks
------

I'd like to thank Paul Dubois, Travis Oliphant and Sasha for the
original masked array package: without you, I would never have started
that (it might be argued that I shouldn't have anyway, but that's
another story...).  I also wish to extend these thanks to Reggie Dugard
and Eric Firing for their suggestions and numerous improvements.


Revision notes
--------------

  * 08/25/2007 : Creation of this page
  * 01/23/2007 : The package has been moved to the SciPy sandbox, and is regularly updated: please check out your SVN version!