How to Handle Missing Values?

The best practice is a combination of being declarative about your invariants and using automated data observability to handle scale.

High-importance, load-bearing data assets need to be treated with priority (you'll hear phrases like "certified gold data," Airbnb has their "Midas Certified" standard, etc.). For these datasets, employ code-based checks such as:

On the scalability point, we actually invest a lot into our Monte Carlo Monitors-as-Code setup, which configures monitors for things like the above. dbt tests can also provide this sort of coverage. We benefit from a mix of monitoring where critical assets, like the feature table immediately serving an ML model, have detailed testing and validation on every important field, while the rest of the pipeline upstream of this has "lighter" monitoring like freshness and volume metadata monitoring that MC has out-of-the-box.

Ryan Kearns, a Founding Data Scientist at Monte Carlo

Missing data often carries inherent signals or patterns within the dataset. Every instance of missing data is typically indicative of a specific underlying cause. Therefore, to effectively mitigate these biases, it is imperative to gain a comprehensive understanding of the data's origin, generation process, and the mechanisms by which it was captured within the system.

Abhishek Pawar, a Senior Data Scientist at Precisely

For example, if one of your APIs is misconfigured and sending junk data, you might end up serving irrelevant ads to customers using the browser where that API is supposed to be working.

You can also make misinformed decisions on the business intelligence side, if data fails to reach a downstream Tableau asset, you might end up convinced that your sales in some segments have tanked when in reality the volume just isn't getting through the pipeline.

Ryan Kearns

Since the introduction of fine-tuning by transfer learning, missing data can be the root cause of significant biases and inequalities in core models like BERT.

We’ve seen this during the introduction of new models being firstly and sometimes solely available within their domain, language, or cultural space, and, despite their more general nature, we can see this ever so clearly with the rise of LLMs.

David Berenstein, a Developer Advocate at Argilla

This is an important question that requires domain knowledge because each approach introduces different kinds of bias.

For example, with a time series where data is missing from 08:00 to 12:00, the decision to delete, impute, or backfill data categorically depends on what the data represents and what you want your model to be doing.

For time series anomaly detection like Holt-Winters, you need data either imputed or backfilled because the model can't fit any seasonal predictions if periods are missing. If instead, you're treating the time axis like a feature and not a time series, you can probably do with deleting the missing data or not include it.

Also, deciding between imputation and backfilling should be context-sensitive – if the data in question represents a meter reading, linear interpolation or median imputation might work, but if it represents page views per hour instead you need to consider whether filling with zeros makes more sense. Even linear interpolation vs. median vs. mean imputation each introduces different biases to the model and statistical details here can get quite complicated.

Ryan Kearns

Besides the general push towards creating core models for lower resource languages and domains, we have also seen solutions that impute knowledge based on:

• Smart heuristics like ensemble model workflows,

• Cross-lingual model capabilities,

• and (even) more general models.

An example of an ensemble model workflow that I have worked with during my graduation thesis is the usage of paraphrasing methods to diversify data. There are several direct paraphrase models available but another great approach for doing this is back-translation.

Here n-hop translations between different languages cause the text to slightly deform. Using Google Translate, we can even do this with a span-preserving setting, which might be useful for span-based tasks like entity extraction. We can even apply a 1-hop translation to use lower-resource-language data for an English transformer model.

Due to lexical and semantic overlap in languages, it has proven to be possible to train models with cross-lingual properties. These models can be used to make predictions on multiple languages, hence, fine-tuning a model on language A will ensure it is possible to make decent predictions for language B as well. This approach cannot easily be benchmarked but it does offer a way to start higher quality data, which in turn can be curated for training a language-specific model.

Lastly, we’ve come to the section of (even) more general models, like LLMs.

Simply said, this is achieved with more data and parameters.

Everyone reading this article has heard stories about the vast volume of data and amount of parameters of the recent OpenAI models, but researchers behind the previously mentioned cross-lingual models and even “Attention is All You Need” found that this direction was a viable solution to significantly improve performance.

David Berenstein

I've refrained from imputing any data that relies on domain-specific expertise, especially in fields like healthcare. In such cases, it's always advisable to seek guidance and consultation from Subject Matter Experts (SMEs) prior to undertaking any imputation processes.

Abhishek Pawar

Synthetic data is an incredibly powerful tool. 

Famously, DeepMind used a hybrid training approach with AlphaZero using lots of synthetic data, so there is precedent to this being done correctly. But you need to be careful (we're not all DeepMind) especially if the real data generating process you're trying to emulate is complicated.

Again, this question is very contextual since certain model architectures will respond to synthetic data in different ways. If your architecture is interrogable, like a gradient-boosted decision tree, you can probably get away with more synthetic data since your model engineers can identify gaps in the representation and iterate on better synthetic data generation in response.

Ryan Kearns