Ask the head of R&D at almost any major cosmetics company whether artificial intelligence has a role to play in formulation, and you’ll get an enthusiastic “yes”. Algorithms can explore ingredient combinations at a speed no human team can match. Predictive models promise to compress development timelines. It’s little wonder the industry is convinced.
And yet, in laboratory after laboratory, those projects are stalling – not because the models are wrong, but because the data they need does not exist in a usable form. This is the hidden bottleneck in cosmetics R&D, and the organizations pulling ahead are not necessarily those with the most sophisticated algorithms. They’re the ones that made their experimental data findable, structured, and learnable.
The Dark Data Problem
In many mature cosmetics R&D organizations, data from years of formulation work are stored haphazardly. Experimental results live in spreadsheets on individual scientists’ hard drives. Formulation stability observations are in shared folders with no consistent naming conventions. Sensory panel data sits in a different system from the rheology measurements taken on the same batch. The scientist who formulated the moisturizer that anchors a product line retired three years ago, taking most of what she knew with her. This fragmentation is not carelessness. It is the accumulated result of decades of documentation practices that prioritized individual productivity over collective reuse. Scientists recorded what they needed to get the next experiment done, not what a machine learning model, or even a new colleague, could later interpret reliably.
The consequence is what data scientists call “dark data”: information generated at real cost that cannot be accessed or acted on systematically. A mid-size personal care company with a 100-person formulation team will typically generate thousands of experiments per year. Most of those results, and the context that gives them meaning, are effectively invisible.
What Good Data Looks Like
Recent work on Bayesian approaches to combinatorial formulation has demonstrated genuine progress. But those machine learning (ML) approaches assume a starting point: clean, structured historical data from which a model can learn relationships between inputs (ingredients, concentrations, process parameters) and outputs (stability, sensory score, viscosity, efficacy). Without this starting point, even the most elegant algorithm has nothing to train on. ML-ready data in a cosmetics context has four requirements that are rarely met simultaneously:
1. Completeness
Every experiment needs a record of what went in and what came out – all results, not just the ones that looked promising. Scientists naturally focus on what advances the project, but a predictive model needs the failures, too.
2. Consistency
The same attribute needs to be recorded the same way across experiments, scientists, and sites. If one scientist records pH to two decimal places and another rounds to one, or if sites use different viscometry methods, the data cannot be meaningfully compared. Models trained on inconsistent data learn noise, not signals.
3. Context
Raw numbers without experimental context are nearly useless for learning. A stability result means something different at 40 °C/75% RH than at ambient conditions. A sensory score means something different depending on the panel protocol and the benchmark it was measured against. Context is often the first thing dropped under time pressure.
4. Linkage
Formulation development connects ingredient evaluation, lab-scale prototyping, scale-up trials, stability testing, QC release, and regulatory documentation. When those connections are broken – for example, when the QC record cannot be traced back to the lab formulation, or a scale-up deviation has no link to the sensory outcome – a large amount of scientific intelligence is permanently lost.
Old Habits Block new Capabilities
The shift to ML-ready data is not primarily a technology problem. It’s a behavioral change problem, and behavioral change is difficult. For decades, the incentive in cosmetics R&D has been to move quickly: get to the next prototype, consumer test, or launch. Documentation was a compliance obligation, not a scientific asset. In many organizations, thorough record-keeping was viewed as a sign of inefficiency: “time in the lab, not time in the notebook.” There’s also the problem of tacit knowledge. Experienced formulators carry enormous amounts of understanding that was never written down because it never needed to be. They know from the feel of the batch whether the emulsification went well. They know which supplier’s version of a particular emollient behaves differently from the catalogue specification. They know that the stability chamber on the third floor runs two degrees hot. None of this is in any database, and none of it will ever appear in a model unless it is captured deliberately.
When those formulators leave through retirement, turnover, or restructuring, that knowledge leaves with them. The industry has spent years worrying about knowledge transfer as a human problem, but it is also, increasingly, an AI readiness problem.
How ready is an Organization’s R&D Data?
Four diagnostic questions tend to surface the most revealing gaps.
The first concerns knowledge retention. If a scientist left tomorrow, how much of their experimental knowledge could a colleague recover? If the honest answer is “very little,” the organization has a dark data problem regardless of its technology stack.
The second concerns consistency. Are formulations recorded in a way that allows direct comparison across sites – shared ingredient names, concentrations, test protocols, result formats? Inconsistency is one of the most common barriers to cross-site predictive models.
The third concerns traceability. Can a product in the current portfolio be traced back to every formulation decision made during development, including the variants that were not chosen? If not, the institutional memory of that history is inaccessible.
The fourth concerns the R&D-to-QC handoff. When a batch deviation is identified, can the full experimental history of that formulation be retrieved in minutes? If the answer requires a phone call or a search through archived files, the data linkage between development and quality is broken.
Together, these questions expose gaps in structure, consistency, linkage, and retrieval that separate a data environment capable of supporting AI from one that is not.
How one Global Organization rebuilt their Data Foundation
The experience of Clariant, the Swiss specialty chemicals company whose active ingredients appear in formulations across the personal care industry, illustrates what this transition looks like in practice.
In 2023, Clariant set out to replace a legacy electronic laboratory notebook (ELN) that had become a source of genuine frustration. Requested changes took not weeks but years to implement, and workarounds had accumulated across the organization. The data existed, but it was not structured, searchable, or connected across teams.
Clariant deployed Uncountable’s integrated ELN and laboratory information management system (LIMS), connecting synthesis and application teams within a single structured environment. More than 800 scientists were onboarded across global sites in under ten months, reaching over 1,000 users across 35 facilities by the end of 2024. A built-in LIMS interface allows lab users to send test requests and receive analytical results directly within their experiment record.
The result was not just a faster notebook. It was a different quality of data: structured by design, consistent across sites, linked across the development workflow. This is the kind of data a machine learning model can actually learn from.
“AI can make us more productive,” said Cesar Chocou, Head of IT at Clariant. “But the real difference is when we can use our own data to accelerate our proprietary know-how.”
Data readiness requires executive sponsorship, scientist buy-in, and a willingness to redesign documentation workflows before predictive capability can be built on top. The technology follows the foundation; it does not replace it.
The Competitive Divide is Opening
Recent advances in ingredient optimization have demonstrated that AI can genuinely accelerate the early stages of cosmetics formulation. The question for most organizations is not whether to adopt these capabilities, but whether they are positioned to benefit when they do.
The companies that will capture the most value from AI in R&D are not necessarily those that move first on algorithms. They are those organizations that recognized, early enough, that the quality of science they do tomorrow depends on the quality of data they capture today. The bottleneck in cosmetics R&D was never the model, but the data the model learns from. Building that foundation is work that cannot be deferred.
Noel Hollingsworth
Noel Hollingsworth, CEO and Co-Founder, Uncountable, San Francisco, CA, USA,
www.uncountable.com