Evaluating Chemical Foundation Models
for Metabolite Property Prediction
A research note on the evaluation protocol we sent to Briefings in Bioinformatics: which chemical foundation models were tested, what tasks they were tested on, and what surprised us.
Findings at a glance
- The current generation of chemical foundation models (ChemBERTa-v2, Uni-Mol, MolFormer, our in-house pre-trained encoder) all cluster within ~1pp MAE on five standard ADMET tasks. None of them is strictly best.
- Scaling pretraining corpus size past ~50M molecules gives diminishing returns on the ADMET tasks we care about.
- What does move the needle: tokenization choices, scaffold-balanced fine-tuning splits, and adding conformer featurization.
- The headline result the field is missing: nobody has a clean benchmark for cross-instrument CCS + ADMET + reaction-condition prediction jointly.
Five chemical foundation models, five tasks,
a lot of variance in protocol. Reproducing published numbers turned
out to be the hardest part.
Why this paper is even necessary
Since the ChemBERTa paper in 2020, the chemical foundation-model space has grown explosively. By early 2026 the standard comparison table in any chemistry-AI paper mentions ten models. Each has its own pretraining corpus, tokenization, and very specific "beats baseline by 0.7pp on Tox21" claim.
Most published evaluations share two weaknesses:
- They don't separate the encoder from the head. A 110M-parameter encoder with a strong head will beat a 300M encoder with a weak head on most reported tables.
- They don't measure how cross-domain the encoder is. The benchmarks are mostly ADMET, which is promiscuous — most molecules interact with most assays.
Our goal for this paper was to remove both effects: fix the head, vary the encoder; and use out-of-domain scaffold splits to expose the cross-domain signal.
Protocol
We selected five foundation models that, between them, span the main design choices in this space:
Models compared. Heights = approximate parameter count (not log), to keep relative sizing legible. The "In-house SSP" is the masked-SMILES Transformer from #3 in this series.
Five downstream tasks, all metabolite-relevant and all scaffold-split to enforce out-of-distribution evaluation:
- LogP (octanol-water partition coefficient), from MoleculeNet.
- Solubility in water (LOG-S), MoleculeNet.
- Tox21 (NR-AhR), toxicity against the aryl hydrocarbon receptor.
- AMR (antimicrobial resistance activity), our in-house dataset of ~12k MIC values.
- CCS, from #2 in this series, held out by chemistry class.
For each (model, task) pair we trained a frozen-encoder + 2-layer MLP head with the same hyperparameters, the same optimizer, and 5 random seeds. Same head across all encoders is the bit we felt most previous evaluations had skipped.
class FrozenHeadProtocol:
def fit(self, encoder, X_train, y_train, X_val, y_val):
# 1. Encode all molecules once
Z_train = encode(encoder, X_train)
Z_val = encode(encoder, X_val)
# 2. Train head (2-layer MLP, fixed hyperparams)
head = HeadRegressor(input_dim=Z_train.shape[1])
head.fit(Z_train, y_train, Z_val, y_val, max_epochs=200)What we saw
The headline results, in percentile relative-error terms (so higher = worse, lower = better):
Task | RDKit+XGB | ChemBERTa-v2 | Uni-Mol | MolFormer | In-house SSP ------------+-----------+--------------+---------+-----------+-------------- LogP | 0.34 | 0.31 | 0.30 | 0.31 | 0.30 Solubility | 0.71 | 0.62 | 0.58 | 0.63 | 0.59 Tox21 (NR) | 0.27 | 0.24 | 0.23 | 0.22 | 0.24 AMR (MIC) | 1.42 | 1.13 | 1.04 | 1.10 | 1.05 CCS | 3.42 | 3.28 | 3.22 | 3.30 | 3.20
Take it in pieces.
Nothing here is state-of-the-art on its own. The interesting thing is that all five models are essentially within measurement noise of each other on every task. — paraphrasing a reviewer comment that we ended up arguing against
Yes, this is what the field already suspects; it's what the paper actually verifies. The reviewer comment was right that the picture is boring, but the picture is also missing.
Three surprises
Surprise #1: tokenization matters more than corpus size. Replacing SMILES with SELFIES or with a substructure tokenizer re-orders the leaderboard without changing the overall spread. Each tokenization scheme carves the same chemistry differently; what you win in one task you lose in another.
Surprise #2: scaffold-balanced splits hurt everyone equally. Previous evaluations that report wildly different scores between models weren't using scaffold splits. When we add them, all five models converge in performance — because they're no longer being asked the same question. Suggests current "leaderboards" reward some implicit data-leakage more than others.
Surprise #3: AMR is the hardest task, by a wide margin. CCS, solubility, LogP — all of those are basically memorized at this point. Antimicrobial resistance is structurally heterogeneous, biologically noisy and adversarial to representation learning. We suspect this is the next frontier for chemical foundation models; nobody has a clean answer yet.
What did not survive peer review
The first version of the paper made a stronger claim — that domain-specific pretraining beats scale-up. Reviewer 2's "show me the full Table 7" burn convinced us to soften it; we now report it as a hypothesis with a citation. We keep the ablation in the supplementary material.
Lesson. If your evaluation protocol decides a winner, check that the difference survives a stricter split. If it doesn't, you probably don't have a winner — you have a benchmark artifact.
Code and data
All training scripts, hyperparameter logs and pickle files for the encoders are available at github.com/grfone/chem_fm_evaluation. We make an effort to keep the codebase useful even after the pretrained checkpoints inevitably drift; the package is pinned on a known-good Transformers and Datasets version pair.
The AMR MIC dataset is a subset. If your lab has MIC measurements for metabolites and you want them added to a future version, the contact page is the right place.