Clarence Lee

Hello, I’m Clarence. I am deeply interested in exploring the frontiers of Large Language Models and making them more generalisable and adaptable by encoding implicit structures in the data. Current state of the art methods use reinforcement learning algorithms and curated environments to generate synthetic data, yet this process is expensive and not sample efficient.

Scaling data will remain important in building capable language models in the coming years. Yet, how to scale this remains an open question, particularly as methods of distillation is on the rise.

I have been involved in pretraining Large Language Models for over 2 years. I worked specifically on developing the pretraining infrastructure to train 8Billion and 70Billion parameter models. I have explored various axes in pretraining, particularly scaling laws, model expansion and maximal update parameterisation.

Beyond pretraining, I also lead efforts in my team to build trillion token datasets for multilingual corpora, including deduplication, data mixing, topic modeling and quality filtering. Recently, I developed a work on diverse pretraining data selection by scaling the G-vendi score (The shannon entropy of the gradient kernel similarity matrix) to pretraining scale corpora.

I am excited about building more sample efficient and capable language models that can serve the community.

news

Aug 22, 2026	I am starting a Visiting Research position at the University of Washington.
Jul 02, 2026	I attended ACL 2026!
Dec 02, 2025	I attended NeurIPs 2025!
Dec 10, 2024	I attended Neurips 2024!
Aug 02, 2024	I was awarded the Singapore National Academy of Science Award, a national award recognizing students for excelling in their respective scientific disciplines!

selected publications

ICCV

DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determiners

Clarence Lee, M Ganesh Kumar, and Cheston Tan

In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

Abs PDF Video

State-of-the-art visual grounding models can achieve high detection accuracy, but they are not designed to distinguish between all objects versus only certain objects of interest. In natural language, in order to specify a particular object or set of objects of interest, humans use determiners such as "my", "either" and "those". Determiners, as an important word class, are a type of schema in natural language about the reference or quantity of the noun. Existing grounded referencing datasets place much less emphasis on determiners, compared to other word classes such as nouns, verbs and adjectives. This makes it difficult to develop models that understand the full variety and complexity of object referencing. Thus, we have developed and released the DetermiNet dataset, which comprises 250,000 synthetically generated images and captions based on 25 determiners. The task is to predict bounding boxes to identify objects of interest, constrained by the semantics of the given determiner. We find that current state-of-the-art visual grounding models do not perform well on the dataset, highlighting the limitations of existing models on reference and quantification tasks.
arxiv

Evaluating the Generation of Spatial Relations in Text and Image Generative Models

Shang Hong Sim, Clarence Lee, Alvin Tan, and 1 more author

arXiv preprint arXiv:2411.07664, 2024

Abs PDF

Understanding spatial relations is a crucial cognitive ability for both humans and AI. While current research has predominantly focused on the benchmarking of text-to-image (T2I) models, we propose a more comprehensive evaluation that includes \textitboth T2I and Large Language Models (LLMs). As spatial relations are naturally understood in a visuo-spatial manner, we develop an approach to convert LLM outputs into an image, thereby allowing us to evaluate both T2I models and LLMs \textitvisually. We examined the spatial relation understanding of 8 prominent generative models (3 T2I models and 5 LLMs) on a set of 10 common prepositions, as well as assess the feasibility of automatic evaluation methods. Surprisingly, we found that T2I models only achieve subpar performance despite their impressive general image-generation abilities. Even more surprisingly, our results show that LLMs are significantly more accurate than T2I models in generating spatial relations, despite being primarily trained on textual data. We examined reasons for model failures and highlight gaps that can be filled to enable more spatially faithful generations.
arxiv

Spokes: Optimizing for Diverse Pretraining Data Selection

Clarence Lee, Yejin Choi, Luke Zettlemoyer, and 2 more authors

arXiv preprint arXiv:2606.15216, 2026

Abs PDF

Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.