Lexical Frequency Profiles and Zipf's Law

Laufer and Nation (1995) proposed that the Lexical Frequency Profile (LFP) can estimate the size of a second‐language writer's productive vocabulary. Meara (2005) questioned the sensitivity and the reliability of LFPs for estimating vocabulary sizes, based on the results obtained from probabilistic simulations of LFPs. However, the underlying mathematical model for the simulations, based on Zipf's law, allows such an analysis to be done directly, without recourse to simulations. The direct analysis has the further advantage of demonstrating how variability estimates obtained from within the 1k band (the 1,000 most frequent words of English) portion of written texts may explain the simulation results. The findings confirm that the ability of LFPs to distinguish between groups diminishes as vocabulary size increases. However, for fairly homogeneous groups, LFPs are able to provide a coarse but reasonable tool for vocabulary size estimation. We also explore modifications to Zipf's law that may result in a more accurate model of word frequencies in natural language.