Solving large‐data consistency problems at Statistics Netherlands using macro‐integration techniques

The generalized multivariate Denton model for achieving consistency between large accounting frameworks developed at Statistics Netherlands (SN) is originally intended for benchmarking of supply and use (SU) tables of national accounts. The success of this application in the production process triggered using the model for other processes within the office. Currently, at SN, many production processes of national accounts, but also other departments, use the modifications of this optimization model for achieving consistency of data obtained from different sources. These include reconciliation and benchmarking of SU tables and of institutional sector accounts, ESA (European system of accounts), revisions of SU tables, benchmarking of gross fixed capital formation, Population Census tables, and energy statistics figures. The mathematical model is based on a quadratic optimization function and combines different features, such as linear constraints, ratio constraints, weights, soft and hard constraints, and inequalities. The optimization problems we deal with can be very large, consisting of 500,000 variables and over 100,000 constraints. This optimization problem is solved using the commercially available software tool XPRESS and the free software tool R. For the reconciliation of trade and transport statistics, similar optimization techniques are used. In this paper, we give an overview of production processes at SN using macro‐integration techniques.