Forecasting the weather is an increasingly data-intensive exercise. Numerical weather prediction (NWP) models are becoming more complex, with higher resolutions, and there are increasing numbers of different models in operation. Selleckchem CBR-470-1 While the forecasting skill of NWP models continues to improve, the number and complexity of these models poses a new challenge for the operational meteorologist how should the information from all available models, each with their own unique biases and limitations, be combined in order to provide stakeholders with well-calibrated probabilistic forecasts to use in decision making? In this paper, we use a road surface temperature example to demonstrate a three-stage framework that uses machine learning to bridge the gap between sets of separate forecasts from NWP models and the 'ideal' forecast for decision support probabilities of future weather outcomes. First, we use quantile regression forests to learn the error profile of each numerical model, and use these to apply empirically derived probability distributions to forecasts. Second, we combine these probabilistic forecasts using quantile averaging. Third, we interpolate between the aggregate quantiles in order to generate a full predictive distribution, which we demonstrate has properties suitable for decision support. Our results suggest that this approach provides an effective and operationally viable framework for the cohesive post-processing of weather forecasts across multiple models and lead times to produce a well-calibrated probabilistic output. This article is part of the theme issue 'Machine learning for weather and climate modelling'.Recent progress in machine learning (ML) inspires the idea of improving (or learning) earth system models directly from the observations. Earth sciences already use data assimilation (DA), which underpins decades of progress in weather forecasting. DA and ML have many similarities they are both inverse methods that can be united under a Bayesian (probabilistic) framework. ML could benefit from approaches used in DA, which has evolved to deal with real observations-these are uncertain, sparsely sampled, and only indirectly sensitive to the processes of interest. DA could also become more like ML and start learning improved models of the earth system, using parameter estimation, or by directly incorporating machine-learnable models. DA follows the Bayesian approach more exactly in terms of representing uncertainty, and in retaining existing physical knowledge, which helps to better constrain the learnt aspects of models. This article makes equivalences between DA and ML in the unifying framework of Bayesian networks. These help illustrate the equivalences between four-dimensional variational (4D-Var) DA and a recurrent neural network (RNN), for example. More broadly, Bayesian networks are graphical representations of the knowledge and processes embodied in earth system models, giving a framework for organising modelling components and knowledge, whether coming from physical equations or learnt from observations. Their full Bayesian solution is not computationally feasible but these networks can be solved with approximate methods already used in DA and ML, so they could provide a practical framework for the unification of the two. Development of all these approaches could address the grand challenge of making better use of observations to improve physical models of earth system processes. This article is part of the theme issue 'Machine learning for weather and climate modelling'.The radiative transfer equations are well known, but radiation parametrizations in atmospheric models are computationally expensive. A promising tool for accelerating parametrizations is the use of machine learning techniques. In this study, we develop a machine learning-based parametrization for the gaseous optical properties by training neural networks to emulate a modern radiation parametrization (RRTMGP). To minimize computa- tional costs, we reduce the range of atmospheric conditions for which the neural networks are applicable and use machine-specific optimized BLAS functions to accelerate matrix computations. To generate training data, we use a set of randomly perturbed atmospheric profiles and calculate optical properties using RRTMGP. Predicted optical properties are highly accurate and the resulting radiative fluxes have average errors within 0.5 W m-2 compared to RRTMGP. Our neural network-based gas optics parametrization is up to four times faster than RRTMGP, depending on the size of the neural networks. We further test the trade-off between speed and accuracy by training neural networks for the narrow range of atmospheric conditions of a single large-eddy simulation, so smaller and therefore faster networks can achieve a desired accuracy. We conclude that our machine learning-based parametrization can speed-up radiative transfer computations while retaining high accuracy. This article is part of the theme issue 'Machine learning for weather and climate modelling'.The advent of digital computing in the 1950s sparked a revolution in the science of weather and climate. Meteorology, long based on extrapolating patterns in space and time, gave way to computational methods in a decade of advances in numerical weather forecasting. Those same methods also gave rise to computational climate science, studying the behaviour of those same numerical equations over intervals much longer than weather events, and changes in external boundary conditions. Several subsequent decades of exponential growth in computational power have brought us to the present day, where models ever grow in resolution and complexity, capable of mastery of many small-scale phenomena with global repercussions, and ever more intricate feedbacks in the Earth system. The current juncture in computing, seven decades later, heralds an end to what is called Dennard scaling, the physics behind ever smaller computational units and ever faster arithmetic. This is prompting a fundamental change in our approach to the simulation of weather and climate, potentially as revolutionary as that wrought by John von Neumann in the 1950s. One approach could return us to an earlier era of pattern recognition and extrapolation, this time aided by computational power. Another approach could lead us to insights that continue to be expressed in mathematical equations. In either approach, or any synthesis of those, it is clearly no longer the steady march of the last few decades, continuing to add detail to ever more elaborate models. In this prospectus, we attempt to show the outlines of how this may unfold in the coming decades, a new harnessing of physical knowledge, computation and data. This article is part of the theme issue 'Machine learning for weather and climate modelling'.