Key Indicator Prediction

© Haodong Li

• Overview
• Data Analysis & Pre-process
• LSTM-based Model Design
• Transformer-based Model Design
• CL-based Model Design
• Experiments

Overview

• In modern industrial processes, it is necessary to monitor the quality variables and critical variables of the production process to improve product quality and yield, maintain the safety of the production process, save energy, etc.
• In this project, we collaborated with a company to apply cutting-edge AI techniques to predict key indicators by using the data provided by them on changes in the content of several chemical elements in iron in the blast furnace ironmaking production process; changes in production environment factors such as CO concentration, air supply temperature, etc.
• We first analyzed the overall data and designed the corresponding data pre-processing method; the second step was to build a deep learning framework and build 4 benchmark models based on long and short term memory network (LSTM) for experiments; the third step was to build a model based on Transformer architecture for application to time series index prediction and conduct experiments; the fourth step was to build a model for continual learning temporal data prediction based on Multi-Head Self-Attention, 1-dimensional Residual Neural Network (ResNet), LSTM, and Multilayer Perceptron (MLP), with reference to the continual learning (CL) theory that mimics the hippocampus and neocortex of the human brain, and conduct experiments.
• Finally, this project recorded and summarized the experimental results. For 6-dimensional key variable prediction, the 4 LSTM-based models built in this project can achieve an \(R^2\) accuracy of 0.9570 with an RMSE loss of 0.03959; the Transformer-based model has an effect of 0.9524 / 0.05201; the CL-based model results in 0.9474 / 0.05312. But in multi-step forecasting, when the time step is greater than 2, the \(R^2\) accuracy of the CL-based model is the highest, and it reaches 0.9146 when the time step is 3; when the time step is greater than 8, the RMSE loss of the CL-based model The lowest, which reaches 0.06789 at a step size of 9. In 1-dimensional quality variable prediction, the current best method (SoTA) results in 0.9334 / 0.03596; the best LSTM-based model built in this project results in 0.9024 / 0.03928; the Transformer- based model results in 0.9902 / 0.00923, outperforming SoTA by about 5.68%; CL- based model results in 0.9352 / 0.03656, which is almost equal to SoTA.
• Code is available at: https://github.com/lebronlihd/key-indicator_prediction.
• Keywords: Blast furnace ironmaking; Temporal data prediction; Continual Learning; Attention Mechanism; Transformer; LSTM; Residual Neural Network

Data Analysis & Pre-process

• The dataset we used covers 29070 time steps of 115 dimensions, among them 6 are key indicators and the rest 109 are auxiliary indicators.

Hot metal Si (01)	Hot metal S (53)	Hot metal Mn (54)
Hot metal P (55)	Hot metal C (56)	Hot metal Ti (57)

• The first row represents the data characteristics & correlation distribution among variables .
  • The upper part of the graph is the distribution line of the data, the yellow is the original data, and the red is the data after mean smoothing;
  • The lower half of the graph represents the distribution of correlation coefficients between this indicator and all 115 indicators.
• Images on the second row are the frequency domain distribution of 6 key variables.
• The data pre-processing method we applied is: Max-Min Normalization & numpy.nan_to_num.

LSTM-based Model Design

• This project designed 4 variants based on LSTM, which are called Simple_LSTM, CNN_LSTM, ResNet_LSTM, and EfficientNetV2_LSTM, details as follows.

Simple_LSTM	ResNet_LSTM
CNN_LSTM	EfficientNetV2_LSTM: tf.keras.applications.EfficientNetV2S + LSTM

Transformer-based Model Design

• Transformer is a major innovation in the field of deep learning in recent years. Although it was born for Neural Machine Translation (NMT), we believe that Transformer based on Multi-Head Self-Attention is very likely to be competent for key variables in the blast furnace ironmaking scene. problem of prediction. This chapter will introduce the Transformer-based model designed in this project.
• As shown in figures below, the Transformer-based model built in this project adopts the form of Sequence-to-Sequence to design DataLoader, that is, the input data is shifted to the right and the prediction step is used as the corresponding target output. The sequence length is set to 50, which is consistent with 4 LSTM-based models and the CL-based model described below. But the difference is that the Transformer-based model contains two input channels,
  • one of which (Channel A) passes through Positional Encoding and Transformer Encoder, the data shape changes from 50×6 to 512×6, and enters as Key and Value Multi-Head Self-Attention of the first layer of Transformer Decoder;
  • another channel (Channel B) after Positional Encoding, the data shape changes to 512×6, and enters Multi-Head Self-Attention of the first layer of Transformer Decoder as a query. After the calculation of Transformer Decoder, the output data of 50×6 is obtained.

Encoder (or Channel A)	Decoder (or Channel B + Decoder)

CL-based Model Design

• The human brain can realize learning behavior through the two learning systems of “hippocampus” and “neocortex”. Among them, the hippocampus focuses on rapid learning; and through memory consolidation, memories in the hippocampus are transferred to the neocortex over time to form long-term memories. However, it should be noted that Continual Learning (CL) in the current academic field is mainly aimed at research in Domain Adaptation (DA) and Transfer Learning (TL) across datasets in the field of Compter Vision (CV), especially based on image classification such as CIFAR100. The nature of the data used in this project is special, and there are no good conditions for model transferability research. Therefore, the CL-based model introduced in this chapter is still a time-series data prediction model, but the model is inspired by CL. The main point is: the two branches of FastNet and SlowNet, study NewData and Memory respectively.

• Technically speaking, the CL-based model designed in this project mainly applies Multi-Head Self-Attention, 1-dim ResNet, LSTM and MLP. As shown in figures above and after, the sequence length of New Data in this CL-based model is set to 50; the sequence length of Memory is set to 500, and New Data will supplement Memory. Macroscopically, the CL-based model mainly consists of two branches, FastNet and SlowNet, each branch contains two sub-models, namely FastNet_1, FastNet_2; and SlowNet_1, SlowNet_2, and finally MLP_End outputs the prediction results of 6 key variables.

FastNet_1	FastNet_2 & SlowNet_2
SlowNet_1	MLP_End

Experiments

Accuracy trend	Loss trend

• Here are the training log & prediction result visualization (take EfficientNetV2_LSTM with 6 key indicators scenario with time_step = 0 for example)

Hot metal Si (01)	Hot metal S (53)	Hot metal Mn (54)
Hot metal P (55)	Hot metal C (56)	Hot metal Ti (57)

• This project aims at the real-time prediction needs of key variables in the blast furnace ironmaking production scenario, and designed several algorithms based on deep learning methods, namely Simple_LSTM, CNN_LSTM, ResNet_LSTM, EfficientNetV2_LSTM, Transformer-based Model, and CL-based Model. The results show that these deep learning models designed in this project preformed well in the prediction of key variables in blast furnace ironmaking.