Transformers are advanced Neural Network architectures useful in Data Analysis, not only for LLMs.
I recently published a paper that presents a method based on a Transformer Autoencoder architecture for analyzing sparse and irregular time series. The autoencoder utilizes a local attention mechanism to identify similarities and anomalies within the complex structure of irregular datasets. It demonstrates notable clustering capabilities for samples that share similar characteristics.
The method is useful in risk estimation. The application presented in this work focuses on detecting accounts at higher risk of non-technical losses in electrical power systems, mainly electricity theft, using real-world consumption data from Greece.
The proposed framework combines classical data cleaning procedures with a state-of-the-art transformer architecture, exploiting the local attention mechanism to capture meaningful short-range temporal patterns without requiring interpolation of missing data.
The paper is available here. Also a read only version and a preprint.
