CaltechAUTHORS
Published October 18, 2018 | Version Published + Submitted
Journal Article Open

Deep learning for inferring cause of data anomalies

Creators

  • 1. ROR icon Massachusetts Institute of Technology
  • 2. ROR icon National Research University Higher School of Economics
  • 3. ROR icon Yandex School of Data Analysis
  • 4. ROR icon European Organization for Nuclear Research
  • 5. ROR icon Texas Tech University
  • 6. ROR icon Skolkovo Institute of Science and Technology
  • 7. ROR icon University of Paris-Saclay
  • 8. ROR icon California Institute of Technology

Abstract

Daily operation of a large-scale experiment is a resource consuming task, particularly from perspectives of routine data quality monitoring. Typically, data comes from different sub-detectors and the global quality of data depends on the combinatorial performance of each of them. In this paper, the problem of identifying channels in which anomalies occurred is considered. We introduce a generic deep learning model and prove that, under reasonable assumptions, the model learns to identify 'channels' which are affected by an anomaly. Such model could be used for data quality manager cross-check and assistance and identifying good channels in anomalous data samples. The main novelty of the method is that the model does not require ground truth labels for each channel, only global flag is used. This effectively distinguishes the model from classical classification methods. Being applied to CMS data collected in the year 2010, this approach proves its ability to decompose anomaly by separate channels.

Additional Information

© 2018 Published under licence by IOP Publishing Ltd. Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. The research leading to these results was partly supported by Russian Science Foundation under grant agreement No 17-72-20127.

Attached Files

Published - Azzolini_2018_J._Phys.__Conf._Ser._1085_042015.pdf

Submitted - 1711.07051.pdf

Files

1711.07051.pdf

Files (1.6 MB)

Name Size
md5:e826a131422ce7d9e4b0910f47811f25
1.0 MB Preview Download
md5:086c787336c6ca3dfc21ffc9c0de32b9
524.4 kB Preview Download

Additional details

Identifiers

Eprint ID
96190
Resolver ID
CaltechAUTHORS:20190606-101559273

Related works

Describes
https://arxiv.org/abs/1711.07051 (URL)

Funding

Russian Science Foundation
17-72-20127

Dates

Created
2019-06-06
Created from EPrint's datestamp field
Updated
2022-07-12
Created from EPrint's last_modified field