Pelloran Inc.
Industry and edge

A machine by its sound

Can KELVIN hear that a machine is not running normally, when it has heard only the machine's normal running, and with no pre-training?

Benchmark

Dot chart, one row per machine, each with three marks: KELVIN, the autoencoder baseline and the MobileNetV2 baseline. On the fan, the gearbox and the bearing, KELVIN's mark lies to the right of both. On the slider, MobileNetV2's mark lies just to the right of KELVIN's; on the valve, far to the right.
Per machine, KELVIN's AUC (copper) beside the two official baselines' published AUC: fan 51.41 against the autoencoder's 41.16 and MobileNetV2's 50.34; gearbox 64.61 against 61.92 and 51.34; bearing 64.00 against 59.93 and 58.23; slider 61.35 against 58.95 and 62.42; valve 54.63 against 54.26 and 72.77. As registered beforehand, KELVIN had to be above both on at least four of the five machines; it is on three.

The question

Can KELVIN tell from its sound that a machine is not running normally, when it has heard only the machine's normal running, and with no pre-training?

What was done

KELVIN learned each machine from 100 normal recordings per section, with no pre-training, on a CPU and no graphics card, with other work running on the same computer. A first listen took a median 311 ms per recording, a finer one 1 268 ms, and a decision 293 to 523 ms.

The result

On the DCASE 2022 Challenge Task 2 evaluation set, five real machine types (fan, gearbox, bearing, slider, valve), KELVIN scores 56.50 by the challenge's own evaluator code. The official baselines published 52.94 (an autoencoder) and 54.02 (MobileNetV2), over seven machine types. The bar we registered before the run, above both baselines on at least 4 of 5 machines, was missed: 3 of 5.

Per machine, KELVIN's AUC is 51.41 on the fan, 64.61 on the gearbox and 64.00 on the bearing, above both official baselines; on the slider 61.35, where MobileNetV2 has 62.42, and on the valve 54.63, where MobileNetV2 has 72.77.

The limit

So the test, as registered before the run, is not established. The baselines' overall scores cover seven machine types, KELVIN's five.

The baselines' per-machine AUCs are as published on the challenge's results page, which does not state how they were aggregated; KELVIN's is the harmonic mean over both domains and three sections. So the comparison per machine is close, but not exactly like for like.

Data and licence

The data: DCASE 2022 Challenge Task 2 data (Hitachi, Ltd. and NTT Corporation), CC BY-NC-SA 4.0; results by Pelloran KELVIN, research use. No recording from the data is shown here.

Whom to talk to

Questions about this study, or a problem like it in your field: write to us.

Write to us → All studies