py-motmetrics: False positives are over-reported

The motmetrics module seems to be reporting false positive tracks in situations when none exist. I confirmed this by running eval_motchallenge with the 2DMOT2015 ground truth files copied and used as the test results. The metric results are below - most are perfect, as would be expected, but FP can get quite high, and this affects downstream metrics like IDF1 and MOTA.

                 IDF1    IDP    IDR   Rcll   Prcn  GT  MT PT ML   FP FN IDs  FM   MOTA   MOTP
KITTI-13        90.1%  82.0% 100.0% 100.0%  82.0%  42  42  0  0  167  0   0   0  78.1% -0.000
ADL-Rundle-8   100.0% 100.0% 100.0% 100.0% 100.0%  28  28  0  0    0  0   0   0 100.0%  0.000
Venice-2       100.0% 100.0% 100.0% 100.0% 100.0%  26  26  0  0    0  0   0   0 100.0%  0.000
TUD-Campus     100.0% 100.0% 100.0% 100.0% 100.0%   8   8  0  0    0  0   0   0 100.0%  0.000
KITTI-17        93.2%  87.3% 100.0% 100.0%  87.3%   9   9  0  0   99  0   0   0  85.5% -0.000
ETH-Bahnhof     82.8%  70.6% 100.0% 100.0%  70.6% 171 171  0  0 2255  0   0   0  58.4%  0.000
PETS09-S2L1     98.1%  96.3% 100.0% 100.0%  96.3%  19  19  0  0  174  0   0   0  96.1%  0.000
TUD-Stadtmitte 100.0% 100.0% 100.0% 100.0% 100.0%  10  10  0  0    0  0   0   0 100.0%  0.000
ADL-Rundle-6   100.0% 100.0% 100.0% 100.0% 100.0%  24  24  0  0    0  0   0   0 100.0%  0.000
ETH-Sunnyday    98.9%  97.7% 100.0% 100.0%  97.7%  30  30  0  0   43  0   0   0  97.7%  0.000
ETH-Pedcross2   96.2%  92.7% 100.0% 100.0%  92.7% 133 133  0  0  495  0   0   0  92.1%  0.000
OVERALL         96.1%  92.5% 100.0% 100.0%  92.5% 500 500  0  0 3233  0   0   0  91.9%  0.000

I will look into this a bit and open a PR if I find the cause.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 18 (6 by maintainers)

Most upvoted comments

@antonmil. Correct, identical .txt files don’t necessarily mean perfect scores. However identical result matrices fed to the core evaluation algorithms give perfect scores.

ergysr on Apr 3, 2019

py-motmetrics has the same behavior as the official devkit. In summary there are differences in how the GT and Result files are parsed leading to seemingly imperfect scores. Yet the evaluation code gives perfect scores when the GT and Result variables are identical.

TUD-Stadtmitte
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
100.0 100.0 100.0|100.0 100.0  0.00|   10  10    0    0|     0     0     0     0| 100.0 100.0 100.0 
TUD-Campus
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
100.0 100.0 100.0|100.0 100.0  0.00|    8   8    0    0|     0     0     0     0| 100.0 100.0 100.0 
PETS09-S2L1
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 98.1 96.3 100.0|100.0  96.3  0.22|   19  19    0    0|   174     0     0     2|  96.1 100.0  96.1 
ETH-Bahnhof
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 82.8 70.6 100.0|100.0  70.6  2.25|  171 171    0    0|  2255     0     0   117|  58.4 100.0  58.4 
ETH-Sunnyday
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 98.9 97.7 100.0|100.0  97.7  0.12|   30  30    0    0|    43     0     0     7|  97.7 100.0  97.7 
ETH-Pedcross2
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 96.2 92.7 100.0|100.0  92.7  0.59|  133 133    0    0|   495     0     0     3|  92.1 100.0  92.1 
ADL-Rundle-6
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
100.0 100.0 100.0|100.0 100.0  0.00|   24  24    0    0|     0     0     0     1| 100.0 100.0 100.0 
ADL-Rundle-8
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
100.0 100.0 100.0|100.0 100.0  0.00|   28  28    0    0|     0     0     0     0| 100.0 100.0 100.0 
KITTI-13
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 90.1 82.0 100.0|100.0  82.0  0.49|   42  42    0    0|   167     0     0     0|  78.1 100.0  78.1 
KITTI-17
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 93.2 87.3 100.0|100.0  87.3  0.68|    9   9    0    0|    99     0     0     0|  85.5 100.0  85.5 
Venice-2
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
100.0 100.0 100.0|100.0 100.0  0.00|   26  26    0    0|     0     0     0     0| 100.0 100.0 100.0 


 ********************* Your MOT15 Results *********************
 IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL 
 96.1 92.5 100.0|100.0  92.5  0.59|  500 500    0    0|  3233     0     0   130|  91.9 100.0  91.9

ergysr on Apr 3, 2019