dogecoin: [bug] estimatefee & estimatesmartfee always return -1 as result

I’m going to propose a series of pull requests to address this issue over the coming week, starting tonight:

• A new default configuration that makes estimatefee and estimatesmartfee functional, also in periods of spam - #3389
• A facility for users to configure this functionality to their liking, because developers should not dictate the fee a user pays.
• Expansive documentation to using smartfees, because using this facility (or any other fee estimation facility) does not come without caveats and risk.

This basically changes the automatic fee estimation feature from being unmaintained and a depreciation candidate, to an actively supported feature, and that is a policy change for Dogecoin Core (with 1.14.0, smartfees was said to no longer be supported.) Because there is plenty of time needed for testers to get results, consider this an open invitation for anyone to opine in on the desirability of supporting this feature.

Please comment your thoughts.

I’m working on a patch, and have done minimal orientation to manage an expedited release for it.

There are two major problems with the fee estimation logic:

1. The mempool is highly volatile, especially because the past couple of days people have been scripting and racing each other with intrinsically economically unsound transactions. The perceived external value makes people spend over thousands times the DOGE they are transacting. This can be fixed but it needs to be carefully tuned, or everyone, including people that do economically meaningful transactions, will overpay significantly (see below for an illustration.) Last year I did a lot of research work to enable the in-progress 1.21 major release with a good baseline, but that was before this behavior occurred and now I have to redo the tuning part to take into account “economic insanity”, which takes time.
2. Blockspace is less reliably used in Dogecoin. Unlike Bitcoin, Dogecoin does not try to prescribe a particular miner block templating behavior. This is because there is no mempool level consensus (also not on Bitcoin) so it makes no sense to have illusions of safety, and the economic incentive for mining Dogecoin blocks is not to extract as much fee as possible, because nowadays its subsidy is higher than all Dogecoin fees and all Litecoin subsidy AND fees combined: a miner has a bigger incentive to not have orphaned blocks than to include every transaction.

Over the past few hours I’ve seen the below estimates on a tuned, reconfigured model, but in reality your transaction will get mined with 0.001 DOGE/kb within 5 blocks with 70% certainty. So this already improved model is still overestimating fee 541 times because of the fee race happening, even though the size of a default configured mempool is < 2 blocks worth of transactions at the moment.

2 block: 0.78259712
3 block: 0.54144695
5 block: 0.54144695
7 block: 0.48850724
10 block: 0.40863303
15 block: 0.40863303
20 block: 0.40863303
25 block: 0.3433009

Thank you for this

estimatefee debugging for estimatesmartfee 6 says:

2023-11-27 17:31:42   6: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  65.07%  109316.6/(151887.1+16120 mempool)
2023-11-27 17:31:42   7: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  67.21%  112919.0/(151887.1+16116 mempool)
2023-11-27 17:31:42   8: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  69.57%  116641.7/(151887.1+15779 mempool)
2023-11-27 17:31:42   9: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  71.00%  119002.1/(151887.1+15730 mempool)
2023-11-27 17:31:42  10: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  72.56%  121622.8/(151887.1+15729 mempool)
2023-11-27 17:31:42  11: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  73.72%  123572.0/(151887.1+15729 mempool)
2023-11-27 17:31:42  12: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  74.91%  125360.1/(151887.1+15471 mempool)
2023-11-27 17:31:42  13: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  75.75%  126730.3/(151887.1+15405 mempool)
2023-11-27 17:31:42  14: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  76.74%  128233.0/(151887.1+15212 mempool)
2023-11-27 17:31:42  15: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  77.41%  129232.0/(151887.1+15054 mempool)
2023-11-27 17:31:42  16: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  78.19%  130525.8/(151887.1+15054 mempool)
2023-11-27 17:31:42  17: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  78.83%  131563.8/(151887.1+15005 mempool)
2023-11-27 17:31:42  18: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  79.45%  132546.7/(151887.1+14941 mempool)
2023-11-27 17:31:42  19: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  79.96%  133382.2/(151887.1+14931 mempool)
2023-11-27 17:31:42  20: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  80.22%  133815.2/(151887.1+14931 mempool)
2023-11-27 17:31:42  21: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  80.49%  134265.0/(151887.1+14926 mempool)
2023-11-27 17:31:42  22: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  80.73%  134656.2/(151887.1+14920 mempool)
2023-11-27 17:31:42  23: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  81.13%  135322.5/(151887.1+14912 mempool)
2023-11-27 17:31:42  24: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  81.56%  136034.5/(151887.1+14912 mempool)
2023-11-27 17:31:42  25: For conf success > 0.95 need feerate >:           -1 from buckets    1e+18 -    1e+18  Cur Bucket stats  81.89%  136590.5/(151887.1+14907 mempool)

So in essence, this will bring smart fees back?

Yes, but note that for now, the scope of my proposal is to bring back the API method into active support, because that’s what this issue is about. If that is successful, we can consider re-hanging it into the wallet as an optional or, if we’re confident that it won’t make everyone overpay, even the default mechanism to determine fee.

@patricklodder thanks for sharing it. I guess we have to wait for proper fix and estimate fee on our own.

Hej @mangekyousharingan 😃 nie chcesz zrobić review tłumaczenia? https://github.com/dogecoin/dogecoin/pull/3431

I’m running clean 1.14.6 and #3389 nodes for monitoring / comparison, and there were 3 moments today where this happened again for me (time in UTC):

Thank you for sharing this. I also installed modified /src/policy/fees.h from #3389 as a patch to test on my full node and estimatesmartfee was returning fee every time for various block parameter values, even when transaction count in mempool was around 50-60K.

Yes, but note that for now, the scope of my proposal is to bring back the API method into active support, because that’s what this issue is about. If that is successful, we can consider re-hanging it into the wallet as an optional or, if we’re confident that it won’t make everyone overpay, even the default mechanism to determine fee.

Nice! Yes, by the looks of it this will be needed. Glad you’re working on it. Thanks! 🙏

estimatefee back to be working as it was

That’s because there isn’t much to estimate at the moment. If there is a peak of traffic, it will go kaput again unless we fix it.

estimatefee back to be working as it was 😉 I guess it got fixed by itself? 🤔 Thank you @patricklodder for all the input! 🙏

I’ve been able to reproduce this right now (although until yesterday it was working fine) so current blocks are triggering a condition to invalidate the result - I’m not sure which one yet.