bitcoin: Coin selection algorithm not working as expected resulting in more fees

One user reported the issue on Bitcoin SE: https://bitcoin.stackexchange.com/questions/100437/bitcoin-cli-0-19-1-wallet-not-sending-from-addresses-with-closest-amount

UTXO A : 0.5 BTC UTXO B-Z : 0.5 BTC

UTXO B-Z are selected instead of UTXO A as the input for the transaction if I try to pay someone 0.5 BTC or more. Size of transaction is increased so user has to pay more fees.

Expected behavior

UTXO A should be selected as input automatically which will decrease the size of transaction and fees.

To reproduce

I tried to reproduce the issue by following below steps:

1. Create 850 addresses in bitcoin core: for ($i=1; $i -le 850; $i++) {.\bitcoin-cli.exe getnewaddress}

2. Add random amounts similar to the transaction mentioned by user in the issue: 81e681e4d7ed58a7645b3d8ae38c584e14954c359e3db8788b5ef645d65e9e64

   CSV file with address and amounts: https://mega.nz/file/ixsyjDYC#19Ff2qZP7LQjCvhz4ISF84uESPBesWrZmMVngJWGMOk

3. Create, Sign and Broadcast a transaction to one of the address in my bitcoin core wallet using above CSV file from electrum wallet

   Tx: https://blockstream.info/testnet/tx/b5c6c6dd204420020795fbf356d4db85a9398dbd99e575c72b76ec5f8d8d78e2

4. Send 0.54 BTC in one more transaction to my bitcoin core wallet:

   https://blockstream.info/testnet/tx/37675b6d44d1fe1b8095349f5d08d12197cee76504a0da6b0ab99fb29a4fa176

5. Finally we have a similar setup in bitcoin core to reproduce the issue. Try to send 0.49 BTC to some address and very small amounts to few other addresses by creating a transaction in bitcoin core wallet as the user did in tx mentioned on SE: https://i.imgur.com/utWKiZN.png

   https://i.imgur.com/WtduQx8.png

6. If I do not use “coin control features” and inputs are selected automatically:

1. If use “coin control features” and select UTXO with 0.54 BTC:

System information

Bitcoin Core v 0.20.0

Additional information

Still trying to understand how Coin Selection Algorithm works, why it didn’t select tb1qztazgd5665j9q0ww7qw24ms42gny6vc2h3z5yg as input automatically and how can we fix this.

The user has mentioned on SE:

I have a feeling that coin control favors more inputs with less combined “change” over less inputs with more combined change. In this case the change address (3BF5VhQgnabWMsB8QhVrBoWD6is2XMKuQs) received 0.00008051 BTC, where the input with the closest full amount amount would have resulted in (0.02544158 - tx fee) in change