bitcoin: Add `blockfilterindex = p2wpkh`
I would like to have a p2wpkh blockfilterindex type to outsource Wasabi’s filter generation to Bitcoin Core, as our filter generation is orders of magnitude slower and we would like to bring this to the client side too, so our users with full nodes would be able to validate that our server does not cheat.
I previously created a blockfilterindex extensibility proposal to enable filter type combinations: https://github.com/bitcoin/bitcoin/issues/18221. So the followup PR of this issue will make sure it adheres to it. Since the PR is ready already, I’ll omit to go into the implementation details in this issue, rather concentrate on the goal, the benefits of end users and how we ruled out the alternative solutions.
The Goal
We would like to enable our users to upgrade Wasabi to be a full node with one click.
This feature is already implemented and rolled out to production (shipping with Core, fee estimation, tx broadcasting, mempool serving etc…) except the (most important) validation part of it is yet to be added.
Alternative Solutions
I’ve been long struggling with how to enable this for our users and we ended up working together with a brilliant dev, @dangershony, who maintained the C# Bitcoin full node for years. Unfortunately we had to rule out all the possible alternatives.
One alternative would be using importmulti, but we would not like to rely on the wallet feature of Bitcoin Core, rather stick to the node functionality.
Another alternative could be to slowly replace our filters with Bitcoin Core’s filters on the client side, but that would require reindexing all of our wallets, which in some cases could take half a day. Also mixing filter types feels dirty and can result in unexpected issues.
We also considered bringing our own filter generation algorithm to the client side, but this is one of the oldest brain class of our codebase that serves a critical functionality so touching this could be problematic. It currently takes 2-3 days for us to create our filters on a powerful server, even though we are using p2wpkh filters and only building them from segwit activation. Bitcoin Core creates these filters within 2 hours. Why we are so slow is out of the scope of this issue, but here’s the clue: input scripts aren’t part of a transaction.
Further we considered just blindly going through all the blocks and process all the transactions and make sure the server doesn’t lie. The performance here is also unacceptable.
Finally we would also don’t like to use bloom filters, as it is on the edge of obsolation and having multiple filters to achieve the same purpose feels dirty from a complexity point of view.
Benefits to the End User
@gmaxwell has suggested to articulate the benefits to the end user for changes to the codebase, thus this section.
Adding new filter types and filter type combinations (https://github.com/bitcoin/bitcoin/issues/18221) would enable a privacy oriented light wallet surge to be built on top of Bitcoin Core.
Maybe more importantly Bitcoin Core providing the filters to Wasabi enables a clean architecture and a seamless UX for users willing to run a full node that comes with Wasabi.
It is quite unique in a sense that no other popular light wallets went to the extent of building a full node into their software, especially not in a way that only requires a a single click configuration, so if our feature gets out of “work in progress” phase there may be an influx of full node users, which is good for the network.
What we stand to lose is that this full node feature will be delayed into the future so far that I may not be here when the next clean solution presents itself and maybe the Wasabi team won’t be willing to work on it as frankly it has little business value.
What’s Next?
This issue described our motivation and the context of the feature request, our pull request is coming up soon, so we discuss the specifics.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 6
- Comments: 23 (16 by maintainers)
The basic filter type covers this application. The small bandwidth savings comes exclusively from the diminished usage of the P2W* outputs and will vanish as adoption patterns change, it comes at an expenses of an (eventual) doubling of disk space usage for filters, and (if used over the network) a reduction in the querying user’s anonymity set.
Additional filters also reduce the prospect of ever having the filters in committed form that could be securely queried by clients that do not trust the server, as the additional cost (delay for computation) for computing multiple filters weighs against a nearly constant benefit that is achieved by the basic filter type.
If the current size of the filters is a killer for an application, the application will end up getting screwed in a couple years regardless-- because the shrunk filters will very likely exceed the current size by a healthy margin by then (due to changing usage patterns as well as simple cumulative growth).
Proliferation of filter types was one of the arguments being made against support for these filters in the first place, and I’m saddened to see those concerns proving prescient.
@MarcoFalke Thanks for bringing up the filter from wallet creation idea. We did not consider it this time. Back then this is how I implemented Hidden Wallet (predecessor of Wasabi) but in Wasabi I dropped the concept because others were bringing up privacy concerns. I personally don’t think there are any leaks that wouldn’t be just nitpicking, yet I dropped the concept in Wasabi with the thinking P2PWPKH filters from SegWit activation will just be fine to create a light wallet feeling and hopefully the filters’ size will grow slower than the tech improves and if not then we implement it.
At this point I don’t feel like implementing it again as long as there is a chance for having a new filter type, but if it is rejected, that could be a way forward.
@Sjors to answer for your very first question. The
P2WPKHvsP2WPKH+P2WSHfilter sizes are249MBvs251MB.@schildbach that’d be awesome if there’d be a mobile wallet that uses client side filters. Finally a mobile wallet I could recommend for privacy!
@dangershony It seems to be there’s a consensus here that
P2WPKH+P2WSHfilters is preferable overP2WPKHfilters, would you mind updating your PR?Closing this given this hasn’t moved since 2020 and related #18221/#18223/https://github.com/bitcoin/bips/pull/896 were closed due to lack of support for additional filters. Can re-open if there is future interest.
I tend to agree with the logic behind a
segwit0filter type, I can compile an alternative PR that will build thesegwit0indexes if that has a better chance to be accepted.I don’t expect exponential growth here.
p2wpkh + p2wshcovers all of SegWit v0, which is great for wallets that don’t want to deal with pre-SegWit legacy. The next filter I would expect is for SegWit v1 by the time that comes out. Again I can see how future wallets would use SegWit v1 and not bother with v0 or legacy. Having access to smaller size filters, and with it a lower false positive rate, is a nice incentive to adopt something more modern (especially on mobile).Too many filters wouldn’t be good, because DNS seeds have to announce them, and because it adds fingerprint bits.
I thought you suggested that Wasabi start serving Bitcoin Core filters, so I reflected to that idea. Your suggestion seems to be the one where Wasabi serves its own filters to the users and users with full nodes are slowly replacing Wasabi filters with Core filters and as Core filters take over it would work with that instead of Wasabi filters. Sorry, I wasn’t clear about the hybrid nature of our full node mode implementation. Wasabi is a light wallet until the full node catches up.
I described this scenario in my post, so let me quote myself:
It’s a matter of ~5GB vs ~200MB filters. This difference presents itself on the user’s computer in storage, initial sync time (+the accompanying bandwidth usage) and filter processing time.