panel: Panel loading is too slow

With the lastest panel version, loading it takes too long. Below is an example:

%%time

import panel as pn
pn.extension()

display(pn.widgets.Button(name='Click me'))

I get the following times:

CPU times: user 1.03 s, sys: 871 ms, total: 1.9 s
Wall time: 25 s