cli: Brew installed version doesn't work with plugins

In the meantime, maybe a note in the documentation around plugins saying “brew doesn’t work with plugins and to pip install httpie for plugin functionality” would save some headaches.

For what it’s worth and perhaps other users googling for an answer, installing a plugin worked fine for me like this:

pip3 install -t /usr/local/Cellar/httpie/2.0.0/libexec/lib/python3.8/site-packages httpie-jwt-auth

Thanks https://github.com/jakubroztocil/httpie/issues/566#issuecomment-285210693

@rshurts you’re right. Plugin installation for Homebrew-based installs doesn’t work out of the box.

There are advantages to keeping Homebrew as the recommended method, though. It makes updates easy and the main package can depend on Python 3 (improved SSL support, etc.).

The ideal world solution would be to have all plugins available as Homebrew packages (brew install httpie-oauth) . But that would mean extra work for maintainers so not all plugins would end up in Homebrew.

Using pip to install plugins looks like the easier way to go. To make it work with Hombrew, it would mean instructing pip to use (currently) /usr/local/Cellar/httpie/0.9.8_2/libexec/lib/python3.6/site-packages as the installation directory.

But plugins installed there won’t survive HTTPie upgrades.

Perhaps it would be possible to tweak the Homebrew formula to makeallow HTTPie load installed plugins from, for example, the Homebrew Python 3 site-packages (not sure if it’s permanent). Or from another location.

…or some sort of custom plugin installer.

🎉 This feature is now publicly available in HTTPie for Terminal 3.0:

👉🏻 https://httpie.io/docs/cli/plugin-manager

We have finally landed httpie plugins interface, though the initial QA on brew is not yet completed. I’ll probably try to complete it within today or tomorrow and then close this ticket.

From an outsiders look, this issue seem like a problem for the people who are actually maintaing plugins rather than the httpie itself. If an X tool is installed through Y platform, then it is fair to expect to get all the plugins to be installed in the same manner. So if the plugin maintainers also package their plugins for brew to support those users, that would be the general and (somewhat) unconvient solution (which is what we currently have with all those python-<xxx> packages on distro indexes).

Possible solutions

If we turn the other way around, and try to offer a more user friendly solution on the httpie side as far as I can tell there are 2 options that I can think of:

–interpreter (less human friendly & more error prone)

Since we don’t have any actual chance of knowing what is the primary python environment they have installed the plugin (we can only assume it is the global one, or whatever python resolves [which might python2 or python3] on the current PATH), we need an explicit way for the users to pass a Python environment.

The general solution that I’ve seen (e.g on pyperf/pyperformance) for this sort of problem is that you pass an --interpreter option with the interpreter you want python3 or .venv/bin/python and it resolves that name and find’s the executable (and then retrieves the site-packages to search through). Which is somewhat esoteric and very unfriendly. Though it might became a little bit less unfriendly by allowing this to be a global option:

[~/.httpie]

{
    "plugins": {
        "interpreter": "C:\Users\Binary\python.exe"
    }
}

2 advantages over directly pointing to site-packages:

• Knowing where all 3rd party packages reside in is not a common knowledge, and it’s location might change depending on the environment (distro/os).
• Having the interpreter allows doing more robust version checks (e.g ensuring interpreter.version_info == sys.version_info)

Advantages over #1108:

• Supports virtual envs
• Explicitl version checks
• Customization on the Python interpreter itself

The implementation would be simple; and works by firing the new python interpreter, retrieving the location for the site-packages and temporarily include them on the entry-point search.

httpie plugin <cmd> <plugin> (more user friendly)

A more user friendlier, but rather complex solution is to httpie to manage it’s own plugins. It will be simply a wrapper around the pip, and manage a custom plugin-base on it’s own. This can either install plugins globally to the httpie environment (so they be accessed outside of httpie, e.g requests-hawk) or it can install them to an isolated place so the normal site package loaders don’t have access to them. (both are fine options IMO)

Being able to directly install plugins is something that this interface can delegate to pip under the hood (which is guaranteed to be bundled from 3.4+, but distros like debian is a bit problematic about it [can be solvable by explicitly depending on python-pip]).

Example scenerio;

$ httpie plugins install httpie-auth
Installed httpie-auth (2.3.0)
$ httpie plugins list
httpie-auth (2.3.0): <short desc>
httpie-xxx (2.3.0): <short desc>
$ httpie plugins upgrade --all
Upgraded httpie-auth (2.4.0)
Upgraded httpie-xxx (1.1.5)
$ httpie plugins uninstall httpie-auth
Removed httpie-auth

Advantages over the --interpreter:

• Much more easier for everyone to understand, has a better UI.
• Plugin loading errors (e.g due to python version mismatch, or any other bug) can be suppressed and delegated to user about what they can do (e.g try httpie plugins upgrade)
• Can be extended further to enable / disable specific plugins
• More isolated management

Existing plugins won’t require any changes except for updating their installation instructions. That will be optional since I think we should continue to load plugins from the main site-packages as well.

Indeed. For regular installations, both options should work (pip + httpie plugins install) with httpie plugins install being the preferred one on the documentation.

But I think a Python installation’s identity shouldn’t change when it’s just upgraded to a new version (e.g. Python 3.6.10 → 3.6.11). Because without that, the user will lose plugins after each Python upgrade. Not sure how to approach that, though.

What I meant by md5(python-identifier) was not exactly the python version, but rather the md5() of the path to the site-packages, or to the python interpreter (sys.executable). There are 2 reasons for that:

• There might be multiple httpie’s installed accross multiple virtual environments with the same python version but different set of plugins, this way since they will all have different paths to their executables the python-identifier will be different and the isolation will stay in place
• We will mirror the python upgrades as they are happening. For example if a minor version bump (3.6.10 -> 3.6.11) does preserve the site-packages, so do we. But for major bumps, we will change it. (This should happen rarely, even with PEP 602 it takes about 1.5/2 years for built packages to bump their interpreter’s major version).

Many plugins depend on HTTPie itself. I think we should prevent its installation into the plugins directory, or uninstall it afterward. We should probably also validate if there’s some specific version requirement.

Yep, this will be taken care of. Since httpie itself is actually a dependency of httpie (as a package), we will already have it and I assume it should be possible for pip to recognize it as present so it won’t bother installing it. Since it is also recognizing it, and it’s metadata pip will find the correct version of the plugin that fits all dependency constraints (e.g httpie>=current_version).

As mentioned above, I think we should continue to support the existing global site-packages installation location.

Agreed.

Then we should gracefully handle multiple plugins installation (define lookup order).

What do you mean by define lookup order? If it is for the installation, pip should do the ordering of dependency graph, but if it is something else not sure.

The plugins directory should only be used to find the entry points for plugins (and to load them and their deps), but we should avoid anything else installed there to be imported by httpie (e.g., if requests are installed there it shouldn’t be touched by httpie).

As I mentioned in the proposal, we will temporarily make the pkg_resources to recognize the plugins directory as a location for site packages and remove it afterwards. It should not affect anything in terms of global state / other imports, only plugins.

Later, we could also consider providing httpie plugins search, etc. (plugin names correspond to their PyPi names).

Makes sense (as well as plugins enable/plugins disable).

Will this cause any issues for Snap and other installation methods?

One important thing to note here is that, we will need to have write access to the following httpie directory. If I am not mistaken, the default home interface for snap does not support dot-files (hidden):

The home interface allows access to non-hidden files owned by the user in the user’s home ($HOME) directory where a user normally stores their personal files and documents.

(from https://snapcraft.io/docs/home-interface)

There seems to be some discussions with workarounds, https://forum.snapcraft.io/t/accessing-xdg-config-home-of-the-host/9780/10, but I am not reliable are them (needs further research).

Managing plugins through an HTTPie-provided command looks like the best solution all things considered.

---

httpie command

Currently, the HTTPie package doesn’t come with a command called httpie (only http and https).

At one point, I was going to add it for sessions management but then decided against it (https://github.com/httpie/httpie/commit/5cc5b13555f359712e0c5784bebdb091c34e6c82) to keep the project simple.

Now, however, with this plugin management use case (and possible upcoming integration with our new companion web & desktop app that we’re working on) it makes sense to revisit the idea.

A new command is much cleaner than trying to enrich the existing ones, where we already have quite complex args parsing logic due to our “DSL”, and as @isidentical pointed out, it doesn’t really belong there anyway.

The new command should only be used for management, not for making requests. When invoked without any args or with args that are invalid (or maybe even better — they look like the user is trying to make a request), we should instruct the user to use http [args] or https [args] (this will actually improve the onboarding experience as well because the project/command name mismatch can be confusing and some users will inevitably type httpie post-installation, so this will give them a useful hint).

---

So, with a custom plugin install command, we can continue to use pip for plugins + deps installation (under the hood) and pkg_resources for installed plugin lookup and loading.

Existing plugins won’t require any changes except for updating their installation instructions. That will be optional since I think we should continue to load plugins from the main site-packages as well.

I really like the idea of generating a site-packages-like directory for plugins for each interpreter in ~/.config/httpie/plugins/<md5(python-identifier)>/.... The isolation is important to keep httpie’s development simple, among other things.

But I think a Python installation’s identity shouldn’t change when it’s just upgraded to a new version (e.g. Python 3.6.10 → 3.6.11). Because without that, the user will lose plugins after each Python upgrade. Not sure how to approach that, though.

We could also consider keeping the desired plugin list in the config file and allowing the user to quickly re-install the missing ones.

Some more thoughts:

• Many plugins depend on HTTPie itself. I think we should prevent its installation into the plugins directory, or uninstall it afterward. We should probably also validate if there’s some specific version requirement.
• As mentioned above, I think we should continue to support the existing global site-packages installation location.
• Then we should gracefully handle multiple plugins installation (define lookup order).
• The plugins directory should only be used to find the entry points for plugins (and to load them and their deps), but we should avoid anything else installed there to be imported by httpie (e.g., if requests are installed there it shouldn’t be touched by httpie).
• Later, we could also consider providing httpie plugins search, etc. (plugin names correspond to their PyPi names).
• Will this cause any issues for Snap and other installation methods?