core: Nissan Leaf component breaks in 0.118.0b3

I’m investigating a fix for this. Apologies for the delay. Thank you for all the pull requests. Hopefully have an updated release of pycarwings2 within the next day or so once I’ve got the fix working in my dev environment.

There is a pull request for a fix, but codeowner is nowhere to be seen. Sadly we need him to get it accepted.

As mentioned in my previous note, it seems that simply setting the User-Agent in the headers= parameter to an empty string ("") instead of None will prevent urllib from adding its own and does not send it out along with the request.

I run Home Assistant in a docker container and by editing the /usr/local/lib/python3.8/site-packages/pycarwings2/pycarwings2.py file and adding the headers={"User-Agent": ""} parameter on line 121, the integration starts working again. This change is also what I (tried to) put into a pull request to filcole/pycarwings2 which seems to be the used version.

I already sent out a pull request to the pycarwings2 as there didn’t seem to be a way to suppress the header (setting it to None, which worked before still added the header), however setting it to an empty string "" will still suppress the header. I’ve closed my pull request, but I’ll set up another one. Let’s hope it gets accepted soon.

Hi, and thank you @andbad for the suggestion.

1. Downgrade to previous version
2. Upgrade to newest
3. Restart HA
4. test integration

I followed the procedure above, but unfortunately no success. Here is my experience: I used “ha core update --version=x.xxx.x” to change between versions In my setup, it appears to have broken between version 0.118.0b2 and 0.118.0b3. Perhaps someone with more programming skills than me can find a change between those versions that may cause this?

0.117.6 ok 0.118.0b0 ok 0.118.0b1 ok 0.118.0b2 ok 0.118.0b3 Broken 0.118.0 Broken 0.118.1 Broken 0.118.2 Broken

I change the line 121 with: req = Request('POST', url=BASE_URL + endpoint, headers={"User-Agent": ""}, data=params).prepare()

and seems to work.

By(t)e

By simply changing the URL to a netcat listener, it seems the newer version has a User-Agent header while the old one does not send one.

New:

POST /InitialApp_v2.php HTTP/1.1
Host: 172.18.0.1:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded
User-Agent: python-urllib3/1.26.2

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=^C

Old:

POST /InitialApp_v2.php HTTP/1.1
Host: 10.0.0.50:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=

It seems that setting the User-Agent to None in older versions of the requests library would get rid of the header altogether, but in newer versions it defaults to using the python-urllib3/(version) header.

I might be way off track and will have to test some more to figure out if this even is the issue.

Shame that HA stops because of one person. Am unable to update from 0.117.6 because of this.

This appears to be broken again.

HA log: carwings error INVALID PARAMS: -2010

An unknown error occurred while connecting to Nissan: <class ‘pycarwings2.pycarwings2.CarwingsError’>

Shame that HA stops because of one person. Am unable to update from 0.117.6 because of this.

It’s also a byproduct of the insistence on using libraries via pip even for very specific things, so HASS relies on that package being updated by the maintainer and if it isn’t then tough luck. I suppose someone could republish the library mind you.

By simply changing the URL to a netcat listener, it seems the newer version has a User-Agent header while the old one does not send one. New:

POST /InitialApp_v2.php HTTP/1.1
Host: 172.18.0.1:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded
User-Agent: python-urllib3/1.26.2

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=^C

Old:

POST /InitialApp_v2.php HTTP/1.1
Host: 10.0.0.50:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=

It seems that setting the User-Agent to None in older versions of the requests library would get rid of the header altogether, but in newer versions it defaults to using the python-urllib3/(version) header. I might be way off track and will have to test some more to figure out if this even is the issue.

I would not be surprised in the slightest if the Nissan API throws an error for one of a forward slash or a period. They may well use it to check if you’re trying to access the API via a browser.

I have created a bugreport with pyrequest regarding urllib3 default header: psf/requests#5671 Lets see how that is handled, i hope that urllib3: urllib3.util.SKIP_HEADER can be introduced into pyrequest for easy means of deactivating headers.

This would be unwise to do in a larger application like Home Assistant, as other components may rely on it in the future and I would imagine that a static variable like that isn’t going to be sandboxed. (Assuming it’s a static variable?)

And why would is so? It has nothing to do with other applications relaying on pyrequest.

If pyrequest could use the variable “user-agent = none” and simply use the urllib3 function: urllib3.util.SKIP_HEADER the problems is solved. And i dont see how that should affect anything, and certanly not ha.

The right way is to use it the way i have pointed out. This is the way pyrequest always have worked and by doing this this way it should not break anything. When urllib3 introduced default user agent header this function is also introduced at the same time.

Any way of doing this on HASSIO?

1. log in using the SSH / Terminal add-on in supervisor
2. enter the following command: ha core update --version 0.117.6

The system will restore to a version that works with carwings.

Ok it seems urllib3 1.26.x has some interesting changes:

https://pypi.org/project/urllib3/

Default user-agent is now added

Added default User-Agent header to every request (Pull #1750)
Added urllib3.util.SKIP_HEADER for skipping User-Agent, Accept-Encoding, and Host headers from being automatically emitted with requests (Pull #2018)

I tried digging some more into this issue. A quick test of pycarwings2 with the get-leaf-info.py (found under examples in the repo) is working fine. There has been no version changes of the integration between 0.118.0b2 and 0.118.0b3. I also tried debug logging of urllib3 to see if there were some differences, but there’s not. The second request to the Nissan servers simply returns {"status":404,"message":"INVALID PARAMS"} in the newer versions, like what you get if you just try to open the URL with no, or invalid, POST data.

As previously stated, https://github.com/home-assistant/core/pull/43279 is the only PR I can see that has any relation to this integration, as the requests package is used by pycarwings2, but I have no idea how, and if, it could be related.

This could be some weird change in the requests library then, however I wouldn’t rule out the Nissan servers being flakey. If you can sometimes get it to login without changing then it’s probably then, as their servers are downright terrible