scikit-learn: Getting HTTPError: HTTP Error 403: Forbidden when trying to load California Housing dataset

Describe the bug

When trying to load the dataset I get an error.

Steps/Code to Reproduce

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split 
from sklearn.preprocessing import StandardScaler

housing = fetch_california_housing()

X_train_full, X_test, y_train_full, y_test = train_test_split(
housing.data, housing.target)
X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_valid_scaled = scaler.transform(X_valid)
X_test_scaled = scaler.transform(X_test)

Expected Results

Dataset loads

Actual Results

HTTPError                                 Traceback (most recent call last)
[/var/folders/wx/mz49j6yd5514yjn5k60sk6900000gn/T/ipykernel_16344/1379907178.py](https://file+.vscode-resource.vscode-cdn.net/var/folders/wx/mz49j6yd5514yjn5k60sk6900000gn/T/ipykernel_16344/1379907178.py) in <module>
      3 from sklearn.preprocessing import StandardScaler
      4 
----> 5 housing = fetch_california_housing()
      6 
      7 X_train_full, X_test, y_train_full, y_test = train_test_split(

[~/opt/anaconda3/lib/python3.9/site-packages/sklearn/datasets/_california_housing.py](https://file+.vscode-resource.vscode-cdn.net/Users/aryamanbhatia/neural%20network%20for%20practice/~/opt/anaconda3/lib/python3.9/site-packages/sklearn/datasets/_california_housing.py) in fetch_california_housing(data_home, download_if_missing, return_X_y, as_frame)
    133     This dataset consists of 20,640 samples and 9 features.
    134     """
--> 135     data_home = get_data_home(data_home=data_home)
    136     if not exists(data_home):
    137         makedirs(data_home)

[~/opt/anaconda3/lib/python3.9/site-packages/sklearn/datasets/_base.py](https://file+.vscode-resource.vscode-cdn.net/Users/aryamanbhatia/neural%20network%20for%20practice/~/opt/anaconda3/lib/python3.9/site-packages/sklearn/datasets/_base.py) in _fetch_remote(remote, dirname)

[~/opt/anaconda3/lib/python3.9/urllib/request.py](https://file+.vscode-resource.vscode-cdn.net/Users/aryamanbhatia/neural%20network%20for%20practice/~/opt/anaconda3/lib/python3.9/urllib/request.py) in urlretrieve(url, filename, reporthook, data)
    237     url_type, path = _splittype(url)
    238 
--> 239     with contextlib.closing(urlopen(url, data)) as fp:
    240         headers = fp.info()
    241 
...
--> 641         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    642 
    643 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

Versions

System:
    python: 3.9.13 (main, Aug 25 2022, 18:29:29)  [Clang 12.0.0 ]
executable: /Users/aryamanbhatia/opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 22.2.2
   setuptools: 63.4.1
      sklearn: 1.0.2
        numpy: 1.26.3
        scipy: 1.9.1
       Cython: 0.29.32
       pandas: 1.4.4
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 36 (21 by maintainers)

Commits related to this issue

Most upvoted comments

I cannot reproduce. Either it was something transient or it is on your network side.

Some kind of summary comment:

Indeed, it’s easy to disconnect the runtime (and then reconnect by reexecuting the cells from the start of the notebook) using the following drop down menu:

image

You have approximately a 50%/50% chance to get rebalanced to a node where the HTTP requests go through.

I got the same error on Google Colab too. Someone said this bug occurs very frequent. Sometime it works, sometime it doesn’t. Is there any solution to it?

@lesteve Great support - thank you!!!

I got an answer from Figshare support saying that they fixed the issue. I tried roughly 10 times on Colab and the same on Kaggle notebooks and I was not able to reproduce the issue 🎉.

I am going to close this one, if you encounter the issue again please comment in this issue!

You can also try the documented work-arounds in https://github.com/scikit-learn/scikit-learn/issues/28297#issuecomment-2178930890 and mention in case that does not fix the issue.

I tried on Kaggle notebooks and it seems like indeed the problem happens from time to time. The work-around is to click on the power button (Stop session), reexecute the code (you will get a different IP address) and cross your fingers 🤞,

@lesteve Thanks for your comment. Yes, that’s exactly what I did. I just reported to possibly help troubleshoot this issue 😃

It’s strange - the code runs perfectly on Colab but fails on Kaggle, at the same time even on the same local machine. Maybe this info can help troubleshoot the issue.

I tried on Kaggle notebooks and it seems like indeed the problem happens from time to time. The work-around is to click on the power button (Stop session), reexecute the code (you will get a different IP address) and cross your fingers 🤞, make sure you have internet enabled as well (just in case).

The debugging command I used is the following one, which downloads the file needed for California housing:

!wget https://ndownloader.figshare.com/files/5976036 --timeout 3

Here is a screenshot where it did not work (HTTP 403) image

Here is a screenshot where it did work (HTTP 200): image

I inquiry the support from figshare to see if they are aware of some limitations on their side.

Same here! (Based in France FYI, but using Colab)

I tried yo change the account and it worked. If you change the account that you are working with and try the same code it should be fine.