zipline: Error retrieving H15 interest rates - ValueError: 'Time Period' is not in list

Dear Zipline Maintainers,

Environment

  • Operating System: Linux 3d62a3c1924c 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Python Version: Python 3.5.2 :: Anaconda 4.3.0 (64-bit)
  • Python Bitness: 64
  • How did you install Zipline: pip
  • Python packages:
alabaster==0.7.9
alembic==0.9.2
anaconda-client==1.6.0
anaconda-navigator==1.4.3
appdirs==1.4.3
APScheduler==3.3.1
arctic==1.38.0
argcomplete==1.0.0
astroid==1.4.9
astropy==1.3
Babel==2.3.4
backports.shutil-get-terminal-size==1.0.0
bayesian-optimization==0.4.0
bcolz==0.12.1
beautifulsoup4==4.5.3
bitarray==0.8.1
blaze==0.10.1
bokeh==0.12.4
boto==2.45.0
Bottleneck==1.2.1
butils==0.2.13
cachetools==2.0.0
cffi==1.9.1
chardet==2.3.0
chest==0.2.3
click==6.7
cloudpickle==0.2.2
clyent==1.2.2
colorama==0.3.7
conda==4.3.9
conda-build==2.1.3
conda-verify==2.0.0
configobj==5.0.6
contextlib2==0.5.5
coverage==4.3.4
coveralls==1.1
cryptography==1.7.1
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.25.2
cytoolz==0.8.2
dask==0.13.0
data==0.3.7
datashape==0.5.4
deap==1.1.0
decorator==4.0.11
dill==0.2.5
docopt==0.6.2
docutils==0.13.1
dynd==0.7.3.dev1
empyrical==0.2.2
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==2.0.7
Flask==0.12
Flask-Cors==3.0.2
future==0.16.0
gevent==1.2.1
greenlet==0.4.11
h5py==2.6.0
HeapDict==1.0.0
hmmlearn==0.2.0
idna==2.2
imagesize==0.7.1
inspyred==1.0.1
intervaltree==2.1.0
ipykernel==4.5.2
ipython==5.1.0
ipython-genutils==0.1.0
ipywidgets==5.2.2
isort==4.2.5
itsdangerous==0.24
jdcal==1.3
jedi==0.9.0
Jinja2==2.9.4
jsonschema==2.5.1
jupyter==1.0.0
jupyter-client==4.4.0
jupyter-console==5.0.0
jupyter-core==4.2.1
jupyterhub==0.7.2
lazy-object-proxy==1.2.2
learning==0.4.19
llvmlite==0.15.0
locket==0.2.0
Logbook==1.0.0
lru-dict==1.1.6
lxml==3.7.2
lz4==0.8.2
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.0
minepy==1.2.0
mistune==0.7.3
mockextras==1.0.2
mpmath==0.19
multipledispatch==0.4.9
nb-anacondacloud==1.2.0
nb-conda==2.0.0
nb-conda-kernels==2.0.0
nbconvert==4.2.0
nbformat==4.2.0
nbpresent==3.0.2
networkx==1.11
nltk==3.2.2
nose==1.3.7
notebook==4.3.1
numba==0.30.1
numexpr==2.6.2
numpy==1.12.1
numpydoc==0.6.0
odo==0.5.0
openpyxl==2.4.1
packaging==16.8
pamela==0.3.0
pandas==0.18.1
pandas-datareader==0.4.0
partd==0.3.7
pathlib2==2.2.0
patsy==0.4.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.0.0
pkginfo==1.4.1
ply==3.9
prompt-toolkit==1.0.9
psutil==5.0.1
ptyprocess==0.5.1
py==1.4.32
pyasn1==0.1.9
PyBrain==0.3.3
pycosat==0.6.1
pycparser==2.17
pycrypto==2.6.1
pycurl==7.43.0
pyflakes==1.5.0
pyfolio==0.7.0
Pygments==2.1.3
pylint==1.6.4
pymongo==3.4.0
pyOpenSSL==16.2.0
pypandoc==1.3.3
pyparsing==2.2.0
pytest==3.0.5
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
PyWavelets==0.5.2
PyYAML==3.12
pyyawt==0.1.1
pyzmq==16.0.2
QtAwesome==0.4.3
qtconsole==4.2.1
QtPy==1.2.1
redis==2.10.5
requests==2.14.2
requests-file==1.4.2
requests-ftp==0.3.1
rope-py3k==0.9.4.post1
rpy2==2.8.5
scikit-image==0.12.3
scikit-learn==0.18.1
scipy==0.19.0
seaborn==0.7.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
snowballstemmer==1.2.1
sockjs-tornado==1.0.3
sortedcontainers==1.5.7
Sphinx==1.5.1
sphinx-rtd-theme==0.1.9
spyder==3.1.2
SQLAlchemy==1.1.10
statsmodels==0.8.0
sympy==1.0
TA-Lib==0.4.10
tables==3.4.2
terminado==0.6
toolz==0.8.2
tornado==4.4.2
TPOT==0.6.8
tqdm==4.11.2
traitlets==4.3.1
tsfresh==0.5.0
tzlocal==1.3
unicodecsv==0.14.1
update-checker==0.16
wcwidth==0.1.7
Werkzeug==0.11.15
widgetsnbextension==1.2.6
wrapt==1.10.8
xgboost==0.6a2
xlrd==1.0.0
XlsxWriter==0.9.6
xlwt==1.2.0
zipline==1.1.0

Error retrieving H15 interest rates - ValueError: ‘Time Period’ is not in list

  • If I try to instantiate a TradingEnvironment, I get a ValueError. The code worked until yesterday and stopped some hours ago. I’m not sure whether on Th or Fr (CEST).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

...
env = TradingEnvironment(bm_symbol=self.benchmark, exchange_tz=self.exchange_tz,
--> 539                                  trading_calendar=cal)
...

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/finance/trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, asset_db_path)
     94             trading_calendar.day,
     95             trading_calendar.schedule.index,
---> 96             self.bm_symbol,
     97         )
     98 

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in load_market_data(trading_day, trading_days, bm_symbol)
    169         first_date,
    170         last_date,
--> 171         now,
    172     )
    173     benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in ensure_treasury_data(bm_symbol, first_date, last_date, now)
    317 
    318     try:
--> 319         data = loader_module.get_treasury_data(first_date, last_date)
    320         data.to_csv(path)
    321     except (OSError, IOError, HTTPError):

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/treasuries.py in get_treasury_data(start_date, end_date)
     74         parse_dates=['Time Period'],
     75         na_values=['ND'],  # Presumably this stands for "No Data".
---> 76         index_col=0,
     77     ).loc[
     78         start_date:end_date

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    644                     skip_blank_lines=skip_blank_lines)
    645 
--> 646         return _read(filepath_or_buffer, kwds)
    647 
    648     parser_f.__name__ = name

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    387 
    388     # Create the parser.
--> 389     parser = TextFileReader(filepath_or_buffer, **kwds)
    390 
    391     if (nrows is not None) and (chunksize is not None):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
    728             self.options['has_index_names'] = kwds['has_index_names']
    729 
--> 730         self._make_engine(self.engine)
    731 
    732     def close(self):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
    921     def _make_engine(self, engine='c'):
    922         if engine == 'c':
--> 923             self._engine = CParserWrapper(self.f, **self.options)
    924         else:
    925             if engine == 'python':

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
   1434                 raise ValueError("Usecols do not match names.")
   1435 
-> 1436         self._set_noconvert_columns()
   1437 
   1438         self.orig_names = self.names

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set_noconvert_columns(self)
   1484                         _set(k)
   1485                 else:
-> 1486                     _set(val)
   1487 
   1488         elif isinstance(self.parse_dates, dict):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set(x)
   1474 
   1475             if not is_integer(x):
-> 1476                 x = names.index(x)
   1477 
   1478             self._reader.set_noconvert(x)

ValueError: 'Time Period' is not in list

What steps have you taken to resolve this already?

df = pd.read_csv(
        "http://www.federalreserve.gov/datadownload/Output.aspx"
        "?rel=H15"
        "&series=bf17364827e38702b42a58cf8eaa3f78"
        "&lastObs="
        "&from="  # An unbounded query is ~2x faster than specifying dates.
        "&to="
        "&filetype=csv"
        "&label=omit"
        "&layout=seriescolumn"
        "&type=package",
        skiprows=1,  # First row is a useless header.
        parse_dates=['Time Period'],
        na_values=['ND'],  # Presumably this stands for "No Data".
        index_col=0,
)

Sincerely,

Rudi

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 30 (1 by maintainers)

Commits related to this issue

Most upvoted comments

Looks like fed changed request params and csv file again.

def get_treasury_data(start_date, end_date): return pd.read_csv( “https://www.federalreserve.gov/datadownload/Output.aspx” “?rel=H15” “&series=bf17364827e38702b42a58cf8eaa3f78” “&lastObs=” “&from=” # An unbounded query is ~2x faster than specifying dates. “&to=” “&filetype=csv” “&label=omit” “&layout=seriescolumn” “&type=package”, skiprows=1, # First row are useless headers. parse_dates=[‘Time Period’], na_values=[‘ND’], # Presumably this stands for “No Data”. index_col=0, ).loc[ start_date:end_date ].dropna( how=‘all’ ).rename( columns=parse_treasury_csv_column ).tz_localize(‘UTC’) * 0.01 # Convert from 2.57% to 0.0257.

works with above request params as usual.

Just a reminder the treasury data gives us the “risk free rate” which is used in the Sharpe calculation.

I’ve just merged #1818 and also ran get_treasury_data() in master and didn’t have any issues. If nobody else is seeing this I’ll go ahead and close this issue 😃 (feel free to re-open if you’re still seeing problems)

Also https://github.com/quantopian/zipline/pull/1812

Currently I’m getting this error, I think because the website isn’t working properly right now. Shouldn’t we be also catching the ValueError in the get_treasure_data() function in treasuries.py? Maybe the data should be cached elsewhere? Or a copy updated with zipline with each release? Don’t be able to seem to run zipline backtesting at all if we can’t download this file at least once.

The URL seems to be

https://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package

screenshot from 2019-02-13 19-38-19

Im having the same problems but in the treasury script its already set to “label=include”, what else can I do to resolve this issue?