zipline: Error retrieving H15 interest rates - ValueError: 'Time Period' is not in list
Dear Zipline Maintainers,
Environment
- Operating System: Linux 3d62a3c1924c 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- Python Version: Python 3.5.2 :: Anaconda 4.3.0 (64-bit)
- Python Bitness: 64
- How did you install Zipline: pip
- Python packages:
alabaster==0.7.9
alembic==0.9.2
anaconda-client==1.6.0
anaconda-navigator==1.4.3
appdirs==1.4.3
APScheduler==3.3.1
arctic==1.38.0
argcomplete==1.0.0
astroid==1.4.9
astropy==1.3
Babel==2.3.4
backports.shutil-get-terminal-size==1.0.0
bayesian-optimization==0.4.0
bcolz==0.12.1
beautifulsoup4==4.5.3
bitarray==0.8.1
blaze==0.10.1
bokeh==0.12.4
boto==2.45.0
Bottleneck==1.2.1
butils==0.2.13
cachetools==2.0.0
cffi==1.9.1
chardet==2.3.0
chest==0.2.3
click==6.7
cloudpickle==0.2.2
clyent==1.2.2
colorama==0.3.7
conda==4.3.9
conda-build==2.1.3
conda-verify==2.0.0
configobj==5.0.6
contextlib2==0.5.5
coverage==4.3.4
coveralls==1.1
cryptography==1.7.1
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.25.2
cytoolz==0.8.2
dask==0.13.0
data==0.3.7
datashape==0.5.4
deap==1.1.0
decorator==4.0.11
dill==0.2.5
docopt==0.6.2
docutils==0.13.1
dynd==0.7.3.dev1
empyrical==0.2.2
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==2.0.7
Flask==0.12
Flask-Cors==3.0.2
future==0.16.0
gevent==1.2.1
greenlet==0.4.11
h5py==2.6.0
HeapDict==1.0.0
hmmlearn==0.2.0
idna==2.2
imagesize==0.7.1
inspyred==1.0.1
intervaltree==2.1.0
ipykernel==4.5.2
ipython==5.1.0
ipython-genutils==0.1.0
ipywidgets==5.2.2
isort==4.2.5
itsdangerous==0.24
jdcal==1.3
jedi==0.9.0
Jinja2==2.9.4
jsonschema==2.5.1
jupyter==1.0.0
jupyter-client==4.4.0
jupyter-console==5.0.0
jupyter-core==4.2.1
jupyterhub==0.7.2
lazy-object-proxy==1.2.2
learning==0.4.19
llvmlite==0.15.0
locket==0.2.0
Logbook==1.0.0
lru-dict==1.1.6
lxml==3.7.2
lz4==0.8.2
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.0
minepy==1.2.0
mistune==0.7.3
mockextras==1.0.2
mpmath==0.19
multipledispatch==0.4.9
nb-anacondacloud==1.2.0
nb-conda==2.0.0
nb-conda-kernels==2.0.0
nbconvert==4.2.0
nbformat==4.2.0
nbpresent==3.0.2
networkx==1.11
nltk==3.2.2
nose==1.3.7
notebook==4.3.1
numba==0.30.1
numexpr==2.6.2
numpy==1.12.1
numpydoc==0.6.0
odo==0.5.0
openpyxl==2.4.1
packaging==16.8
pamela==0.3.0
pandas==0.18.1
pandas-datareader==0.4.0
partd==0.3.7
pathlib2==2.2.0
patsy==0.4.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.0.0
pkginfo==1.4.1
ply==3.9
prompt-toolkit==1.0.9
psutil==5.0.1
ptyprocess==0.5.1
py==1.4.32
pyasn1==0.1.9
PyBrain==0.3.3
pycosat==0.6.1
pycparser==2.17
pycrypto==2.6.1
pycurl==7.43.0
pyflakes==1.5.0
pyfolio==0.7.0
Pygments==2.1.3
pylint==1.6.4
pymongo==3.4.0
pyOpenSSL==16.2.0
pypandoc==1.3.3
pyparsing==2.2.0
pytest==3.0.5
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
PyWavelets==0.5.2
PyYAML==3.12
pyyawt==0.1.1
pyzmq==16.0.2
QtAwesome==0.4.3
qtconsole==4.2.1
QtPy==1.2.1
redis==2.10.5
requests==2.14.2
requests-file==1.4.2
requests-ftp==0.3.1
rope-py3k==0.9.4.post1
rpy2==2.8.5
scikit-image==0.12.3
scikit-learn==0.18.1
scipy==0.19.0
seaborn==0.7.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
snowballstemmer==1.2.1
sockjs-tornado==1.0.3
sortedcontainers==1.5.7
Sphinx==1.5.1
sphinx-rtd-theme==0.1.9
spyder==3.1.2
SQLAlchemy==1.1.10
statsmodels==0.8.0
sympy==1.0
TA-Lib==0.4.10
tables==3.4.2
terminado==0.6
toolz==0.8.2
tornado==4.4.2
TPOT==0.6.8
tqdm==4.11.2
traitlets==4.3.1
tsfresh==0.5.0
tzlocal==1.3
unicodecsv==0.14.1
update-checker==0.16
wcwidth==0.1.7
Werkzeug==0.11.15
widgetsnbextension==1.2.6
wrapt==1.10.8
xgboost==0.6a2
xlrd==1.0.0
XlsxWriter==0.9.6
xlwt==1.2.0
zipline==1.1.0
Error retrieving H15 interest rates - ValueError: ‘Time Period’ is not in list
- If I try to instantiate a
TradingEnvironment
, I get aValueError
. The code worked until yesterday and stopped some hours ago. I’m not sure whether on Th or Fr (CEST).
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
...
env = TradingEnvironment(bm_symbol=self.benchmark, exchange_tz=self.exchange_tz,
--> 539 trading_calendar=cal)
...
/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/finance/trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, asset_db_path)
94 trading_calendar.day,
95 trading_calendar.schedule.index,
---> 96 self.bm_symbol,
97 )
98
/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in load_market_data(trading_day, trading_days, bm_symbol)
169 first_date,
170 last_date,
--> 171 now,
172 )
173 benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]
/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in ensure_treasury_data(bm_symbol, first_date, last_date, now)
317
318 try:
--> 319 data = loader_module.get_treasury_data(first_date, last_date)
320 data.to_csv(path)
321 except (OSError, IOError, HTTPError):
/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/treasuries.py in get_treasury_data(start_date, end_date)
74 parse_dates=['Time Period'],
75 na_values=['ND'], # Presumably this stands for "No Data".
---> 76 index_col=0,
77 ).loc[
78 start_date:end_date
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
644 skip_blank_lines=skip_blank_lines)
645
--> 646 return _read(filepath_or_buffer, kwds)
647
648 parser_f.__name__ = name
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
387
388 # Create the parser.
--> 389 parser = TextFileReader(filepath_or_buffer, **kwds)
390
391 if (nrows is not None) and (chunksize is not None):
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
728 self.options['has_index_names'] = kwds['has_index_names']
729
--> 730 self._make_engine(self.engine)
731
732 def close(self):
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
921 def _make_engine(self, engine='c'):
922 if engine == 'c':
--> 923 self._engine = CParserWrapper(self.f, **self.options)
924 else:
925 if engine == 'python':
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
1434 raise ValueError("Usecols do not match names.")
1435
-> 1436 self._set_noconvert_columns()
1437
1438 self.orig_names = self.names
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set_noconvert_columns(self)
1484 _set(k)
1485 else:
-> 1486 _set(val)
1487
1488 elif isinstance(self.parse_dates, dict):
/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set(x)
1474
1475 if not is_integer(x):
-> 1476 x = names.index(x)
1477
1478 self._reader.set_noconvert(x)
ValueError: 'Time Period' is not in list
What steps have you taken to resolve this already?
- If I run the corresponding code manually, I get the same error. https://github.com/quantopian/zipline/blob/master/zipline/data/treasuries.py#L61
df = pd.read_csv(
"http://www.federalreserve.gov/datadownload/Output.aspx"
"?rel=H15"
"&series=bf17364827e38702b42a58cf8eaa3f78"
"&lastObs="
"&from=" # An unbounded query is ~2x faster than specifying dates.
"&to="
"&filetype=csv"
"&label=omit"
"&layout=seriescolumn"
"&type=package",
skiprows=1, # First row is a useless header.
parse_dates=['Time Period'],
na_values=['ND'], # Presumably this stands for "No Data".
index_col=0,
)
-
The request to the hard encoded link (to obtain the H15 interest rates) fails: “Unable to find the output file. Please contact administrator for assistance.” http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=omit&layout=seriescolumn&type=package
-
I played around with the Data Download Program and compared the parameters of the generated request strings. https://www.federalreserve.gov/releases/h15/
-
It seems as if
label=omit
is not accepted anymore. If I omit thelabel
parameter or set it toinclude
, it seems to work. http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package -
I am curious if I am the only one having this problem. If not, how can we fix this problem permanently?
Sincerely,
Rudi
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 30 (1 by maintainers)
Looks like fed changed request params and csv file again.
def get_treasury_data(start_date, end_date): return pd.read_csv( “https://www.federalreserve.gov/datadownload/Output.aspx” “?rel=H15” “&series=bf17364827e38702b42a58cf8eaa3f78” “&lastObs=” “&from=” # An unbounded query is ~2x faster than specifying dates. “&to=” “&filetype=csv” “&label=omit” “&layout=seriescolumn” “&type=package”, skiprows=1, # First row are useless headers. parse_dates=[‘Time Period’], na_values=[‘ND’], # Presumably this stands for “No Data”. index_col=0, ).loc[ start_date:end_date ].dropna( how=‘all’ ).rename( columns=parse_treasury_csv_column ).tz_localize(‘UTC’) * 0.01 # Convert from 2.57% to 0.0257.
works with above request params as usual.
Just a reminder the treasury data gives us the “risk free rate” which is used in the Sharpe calculation.
I’ve just merged #1818 and also ran
get_treasury_data()
in master and didn’t have any issues. If nobody else is seeing this I’ll go ahead and close this issue 😃 (feel free to re-open if you’re still seeing problems)Also https://github.com/quantopian/zipline/pull/1812
Currently I’m getting this error, I think because the website isn’t working properly right now. Shouldn’t we be also catching the ValueError in the get_treasure_data() function in treasuries.py? Maybe the data should be cached elsewhere? Or a copy updated with zipline with each release? Don’t be able to seem to run zipline backtesting at all if we can’t download this file at least once.
The URL seems to be
https://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package
Im having the same problems but in the treasury script its already set to “label=include”, what else can I do to resolve this issue?