requests: POST Request in Heroku - 403 Forbidden

I’m new to using Requests, and I’m trying to scrape a schedule of classes. My POST request has resulted in “403 Forbidden” on Heroku, but works fine on my local machine. Here’s a code snippet I’m having trouble with in my application, using Python 2.7.4, Bottle, BeautifulSoup4, and of course, Requests.

import requests
from bs4 import BeautifulSoup as Soup

url = "https://telebears.berkeley.edu/enrollment-osoc/osc"
code = "26187"
values = dict(_InField1 = "RESTRIC", _InField2 = code, _InField3 = "13D2")
html = requests.post(url, params=values)
soup = Soup(html.content, from_encoding="utf-8")

sp = soup.find_all("div", {"class" : "layout-div"})[2]
print sp.text

It gives me back the string “Computer Science 61A P 001 LEC:” in development. However, when I tried to run it on Heroku (using heroku run bash and then run python), I got back “403 Forbidden”.

My attempts so far include manually setting User Agent headers, changing to verify=False, but neither worked. At first I thought it’s the school settings, but then I was wondering why it works locally without any problem… Any explanation/suggestion would be really appreciated! Thanks in advance.

About this issue

  • Original URL
  • State: closed
  • Created 11 years ago
  • Comments: 29 (16 by maintainers)

Most upvoted comments

@kqdtran did you ever figure out this issue? I’m facing a similar Heroku-specific problem with a GET request.