scrapyd: Wrong type adaption for BLOB in scrapyd sqlite3 utils

When I execute

curl http://localhost:6800/schedule.json -d project=default -d spider=spider1

I go to scrapyd console and I see the Exception:

[-] Unhandled Error
Traceback (most recent call last):
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/poller.py", line 23, in poll
    msg = yield maybeDeferred(q.pop)
--- <exception caught here> ---
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/spiderqueue.py", line 21, in pop
    return self.q.pop()
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/sqlite.py", line 119, in pop
    return self.decode(msg)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/sqlite.py", line 170, in decode
    return json.loads(text)
  File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
exceptions.ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

After that I can’t run next Spider, I get {"status": "error", "message": "No JSON object could be decoded", "node_name": "spider-test2"}

How to determine what that error is related to?

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 32 (19 by maintainers)

Most upvoted comments

Finally! But you also need to add self.conn.text_factory = bytes to SqliteDict class after connect creation. Now everything works perfectly! Thank you!

dkachan1941 on May 20, 2016

Probably a bug in sqlite or the python module. Try this:

diff --git a/scrapyd/sqlite.py b/scrapyd/sqlite.py
index 69f44a6..3d9cf80 100644
--- a/scrapyd/sqlite.py
+++ b/scrapyd/sqlite.py
@@ -93,6 +93,7 @@ class SqlitePriorityQueue(object):
         self.table = table
         # about check_same_thread: http://twistedmatrix.com/trac/ticket/4040
         self.conn = sqlite3.connect(self.database, check_same_thread=False)
+        self.conn.text_factory = bytes
         q = "create table if not exists %s (id integer primary key, " \
             "priority real key, message blob)" % table
         self.conn.execute(q)

Digenis on May 20, 2016