hydrator: Hydrator closes suddenly with no errors

I am Hydrating GeoCoV19 dataset which corresponds to May the 1st. Hydrator was working fine till it stopped hydrating and suddenly closed with no error messages.

Reopening the program and clicking Start would trigger the same behaviour: it simply shuts down with no explanations.

I checked the ids around where it stopped and they are legit, without any overflow. I restarted the machine as well to no avail. The jsonl file as of now is ~21GB in size.

Total Tweet Ids:
7,298,409

Tweet Ids Read:
4,485,700

Tweets Hydrated:
3,760,528

Percent Deleted:
16%

Any ideas on what I can do?

About this issue

Original URL
State: open
Created 4 years ago
Comments: 21 (9 by maintainers)

Most upvoted comments

Many thanks for debugging this @Tipphead!I will leave this open until i figure out the serialization issue

edsu on Feb 27, 2021

Update: Windows host, Windows VMs, and Ubuntu VMs are all running fine. The /Twitter/to/trump path was the issue. There must have been a point where either Twitter was being escaped by being the //shared folder or by me not realizing the shared folder did not begin with a T.

I just want to point out that when Hydrator runs on Linux, it will actually catch the issue and notify you where Windows will just shut down. Also, on Linux hydrator automatically converts to .jsonl where Windows goes to .txt. That’s fine as I prefer working with .txt. Another bug I’ve found is that on Linux, Hydrator has no icon in the task bar (not big deal just letting you know). Again, thanks for the program!

shullaw on Feb 27, 2021