moon: [bug] Build timing out in GitHub Actions
Describe the bug
After updating from 0.22.0 to 0.23.4 builds in GitHub Actions are timing out. However while running the build locally it succeeds. I suspect, though I can not be sure, that the action runner is encountering memory constraints and then stalling.
Steps to reproduce
Unfortunately, I don’t have reproduction steps here. At least not steps I can give because I think it has to do with the setup of my specific project which is not open source.
Expected behavior
The build runs to completion.
Environment
I capturee this information with npx envinfo
. It is a GitHub Actions runner based on ubuntu-20.04
.
System:
OS: Linux 5.15 Ubuntu 20.04.5 LTS (Focal Fossa)
CPU: (2) x64 Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Memory: 5.72 GB / 6.78 GB
Container: Yes
Shell: 5.0.17 - /bin/bash
Binaries:
Node: 16.19.0 - /opt/hostedtoolcache/node/16.19.0/x64/bin/node
Yarn: 3.3.0 - /usr/local/bin/yarn
npm: 8.19.3 - /opt/hostedtoolcache/node/16.19.0/x64/bin/npm
Managers:
Apt: 2.0.9 - /usr/bin/apt
Cargo: 1.67.0 - ~/.cargo/bin/cargo
Composer: 2.5.1 - /usr/bin/composer
Gradle: 7.6 - /usr/bin/gradle
Maven: 3.8.7 - /usr/bin/mvn
pip3: 20.0.2 - /usr/bin/pip3
RubyGems: 3.1.2 - /usr/bin/gem
Utilities:
Bazel: 6.0.0 - /usr/local/bin/bazel
CMake: 3.[25](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:26).2 - /usr/local/bin/cmake
Make: 4.2.1 - /usr/bin/make
GCC: 9.4.0 - /usr/bin/gcc
Git: 2.39.1 - /usr/bin/git
Clang: 11.0.0 - /usr/bin/clang
Mercurial: 5.3.1 - /usr/bin/hg
Subversion: 1.13.0 - /usr/bin/svn
Servers:
Nginx: 1.18.0 - /usr/sbin/nginx
Virtualization:
Docker: 20.10.22+ - /usr/bin/docker
SDKs:
Android SDK:
API Levels: [27](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:28), [28](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:29), [29](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:30), [30](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:31), [31](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:32), [32](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:33), [33](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:34), 33
Build Tools: 27.0.0, 27.0.1, 27.0.2, 27.0.3, 28.0.0, 28.0.1, 28.0.2, 28.0.3, 29.0.0, 29.0.1, 29.0.2, 29.0.3, 30.0.0, 30.0.1, 30.0.2, 30.0.3, 31.0.0, 32.0.0, 33.0.0, 33.0.1
Android NDK: 25.1.89[37](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:38)[39](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:40)3
IDEs:
Nano: 4.8 - /usr/bin/nano
Vim: 8.1 - /usr/bin/vim
Languages:
Bash: 5.0.17 - /usr/bin/bash
Go: 1.17.13 - /usr/bin/go
Erlang: 25.0.4 - /usr/bin/erl
Java: 11.0.11 - /opt/hostedtoolcache/Java_Adopt_jdk/11.0.11-9/x64/bin/javac
Perl: 5.30.0 - /usr/bin/perl
PHP: 8.2.1 - /usr/bin/php
Python: 3.8.10 - /usr/bin/python
Python3: 3.8.10 - /usr/bin/python3
R: 4.2.2 - /usr/bin/R
Ruby: 2.7.0 - /usr/bin/ruby
Rust: 1.67.0 - /home/runner/.cargo/bin/rustc
Databases:
MongoDB: 5.0.14 - /usr/bin/mongo
MySQL: 0.20.04.2 - /usr/bin/mysql
SQLite: 3.31.1 - /usr/bin/sqlite3
Browsers:
Chrome: 109.0.5[41](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:42)4.119
Chromium: 109.0.[54](https://github.com/littlecinemadigital/backstage/actions/runs/4115321414/jobs/7103877260#step:4:55)14.0
Firefox: 109.0.1
Monorepos:
Yarn Workspaces: 3.3.0
Additional context
I have attached as a .zip
archive the text file output of the run execution with MOON_LOG
set to trace. I did abort the run once it appeared to be stalled (stopped printing anything for a couple seconds). I’ll include the last 100 or so lines inline here as well.
2023-02-07T14:02:08.2972425Z ▪▪▪▪ messaging-api:gen-prisma-client (17s 398ms, f1d31d84)
2023-02-07T14:02:08.3003445Z [trace 14:02:08] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-prisma-client/lastRun.json
2023-02-07T14:02:08.3005076Z [debug 14:02:08] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/messaging-api to ~/work/backstage/backstage/.moon/cache/outputs/f1d31d84e50812c2f75599f8a3bf7ab5e279e79d56e32381526a98f28c1e2f31.tar.gz
2023-02-07T14:02:08.3006545Z [trace 14:02:08] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-prisma-client/stderr.log
2023-02-07T14:02:08.3007689Z [trace 14:02:08] moon:archive:tar Packing directory ~/work/backstage/backstage/packages/messaging-api/src/generated/prisma-client
2023-02-07T14:02:11.8393078Z [trace 14:02:11] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-prisma-client/stdout.log
2023-02-07T14:02:11.8400476Z [trace 14:02:11] moon:action-pipeline:batch:5:3 Processed action RunTarget(messaging-api:gen-prisma-client) in 21.027832638s
2023-02-07T14:02:16.4723674Z [trace 14:02:16] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/lambda-helpers/build/lastRun.json
2023-02-07T14:02:16.4724448Z ▪▪▪▪ lambda-helpers:build (25s 575ms, c821d0c9)
2023-02-07T14:02:16.4725916Z [debug 14:02:16] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/lambda-helpers to ~/work/backstage/backstage/.moon/cache/outputs/c821d0c9096306f990d1f3ceb884c278d4dc5f7a32953bd80517670fa55c452b.tar.gz
2023-02-07T14:02:16.4727079Z [trace 14:02:16] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/lambda-helpers/build/stderr.log
2023-02-07T14:02:16.4748597Z [trace 14:02:16] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/lambda-helpers/build/stdout.log
2023-02-07T14:02:16.4749623Z [trace 14:02:16] moon:archive:tar Packing directory ~/work/backstage/backstage/packages/lambda-helpers/dist
2023-02-07T14:02:16.4780746Z [trace 14:02:16] moon:action-pipeline:batch:5:1 Processed action RunTarget(lambda-helpers:build) in 25.667651435s
2023-02-07T14:02:17.2004196Z [trace 14:02:17] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/analytics-api/build/lastRun.json
2023-02-07T14:02:17.2004995Z ▪▪▪▪ analytics-api:build (26s 312ms, ab59bfc4)
2023-02-07T14:02:17.2006402Z [debug 14:02:17] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/analytics-api to ~/work/backstage/backstage/.moon/cache/outputs/ab59bfc4cdb544cf6c605debe83dacfb9b21589eed88833090a2ab62a2ef2dd5.tar.gz
2023-02-07T14:02:17.2007637Z [trace 14:02:17] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/analytics-api/build/stderr.log
2023-02-07T14:02:17.2008579Z [trace 14:02:17] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/analytics-api/build/stdout.log
2023-02-07T14:02:17.2009485Z [trace 14:02:17] moon:archive:tar Packing directory ~/work/backstage/backstage/packages/analytics-api/dist
2023-02-07T14:02:17.2043516Z [trace 14:02:17] moon:action-pipeline:batch:5:2 Processed action RunTarget(analytics-api:build) in 26.393082622s
2023-02-07T14:02:18.6283300Z ▪▪▪▪ asset-management-api:build (27s 737ms, 9e2cd33f)
2023-02-07T14:02:18.6284675Z [trace 14:02:18] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/asset-management-api/build/lastRun.json
2023-02-07T14:02:18.6286179Z [debug 14:02:18] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/asset-management-api to ~/work/backstage/backstage/.moon/cache/outputs/9e2cd33f484bdcd2786a47f06c6a6f1c551b6c70bcdc00a8c0e8571c79eafc77.tar.gz
2023-02-07T14:02:18.6287465Z [trace 14:02:18] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/asset-management-api/build/stderr.log
2023-02-07T14:02:18.6288515Z [trace 14:02:18] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/asset-management-api/build/stdout.log
2023-02-07T14:02:18.6289502Z [trace 14:02:18] moon:archive:tar Packing directory ~/work/backstage/backstage/packages/asset-management-api/dist
2023-02-07T14:02:18.6320371Z [trace 14:02:18] moon:action-pipeline:batch:5:5 Processed action RunTarget(asset-management-api:build) in 27.81867154s
2023-02-07T14:02:18.6321553Z [trace 14:02:18] moon:action-pipeline:batch:6 Running 3 actions in batch 6
2023-02-07T14:02:18.6322972Z [trace 14:02:18] moon:action-pipeline:batch:6:1 Processing action RunTarget(messaging-api:gen-admin-schema)
2023-02-07T14:02:18.6324058Z [trace 14:02:18] moon:cache:item Cache miss for ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-admin-schema/lastRun.json, does not exist
2023-02-07T14:02:18.6325066Z [debug 14:02:18] moon:action:run-target Running target messaging-api:gen-admin-schema
2023-02-07T14:02:18.6329264Z [trace 14:02:18] moon:action-pipeline:batch:6:2 Processing action SyncNodeProject(root)
2023-02-07T14:02:18.6330157Z [trace 14:02:18] moon:action-pipeline:batch:6:3 Processing action RunTarget(messaging-api:gen-attendee-schema)
2023-02-07T14:02:18.6331226Z [trace 14:02:18] moon:utils:process Running command git ls-files --cached --modified --others --full-name --exclude-standard packages/messaging-api (in ~/work/backstage/backstage)
2023-02-07T14:02:18.6332310Z [trace 14:02:18] moon:cache:item Cache miss for ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-attendee-schema/lastRun.json, does not exist
2023-02-07T14:02:18.6333121Z [debug 14:02:18] moon:action:run-target Running target messaging-api:gen-attendee-schema
2023-02-07T14:02:18.6425993Z [trace 14:02:18] moon:utils:process Running command git hash-object --stdin-paths - (truncated files list) (in ~/work/backstage/backstage)
2023-02-07T14:02:18.6501498Z [trace 14:02:18] moon:utils:process Running command git status --porcelain --untracked-files -z (in ~/work/backstage/backstage)
2023-02-07T14:02:18.6700535Z [debug 14:02:18] moon:runner Generated hash 83a2046b03241112e100a519ab6c2457f843d11b5c70fb728c3c7c8b29005322 for target messaging-api:gen-admin-schema
2023-02-07T14:02:18.6701689Z [trace 14:02:18] moon:cache Writing hash manifest ~/work/backstage/backstage/.moon/cache/hashes/83a2046b03241112e100a519ab6c2457f843d11b5c70fb728c3c7c8b29005322.json
2023-02-07T14:02:18.6702801Z [debug 14:02:18] moon:runner Cache miss for hash 83a2046b03241112e100a519ab6c2457f843d11b5c70fb728c3c7c8b29005322, continuing run
2023-02-07T14:02:18.6729505Z [debug 14:02:18] moon:action:sync-project Syncing project root
2023-02-07T14:02:18.6730068Z ▪▪▪▪ messaging-api:gen-attendee-schema
2023-02-07T14:02:18.6730694Z [trace 14:02:18] moon:lang:config Syncing ~/work/backstage/backstage/tsconfig.json with changes
2023-02-07T14:02:18.6731488Z [trace 14:02:18] moon:action-pipeline:batch:6:2 Processed action SyncNodeProject(root) in 37.708484ms
2023-02-07T14:02:18.6732429Z [debug 14:02:18] moon:runner Generated hash c047a55b47e9c04c43472d93e5b33ad1a5713b0bba90bc3b02c4c0770cb13bf9 for target messaging-api:gen-attendee-schema
2023-02-07T14:02:18.6733311Z [trace 14:02:18] moon:cache Writing hash manifest ~/work/backstage/backstage/.moon/cache/hashes/c047a55b47e9c04c43472d93e5b33ad1a5713b0bba90bc3b02c4c0770cb13bf9.json
2023-02-07T14:02:18.6734526Z [debug 14:02:18] moon:runner Cache miss for hash c047a55b47e9c04c43472d93e5b33ad1a5713b0bba90bc3b02c4c0770cb13bf9, continuing run
2023-02-07T14:02:18.6735599Z [debug 14:02:18] moon:runner Creating messaging-api:gen-attendee-schema command (in working directory ~/work/backstage/backstage/packages/messaging-api)
2023-02-07T14:02:18.6736601Z [trace 14:02:18] moon:cache:runfile Writing runfile ~/work/backstage/backstage/.moon/cache/states/messaging-api/runfile.json
2023-02-07T14:02:18.6742063Z [trace 14:02:18] moon:utils:process Running command ~/.moon/tools/node/16.19.0/node --title messaging-api:gen-attendee-schema ../../node_modules/ts-node/dist/bin.js --transpile-only src/index.ts --attendee-schema (in ~/work/backstage/backstage/packages/messaging-api)
2023-02-07T14:02:18.6751551Z MOON_PROJECT_SOURCE=packages/messaging-api
2023-02-07T14:02:18.6752079Z MOON_TARGET=messaging-api:gen-attendee-schema
2023-02-07T14:02:18.6752421Z NODE_ENV=development
2023-02-07T14:02:18.6752812Z MOON_WORKSPACE_ROOT=/home/runner/work/backstage/backstage
2023-02-07T14:02:18.6753729Z MOON_PROJECT_ROOT=/home/runner/work/backstage/backstage/packages/messaging-api
2023-02-07T14:02:18.6754373Z MOON_WORKING_DIR=/home/runner/work/backstage/backstage
2023-02-07T14:02:18.6754756Z MOON_TOOLCHAIN_DIR=/home/runner/.moon
2023-02-07T14:02:18.6755187Z MOON_CACHE_DIR=/home/runner/work/backstage/backstage/.moon/cache
2023-02-07T14:02:18.6756248Z MOON_PROJECT_RUNFILE=/home/runner/work/backstage/backstage/.moon/cache/states/messaging-api/runfile.json
2023-02-07T14:02:18.6756632Z MOON_PROJECT_ID=messaging-api
2023-02-07T14:02:18.6761367Z [debug 14:02:18] moon:runner Creating messaging-api:gen-admin-schema command (in working directory ~/work/backstage/backstage/packages/messaging-api)
2023-02-07T14:02:18.6761982Z ▪▪▪▪ messaging-api:gen-admin-schema
2023-02-07T14:02:18.6762999Z [trace 14:02:18] moon:cache:runfile Writing runfile ~/work/backstage/backstage/.moon/cache/states/messaging-api/runfile.json
2023-02-07T14:02:18.6764440Z [trace 14:02:18] moon:utils:process Running command ~/.moon/tools/node/16.19.0/node --title messaging-api:gen-admin-schema ../../node_modules/ts-node/dist/bin.js --transpile-only src/index.ts --admin-schema (in ~/work/backstage/backstage/packages/messaging-api)
2023-02-07T14:02:18.6765644Z MOON_PROJECT_SOURCE=packages/messaging-api
2023-02-07T14:02:18.6766152Z MOON_TARGET=messaging-api:gen-admin-schema
2023-02-07T14:02:18.6766501Z NODE_ENV=development
2023-02-07T14:02:18.6766905Z MOON_WORKSPACE_ROOT=/home/runner/work/backstage/backstage
2023-02-07T14:02:18.6767490Z MOON_PROJECT_ROOT=/home/runner/work/backstage/backstage/packages/messaging-api
2023-02-07T14:02:18.6767970Z MOON_WORKING_DIR=/home/runner/work/backstage/backstage
2023-02-07T14:02:18.6768364Z MOON_TOOLCHAIN_DIR=/home/runner/.moon
2023-02-07T14:02:18.6768786Z MOON_CACHE_DIR=/home/runner/work/backstage/backstage/.moon/cache
2023-02-07T14:02:18.6769439Z MOON_PROJECT_RUNFILE=/home/runner/work/backstage/backstage/.moon/cache/states/messaging-api/runfile.json
2023-02-07T14:02:18.6769943Z MOON_PROJECT_ID=messaging-api
2023-02-07T14:02:27.2586487Z [messaging-api:gen-attendee-schema] Generated Artifacts:
2023-02-07T14:02:27.2587874Z [messaging-api:gen-attendee-schema] TypeScript Types ==> /home/runner/work/backstage/backstage/packages/messaging-api/src/schema/../generated/attendee-nexus.ts
2023-02-07T14:02:27.2589263Z [messaging-api:gen-attendee-schema] GraphQL Schema ==> /home/runner/work/backstage/backstage/packages/messaging-api/src/schema/../../attendee-schema.graphql
2023-02-07T14:02:27.2787643Z [trace 14:02:27] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-attendee-schema/lastRun.json
2023-02-07T14:02:27.2788484Z ▪▪▪▪ messaging-api:gen-attendee-schema (8s 605ms, c047a55b)
2023-02-07T14:02:27.2790119Z [debug 14:02:27] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/messaging-api to ~/work/backstage/backstage/.moon/cache/outputs/c047a55b47e9c04c43472d93e5b33ad1a5713b0bba90bc3b02c4c0770cb13bf9.tar.gz
2023-02-07T14:02:27.2791471Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/packages/messaging-api/src/generated/attendee-nexus.ts
2023-02-07T14:02:27.2792649Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-attendee-schema/stderr.log
2023-02-07T14:02:27.2793775Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-attendee-schema/stdout.log
2023-02-07T14:02:27.2794773Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/packages/messaging-api/attendee-schema.graphql
2023-02-07T14:02:27.2795864Z [trace 14:02:27] moon:action-pipeline:batch:6:3 Processed action RunTarget(messaging-api:gen-attendee-schema) in 8.645823868s
2023-02-07T14:02:27.5227881Z [messaging-api:gen-admin-schema] Generated Artifacts:
2023-02-07T14:02:27.5229221Z [messaging-api:gen-admin-schema] TypeScript Types ==> /home/runner/work/backstage/backstage/packages/messaging-api/src/schema/../generated/admin-nexus.ts
2023-02-07T14:02:27.5230823Z [messaging-api:gen-admin-schema] GraphQL Schema ==> /home/runner/work/backstage/backstage/packages/messaging-api/src/schema/../../admin-schema.graphql
2023-02-07T14:02:27.5409903Z ▪▪▪▪ messaging-api:gen-admin-schema (8s 864ms, 83a2046b)
2023-02-07T14:02:27.5422623Z [trace 14:02:27] moon:cache:item Writing cache ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-admin-schema/lastRun.json
2023-02-07T14:02:27.5423969Z [debug 14:02:27] moon:archive:tar Packing tar archive from ~/work/backstage/backstage/packages/messaging-api to ~/work/backstage/backstage/.moon/cache/outputs/83a2046b03241112e100a519ab6c2457f843d11b5c70fb728c3c7c8b29005322.tar.gz
2023-02-07T14:02:27.5425025Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/packages/messaging-api/admin-schema.graphql
2023-02-07T14:02:27.5425927Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-admin-schema/stdout.log
2023-02-07T14:02:27.5426828Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/packages/messaging-api/src/generated/admin-nexus.ts
2023-02-07T14:02:27.5427715Z [trace 14:02:27] moon:archive:tar Packing file ~/work/backstage/backstage/.moon/cache/states/messaging-api/gen-admin-schema/stderr.log
2023-02-07T14:02:27.5428591Z [trace 14:02:27] moon:action-pipeline:batch:6:1 Processed action RunTarget(messaging-api:gen-admin-schema) in 8.909929172s
2023-02-07T14:02:27.5429299Z [trace 14:02:27] moon:action-pipeline:batch:7 Running 1 actions in batch 7
2023-02-07T14:02:27.5442941Z [trace 14:02:27] moon:action-pipeline:batch:7:1 Processing action RunTarget(root:gen-gql-types)
2023-02-07T14:02:27.5443837Z [trace 14:02:27] moon:cache:item Cache miss for ~/work/backstage/backstage/.moon/cache/states/root/gen-gql-types/lastRun.json, does not exist
2023-02-07T14:02:27.5444564Z [debug 14:02:27] moon:action:run-target Running target root:gen-gql-types
2023-02-07T14:02:27.5447488Z [trace 14:02:27] moon:utils:process Running command git hash-object --stdin-paths - codegen.yml packages/messaging-api/admin-schema.graphql packages/messaging-api/attendee-schema.graphql (in ~/work/backstage/backstage)
2023-02-07T14:02:27.5468928Z [trace 14:02:27] moon:utils:process Running command git ls-files --cached --modified --others --full-name --exclude-standard . (in ~/work/backstage/backstage)
2023-02-07T14:02:27.5738133Z [trace 14:02:27] moon:utils:process Running command git hash-object --stdin-paths - (truncated files list) (in ~/work/backstage/backstage)
2023-02-07T14:04:27.5328732Z ##[error]The operation was canceled.
Happy to provide any other information I can. It looks to me like the last command is to run git hash-object
on too many files but I don’t know enough about the internals that changed or how I could define my tasks differently to avoid it.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 24
This is exciting! I had a hunch that
git ls-files
was grabbing everything because it was a root project, so moving root to the glob pattern seemed like the best approach.Ok 0.24, try it and let me know what happens. Pass
MOON_DEBUG_PROCESS_INPUT=1
to see all the hashed files also.