quickstart: Stellar Quickstart not working

What version are you using?

“horizon_version”: “horizon-v2.28.2-(built-from-source)”, “core_version”: “v20.1.0”, “ingest_latest_ledger”: 0, “history_latest_ledger”: 0, “history_latest_ledger_closed_at”: “0001-01-01T00:00:00Z”, “history_elder_ledger”: 0, “core_latest_ledger”: 50367925, “network_passphrase”: “Public Global Stellar Network ; September 2015”, “current_protocol_version”: 19, “supported_protocol_version”: 20, “core_supported_protocol_version”: 20

What did you do?

• Fresh new VM on Hetzner (8 Cores / 16GB RAM)
• Debian 11
• Installed Docked
• Installed stellar/quickstart using:
• Started docker container: docker run --rm -it -p "5432:5432" -p "8000:8000" -p "6060:6060" -p "6061:6061" -p "11625:11625" -p "11626:11626" -v "/opt/stellar:/opt/stellar" --name stellar stellar/quickstart --pubnet --> Killed it after it created all necessary folders
• Changed only one flag in /opt/stellar/horizon/etc/horizon.env --> export PER_HOUR_RATE_LIMIT=0
• Restarted docker container: docker run --rm -d -p "5432:5432" -p "8000:8000" -p "6060:6060" -p "6061:6061" -p "11625:11625" -p "11626:11626" -v "/opt/stellar:/opt/stellar" --name stellar stellar/quickstart --pubnet
• When inspecting Horizon logs ingestion breaks at one point and start over an over again:

time="2024-02-14T13:10:49.835Z" level=info msg="Processing ledger entry changes" pid=26610 processed_entries=11650000 progress="55.79%" sequence=50367743 service=ingest source=historyArchive time="2024-02-14T13:10:50.926Z" level=info msg="Processing ledger entry changes" pid=26610 processed_entries=11700000 progress="55.87%" sequence=50367743 service=ingest source=historyArchive time="2024-02-14T13:10:51.398Z" level=info msg="Processing ledger entry changes" pid=26610 processed_entries=11750000 progress="55.95%" sequence=50367743 service=ingest source=historyArchive time="2024-02-14T13:10:53.036Z" level=info msg="Processing ledger entry changes" pid=26610 processed_entries=11800000 progress="56.03%" sequence=50367743 service=ingest source=historyArchive time="2024-02-14T13:10:53.568Z" level=info msg="Processing ledger entry changes" pid=26610 processed_entries=11850000 progress="56.11%" sequence=50367743 service=ingest source=historyArchive time="2024-02-14T13:10:57.363Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.552537392 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:44946" method=GET path=/ pid=26610 referer=undefined req=779a6c970550/nEUgsAq4kx-000110 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:10:59.628Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:10:59.759Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:10:59.850Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:00.018Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:05.212Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:05.362Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:05.486Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:05.593Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:10.717Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:10.988Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:11.142Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:11.261Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:16.378Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:16.504Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:16.568Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:16.597Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:21.675Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:21.684Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:21.721Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:21.749Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:23.136Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.726657983 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:44102" method=GET path=/ pid=26610 referer=undefined req=779a6c970550/nEUgsAq4kx-000111 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:11:26.840Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:26.885Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:26.983Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:27.129Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:32.256Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:32.417Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:32.518Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:32.546Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 time="2024-02-14T13:11:34.663Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.082609845 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:34774" method=GET path=/ pid=26610 referer=undefined req=779a6c970550/nEUgsAq4kx-000112 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:11:41.154Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 time="2024-02-14T13:11:41.290Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=26610 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:41.386Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=26610 time="2024-02-14T13:11:41.442Z" level=warning msg="error ticking app: context deadline exceeded" pid=26610 starting horizon... 2024/02/14 13:11:49 DEPRECATED - The usage of the flag --enable-captive-core-ingestion has been deprecated. Horizon now uses Captive-Core ingestion by default and this flag will soon be removed in the future. 2024/02/14 13:11:49 Checking DB migrations... 2024/02/14 13:11:49 Preparing captive core... time="2024-02-14T13:11:49.441Z" level=info msg="Initializing horizon..." pid=29345 time="2024-02-14T13:11:51.611Z" level=info msg="Initializing database..." pid=29345 time="2024-02-14T13:11:51.611Z" level=info msg="Establishing database session for history" pid=29345 time="2024-02-14T13:11:51.619Z" level=info msg="Establishing database session for ingest" pid=29345 time="2024-02-14T13:11:51.672Z" level=info msg="Starting horizon on :8001 (ingest: true)" pid=29345 time="2024-02-14T13:11:51.672Z" level=info msg="Starting internal server on :6060" pid=29345 time="2024-02-14T13:11:51.673Z" level=info msg="Ingestion system initial state" current_state=start pid=29345 service=ingest time="2024-02-14T13:11:51.674Z" level=info msg="Starting to serve requests" pid=29345 time="2024-02-14T13:11:51.674Z" level=info msg="Starting internal server on :6060" pid=29345 time="2024-02-14T13:11:52.456Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000421683 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:49248" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000001 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:11:52.917Z" level=info msg="Ingestion system state machine transition" current_state=start next_state="buildFromCheckpoint(checkpointLedger=50367871, skipChecks=false)" pid=29345 service=ingest time="2024-02-14T13:11:52.917Z" level=info msg="Preparing range" from=50367871 pid=29345 service=ingest time="2024-02-14T13:11:53.310Z" level=info msg="default: Config from /opt/stellar/horizon/captive-core/captive-core/stellar-core.conf" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="default: Generated QUORUM_SET: {" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 5," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"Blockdaemon_Validator_1\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"Blockdaemon_Validator_2\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"Blockdaemon_Validator_3\"" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [ \"sdf_3\", \"sdf_1\", \"sdf_2\" ]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [ \"coinqvest_finland\", \"coinqvest_hong_kong\", \"coinqvest_germany\" ]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [ \"satoshipay_iowa\", \"satoshipay_singapore\", \"satoshipay_frankfurt\" ]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [ \"FT_SCV_1\", \"FT_SCV_3\", \"FT_SCV_2\" ]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 3," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"lobstr_5_australia\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"lobstr_4_asia\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"lobstr_1_europe\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"lobstr_2_europe\"," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"lobstr_3_north_america\"" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="{" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"t\" : 2," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="\"v\" : [ \"hercules\", \"lyra\", \"bootes\" ]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="}" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="default: Assigning calculated value of 2 to FAILURE_SAFETY" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.311Z" level=info msg="default: Starting stellar-core v20.1.0" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.312Z" level=info msg="Database: Connecting to: sqlite3:///opt/stellar/horizon/captive-core/stellar.db" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.313Z" level=info msg="Ledger: Streaming metadata to file descriptor 3" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.314Z" level=info msg="SCP: LocalNode::LocalNode@ME qSet: b4cccf" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.314Z" level=info msg="Database: DB schema is in current version" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.314Z" level=info msg="default: Dropping accounts" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.314Z" level=info msg="default: Dropping trustlines" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.314Z" level=info msg="default: Dropping accountdata" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping claimablebalances" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping liquiditypools" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping contractdata" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping contractcode" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping configsettings" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.315Z" level=info msg="default: Dropping ttl" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Bucket: BucketListDB enabled: pageSizeExponent: 12 indexCutOff: 20MB, persist indexes: true" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: Connection effective settings:" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: TARGET_PEER_CONNECTIONS: 8" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: MAX_ADDITIONAL_PEER_CONNECTIONS: 64" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: MAX_PENDING_CONNECTIONS: 500" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: MAX_OUTBOUND_PENDING_CONNECTIONS: 56" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="default: MAX_INBOUND_PENDING_CONNECTIONS: 444" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'Blockdaemon_Validator_1' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'Blockdaemon_Validator_2' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'Blockdaemon_Validator_3' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'FT_SCV_1' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'FT_SCV_2' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'FT_SCV_3' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'bootes' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'coinqvest_finland' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'coinqvest_germany' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'coinqvest_hong_kong' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'hercules' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lobstr_1_europe' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lobstr_2_europe' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lobstr_3_north_america' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lobstr_4_asia' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lobstr_5_australia' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'lyra' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'satoshipay_frankfurt' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'satoshipay_iowa' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'satoshipay_singapore' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'sdf_1' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'sdf_2' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="History: Archive 'sdf_3' has 'get' command only, will not be written" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=warning msg="History: No writable archives configured, history will not be written." pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Ledger: Last closed ledger (LCL) hash is ac3c9a0799978670cf4fbe7b03a3df7b36b56be109a1256b093eff6ffe7ff9bb" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Ledger: Loaded LCL header from database: [seq=50367764, hash=ac3c9a]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Ledger: Starting up application" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Ledger: Last closed ledger (LCL) hash is ac3c9a0799978670cf4fbe7b03a3df7b36b56be109a1256b093eff6ffe7ff9bb" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:53.317Z" level=info msg="Ledger: Loaded LCL header from database: [seq=50367764, hash=ac3c9a]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.267Z" level=info msg="Bucket: Restarted merge on BucketList level 1" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.268Z" level=info msg="Ledger: Assumed bucket-state for LCL: [seq=50367764, hash=ac3c9a]" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.268Z" level=info msg="Overlay: Binding to endpoint 0.0.0.0:11725" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.268Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.278Z" level=info msg="Bucket: BucketIndex::createIndex() indexing individual keys in bucket buckets/tmp/bucket-3f50def9bce2b0f5/tmp-bucket-595e13435b62b07f.xdr" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.458Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:54.544Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:55.328Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:55.656Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:55.883Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:56.639Z" level=info msg="Overlay: Connected to 51.222.248.104:11625" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:56.803Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:56.834Z" level=info msg="Overlay: Connected to 65.109.98.186:11625" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:56.889Z" level=info msg="Overlay: Connected to 31.135.8.163:11625" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:56.949Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:57.455Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:57.485Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000242396 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:38520" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000002 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:11:57.883Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.392Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.442Z" level=warning msg="could not load stellar-core info: http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=29345 time="2024-02-14T13:11:58.442Z" level=error msg="failed to load the stellar-core info" err="http request errored: Get \"http://localhost:11626/info\": context deadline exceeded" pid=29345 stack="[main.go:43 client.go:139 app.go:233 app.go:453 asm_amd64.s:1598]" time="2024-02-14T13:11:58.442Z" level=error msg="could not start repeatable read transaction for txsub tick" error="canceling statement due to lack of response within timeout period" pid=29345 time="2024-02-14T13:11:58.442Z" level=warning msg="error ticking app: context deadline exceeded" pid=29345 time="2024-02-14T13:11:58.811Z" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.901Z" level=warning msg="Process: process 29451 exited 22: curl -sf https://stellar-archive-1-lobstr.s3.amazonaws.com/.well-known/stellar-history.json -o buckets/tmp/history-56c36f98ccaee16c/4ef95fe7d4fd4cb8-stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.901Z" level=error msg="History: Could not download file: archive lobstr_1_europe maybe missing file .well-known/stellar-history.json" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.901Z" level=error msg="History: Could not download .well-known/stellar-history.json file: corrupt archive lobstr_1_europe" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:11:58.901Z" level=error msg="History: Failed to obtain archive information: lobstr_1_europe" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:12:02.511Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000206759 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:38530" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000003 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:07.543Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000221426 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:55584" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000004 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:12.571Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.00038935 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:55600" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000005 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:17.610Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000269346 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:50862" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000006 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:22.639Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000365506 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:50870" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000007 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:27.679Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=4106 client_name=undefined client_version=undefined duration=0.000234791 host="localhost:8001" ip=127.0.0.1 ip_port="127.0.0.1:48854" method=GET path=/ pid=29345 referer=undefined req=779a6c970550/e2ePctZ0Op-000008 route=/ status=200 streaming=false x_forwarder_for= time="2024-02-14T13:12:29.268Z" level=warning msg="Herder: Lost track of consensus" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:12:29.268Z" level=warning msg="Herder: Out of sync context: {" pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:12:29.268Z" level=info msg="\"queue\" : null," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:12:29.268Z" level=info msg="\"scp\" : null," pid=29345 service=ingest subservice=stellar-core time="2024-02-14T13:12:29.268Z" level=info msg="\"you\" : \"ME\"" pid=29345 service=ingest subservice=stellar-core

What did you expect to see?

What did you see instead?