age: Unexpected termination on looping cypher query - could not find rte

Have encountered an intriguing issue with loading a table with a large number of rows into Apache AGE. The process is looping through PL/pgSQL. The looping terminates with the following message.

SQL Error [42703]: ERROR: could   not find rte for a01a724103fbb3d059b8387bf043dbc8     Where: PL/pgSQL function analysis.create_trips(text,text,text,text,text,text,integer,text,integer)   line 5 at EXECUTE

The database version is PostgreSQL 15.4 (Debian 15.4-2.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit. The Apache AGE version is 1.4.0 for PG15 built from the repository. The error results from the following process.

SELECT create_graph('transport_network');

CREATE OR REPLACE FUNCTION analysis.create_trips
(graph_name text,
service_key text,service_id text, trip_id text, trip_headsign text, route_id text, direction_id int, shape_id text, wheelchair_accessible int)
returns text
LANGUAGE plpgsql
VOLATILE
as $trips$
declare
nodename text := graph_name || '.' || 'trips';
BEGIN
execute
format ('select * from cypher(''%1$s'', $$match (v:routes {id: %6$s})
create(v)-[:USES]->
(t:trips
{service_key: %2$s, service_id: %3$s, id: %4$s, headsign: %5$s, route_id: %6$s, direction_id: %7$s, shape_id: %8$s,
wheelchair_accessible: %9$s})$$) as (t agtype);',
quote_ident(graph_name),
quote_ident(service_key),quote_ident(service_id),
quote_ident(trip_id),quote_ident(trip_headsign),
quote_ident(route_id),to_char(direction_id,'9'),quote_ident(shape_id),to_char(wheelchair_accessible,'9'));
return nodename;
END
$trips$
;

select create_vlabel('transport_network','trips');

do $$
declare temprow record;
graph_name text:='transport_network';
counter integer := 0 ;
begin
for temprow in select service_key, service_id, trip_id from allservices.trips
order by service_key,trip_id
loop
counter := counter+1; -- Prevent replication of row
perform
analysis.create_trips
(graph_name,
a.service_key, a.service_id,
a.trip_id, a.trip_headsign,
a.route_id, a.direction_id, a.shape_id,
a.wheelchair_accessible)
from
(select row_number() over (order by service_key,trip_id) as row_num,
service_key, service_id,
trip_id, trip_headsign,
route_id, direction_id, shape_id,
coalesce(wheelchair_accessible,0) as wheelchair_accessible from allservices.trips) a
where a.row_num=counter
;
end loop;
end; $$;

In this case the string after the rte in the error message refers to the service_key value in the table allservices.trips. The first instance of the record in this instance is.

row_num service_key service_id trip_id trip_headsign route_id direction_id shape_id wheelchair_accessible
7741 a01a724103fbb3d059b8387bf043dbc8 FR 307 Gunghalin Pl X1 0 1002 1

The attributes of the table being processed are:

  • Total size with indexes: 60MB
  • Number of rows 231,131

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Reactions: 1
  • Comments: 32 (20 by maintainers)

Most upvoted comments

@I33Buckler The pg_dump requires the table definitions (creates) in order to load them. Did you miss adding some definition files by chance? Or, I could be doing it incorrectly,…

Also, could you also give access to @rafsun42

@I33Buckler I’m also thinking it might be best if we try to reproduce this locally. It will save a lot of back and forth. How can we create the dataset that you have? I can provide you with the tables and the query at a shared link. You should receive an email inviting access to a folder.

@I33Buckler Error messages can be misleading at times. Although, in this particular case, I’m not sure if any specific error message would have been able to help. The lack of quotes on that parameter seemed to change the cypher command structure making it something that was technically legal but didn’t make sense when transformed.

We will keep it in mind, though, and try to see if there might be a way to better address these.

@I33Buckler The pg_dump requires the table definitions (creates) in order to load them. Did you miss adding some definition files by chance? Or, I could be doing it incorrectly,…

Also, could you also give access to @rafsun42

Thanks. Had the wrong configuration.

@I33Buckler As part of debugging indexes, I’ve created graphs with over 1 million rows, in the same graph, with the same label, and containing properties - using pgsql functions. So, I don’t think this is a limit based on the number of rows. Additionally, if it were, it would likely land on an obvious base 2 number. But, I could be wrong.

We need to break this down by removing items that might be extraneous. What happens if you reduce, by 1, the number of parameters, and thus the number of properties, to the create_trips function? Try removing each parameter, while keeping the rest, to see if a specific parameter may be an issue?

Further investigation has determined that the process continues until failing on row 123,487 of the 231,131 rows in the table. Would be interested to know what could be the cause of this hard limit on processing the table as there is no obvious reason for it. The row number in the previous comment was a result not sorting the table being processed to match the indexing.