Memory usage problem #71
Description
Hi,
I am having issues with neo4j using too much memory as a result of goodhound execution, I have not been able to get goodhound to finish running at all because of this, even with a low '-r' parameter.
I have set the max heap size to 28GB, even then it fails with the error below, I cannot tell what specific query is causing this. Can the script or query be optimized in anyway to avoid this problem?
DEBUG:Adding connection pool for profile ConnectionProfile('bolt://localhost:7687')
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:[#0000] C: (Dialing <localhost:7687>)
DEBUG:[#EB39] S: (Accepted)
DEBUG:[#EB39] C: <BOLT>
DEBUG:[#EB39] C: <PROTOCOL> 4.3~4.0 | 4.0 | 3.0 | 2.0
DEBUG:[#EB39] S: <PROTOCOL> 4.3
DEBUG:[#EB39] C: HELLO {'user_agent': 'py2neo/2021.2.3 Python/3.11.1-final-0 (win32)', 'scheme': 'basic', 'principal': 'neo4j', 'credentials': '*******'}
DEBUG:[#EB39] C: (Sent 133 bytes)
DEBUG:[#EB39] S: SUCCESS {'server': 'Neo4j/4.4.12', 'connection_id': 'bolt-138', 'hints': {}}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> does not belong to pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
Warming up database
DEBUG:Attempting to acquire read-write connection to default database
DEBUG:Using connection pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:[#EB39] C: RUN 'MATCH (n) OPTIONAL MATCH (n)-[r]->() RETURN count(n.name) + count(r.isacl)' {} {}
DEBUG:[#EB39] C: PULL {'n': -1, 'qid': -1}
DEBUG:[#EB39] C: (Sent 99 bytes)
DEBUG:[#EB39] S: SUCCESS {'t_first': 15, 'fields': ['count(n.name) + count(r.isacl)']}
DEBUG:[#EB39] S: RECORD * 1
DEBUG:[#EB39] S: SUCCESS {'bookmark': 'FB:', 'type': 'r', 't_last': 31939, 'db': 'neo4j'}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
INFO:Setting cost.
DEBUG:Attempting to acquire read-write connection to default database
DEBUG:Using connection pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:[#EB39] C: RUN 'MATCH (n)-[r:MemberOf]->(m:Group) SET r.cost = 0' {} {}
DEBUG:[#EB39] C: PULL {'n': -1, 'qid': -1}
DEBUG:[#EB39] C: (Sent 73 bytes)
DEBUG:[#EB39] S: SUCCESS {'t_first': 12547, 'fields': []}
DEBUG:[#EB39] S: FAILURE {'code': 'Ne
o.TransientError.General.OutOfMemoryError', 'message': "There is not enough memory to perform the current task. Please try increasing 'dbms.memory.heap.max_size' in the neo4j configuration (normally in 'conf/neo4j.conf' or, if you are using Neo4j Desktop, found through the user interface) or if you are running an embedded installation increase the heap by using '-Xmx' command line flag, and then restart the database."}
DEBUG:[#EB39] C: RESET
DEBUG:[#EB39] C: (Sent 6 bytes)
DEBUG:[#EB39] S: SUCCESS {}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
WARNING:Error setting cost!