Error deleting target in MOP.

[rachel3834]

I tried to delete this target from MOP and the process timed out.
After reloading the target list page I realised the target was still in the database, so I tried to load the target detail page for it which produced the following log output.
Note that the messages with ALL CAPS are checkpoint output from custom MOP views.

I'd appreciate feedback on the ShareLock, ExclusiveLock errors we seem to be running into during this query @phycodurus @jchate6 .
This seems to be a Postgres issue.

mop-5465fcd69c-whhh4 nginx 10.100.18.199 - - [28/Aug/2025:18:10:36 +0000] "POST /targets/55284/delete/ HTTP/1.1" 499 0 "https://mop.lco.global/targets/55284/delete/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "207.71.224.112"
mop-5465fcd69c-gtxx4 mop 127.0.0.1 - - [28/Aug/2025:18:10:39 +0000] "GET /targets/ HTTP/1.1" 200 29451 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
mop-5465fcd69c-gtxx4 nginx 10.100.21.85 - - [28/Aug/2025:18:10:39 +0000] "GET /targets/ HTTP/1.1" 200 29451 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "207.71.224.112"
mop-5465fcd69c-gtxx4 mop STARTING TargetDetail page load: 2025-08-28 18:10:43.342355
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 1, time taken 0:00:00.040834
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 2, time taken 0:00:00.000462
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 3, time taken 0:00:00.000181
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: get took 0:00:00.041477
mop-5465fcd69c-gtxx4 mop STARTED GET_CONTEXT 2025-08-28 18:10:43.392631
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop queryTools: Retrieved associated data for 1 Targets
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop queryTools: Time taken: 0:00:00.000386
mop-5465fcd69c-gtxx4 mop queryTools: collating data on microlensing event set
mop-5465fcd69c-gtxx4 mop queryTools: collated data for target ZTF18acvinru, 0 out of 1
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop queryTools: Collated data for 1 targets in 0:00:00.043933
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop GET_CONTEXT took 0:00:00.060432
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop FINISHED GET_CONTEXT 2025-08-28 18:10:43.523032
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-gtxx4 mop MULENS TARGET DATA time taken: 0:00:00.000294
mop-5465fcd69c-gtxx4 mop CLASS FORM started at 2025-08-28 18:10:43.535978
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.77MiB
mop-5465fcd69c-whhh4 nginx 10.100.18.199 - - [28/Aug/2025:18:11:31 +0000] "GET /accounts/login/ HTTP/1.1" 499 0 "https://mop.lco.global/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0" "98.185.248.56"
mop-5465fcd69c-gtxx4 nginx 10.100.18.199 - - [28/Aug/2025:18:11:31 +0000] "GET /accounts/login/ HTTP/1.1" 200 5568 "https://mop.lco.global/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0" "98.185.248.56"
mop-5465fcd69c-gtxx4 mop 127.0.0.1 - - [28/Aug/2025:18:11:31 +0000] "GET /accounts/login/ HTTP/1.1" 200 5568 "https://mop.lco.global/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0"
mop-5465fcd69c-whhh4 mop Total removed orphan object permissions instances: 1524
mop-5465fcd69c-whhh4 mop Internal Server Error: /targets/55284/delete/
mop-5465fcd69c-whhh4 mop Traceback (most recent call last):
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
mop-5465fcd69c-whhh4 mop     return self.cursor.execute(sql, params)
mop-5465fcd69c-whhh4 mop psycopg2.errors.DeadlockDetected: deadlock detected
mop-5465fcd69c-whhh4 mop DETAIL:  Process 15741 waits for ShareLock on transaction 3247735538; blocked by process 18386.
mop-5465fcd69c-whhh4 mop Process 18386 waits for ExclusiveLock on tuple (7,7) of relation 86721612 of database 19613; blocked by process 17373.
mop-5465fcd69c-whhh4 mop Process 17373 waits for ShareLock on transaction 3247733895; blocked by process 15741.
mop-5465fcd69c-whhh4 mop HINT:  See server log for query details.
mop-5465fcd69c-whhh4 mop CONTEXT:  while deleting tuple (7,65) in relation "tom_targets_basetarget"
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop The above exception was the direct cause of the following exception:
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop Traceback (most recent call last):
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/silk/sql.py", line 89, in execute_sql
mop-5465fcd69c-whhh4 mop     return self._execute_sql(*args, **kwargs)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
mop-5465fcd69c-whhh4 mop     cursor.execute(sql, params)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
mop-5465fcd69c-whhh4 mop     return self._execute_with_wrappers(
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
mop-5465fcd69c-whhh4 mop     return executor(sql, params, many, context)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
mop-5465fcd69c-whhh4 mop     with self.db.wrap_database_errors:
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
mop-5465fcd69c-whhh4 mop     raise dj_exc_value.with_traceback(traceback) from exc_value
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
mop-5465fcd69c-whhh4 mop     return self.cursor.execute(sql, params)
mop-5465fcd69c-whhh4 mop django.db.utils.OperationalError: deadlock detected
mop-5465fcd69c-whhh4 mop DETAIL:  Process 15741 waits for ShareLock on transaction 3247735538; blocked by process 18386.
mop-5465fcd69c-whhh4 mop Process 18386 waits for ExclusiveLock on tuple (7,7) of relation 86721612 of database 19613; blocked by process 17373.
mop-5465fcd69c-whhh4 mop Process 17373 waits for ShareLock on transaction 3247733895; blocked by process 15741.
mop-5465fcd69c-whhh4 mop HINT:  See server log for query details.
mop-5465fcd69c-whhh4 mop CONTEXT:  while deleting tuple (7,65) in relation "tom_targets_basetarget"
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop During handling of the above exception, another exception occurred:
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop Traceback (most recent call last):
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
mop-5465fcd69c-whhh4 mop     return self.cursor.execute(sql, params)
mop-5465fcd69c-whhh4 mop psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop The above exception was the direct cause of the following exception:
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-whhh4 mop Traceback (most recent call last):
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
mop-5465fcd69c-whhh4 mop     response = get_response(request)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 199, in _get_response
mop-5465fcd69c-whhh4 mop     response = self.process_exception_by_middleware(e, request)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 362, in process_exception_by_middleware
mop-5465fcd69c-whhh4 mop     response = middleware_method(request, exception)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/tom_common/middleware.py", line 29, in process_exception
mop-5465fcd69c-whhh4 mop     raise exception
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
mop-5465fcd69c-whhh4 mop     response = wrapped_callback(request, *callback_args, **callback_kwargs)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/base.py", line 104, in view
mop-5465fcd69c-whhh4 mop     return self.dispatch(request, *args, **kwargs)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
mop-5465fcd69c-whhh4 mop     return super().dispatch(request, *args, **kwargs)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/base.py", line 143, in dispatch
mop-5465fcd69c-whhh4 mop     return handler(request, *args, **kwargs)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/edit.py", line 278, in post
mop-5465fcd69c-whhh4 mop     return self.form_valid(form)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/edit.py", line 284, in form_valid
mop-5465fcd69c-whhh4 mop     self.object.delete()
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/models/base.py", line 1132, in delete
mop-5465fcd69c-whhh4 mop     return collector.delete()
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/models/deletion.py", line 506, in delete
mop-5465fcd69c-whhh4 mop     count = query.delete_batch(pk_list, self.using)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/models/sql/subqueries.py", line 42, in delete_batch
mop-5465fcd69c-whhh4 mop     num_deleted += self.do_query(
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/models/sql/subqueries.py", line 20, in do_query
mop-5465fcd69c-whhh4 mop     cursor = self.get_compiler(using).execute_sql(CURSOR)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/silk/sql.py", line 96, in execute_sql
mop-5465fcd69c-whhh4 mop     query_dict['analysis'] = _explain_query(self.connection, q, params)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/silk/sql.py", line 58, in _explain_query
mop-5465fcd69c-whhh4 mop     cur.execute(prefixed_query, params)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
mop-5465fcd69c-whhh4 mop     return self._execute_with_wrappers(
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
mop-5465fcd69c-whhh4 mop     return executor(sql, params, many, context)
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
mop-5465fcd69c-whhh4 mop     with self.db.wrap_database_errors:
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
mop-5465fcd69c-whhh4 mop     raise dj_exc_value.with_traceback(traceback) from exc_value
mop-5465fcd69c-whhh4 mop   File "/mop/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
mop-5465fcd69c-whhh4 mop     return self.cursor.execute(sql, params)
mop-5465fcd69c-whhh4 mop django.db.utils.InternalError: current transaction is aborted, commands ignored until end of transaction block
mop-5465fcd69c-whhh4 mop 
mop-5465fcd69c-gtxx4 nginx 10.100.20.210 - - [28/Aug/2025:18:11:43 +0000] "GET /targets/55284/ HTTP/1.1" 499 0 "https://mop.lco.global/targets/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "207.71.224.112"
mop-5465fcd69c-whhh4 nginx 10.100.20.210 - - [28/Aug/2025:18:12:43 +0000] "GET /targets/55284/ HTTP/1.1" 499 0 "https://mop.lco.global/targets/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "207.71.224.112"
mop-5465fcd69c-gtxx4 mop STARTING TargetDetail page load: 2025-08-28 18:12:43.349497
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.97MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 1, time taken 0:00:00.019577
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.97MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 2, time taken 0:00:00.000516
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.97MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: chk 3, time taken 0:00:00.000126
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.97MiB
mop-5465fcd69c-gtxx4 mop TARGETDETAIL: get took 0:00:00.020219
mop-5465fcd69c-gtxx4 mop STARTED GET_CONTEXT 2025-08-28 18:12:43.378011
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.97MiB
mop-5465fcd69c-gtxx4 mop queryTools: Retrieved associated data for 1 Targets
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop queryTools: Time taken: 0:00:00.000420
mop-5465fcd69c-gtxx4 mop queryTools: collating data on microlensing event set
mop-5465fcd69c-gtxx4 mop queryTools: collated data for target ZTF18acvinru, 0 out of 1
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop queryTools: Collated data for 1 targets in 0:00:00.045867
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop GET_CONTEXT took 0:00:00.308575
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop FINISHED GET_CONTEXT 2025-08-28 18:12:43.752338
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-gtxx4 mop MULENS TARGET DATA time taken: 0:00:00.000608
mop-5465fcd69c-gtxx4 mop CLASS FORM started at 2025-08-28 18:12:43.765888
mop-5465fcd69c-gtxx4 mop CHECKPOINT: N DB connections: 0, memory: 678.09MiB
mop-5465fcd69c-whhh4 mop [2025-08-28 18:13:23 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:7)
mop-5465fcd69c-whhh4 mop 127.0.0.1 - - [28/Aug/2025:18:13:23 +0000] "POST /targets/55284/delete/ HTTP/1.1" 500 145 "https://mop.lco.global/targets/55284/delete/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
mop-5465fcd69c-whhh4 mop [2025-08-28 18:13:23 +0000] [7] [INFO] Worker exiting (pid: 7)
mop-5465fcd69c-whhh4 mop [2025-08-28 18:13:24 +0000] [1] [ERROR] Worker (pid:7) was sent SIGKILL! Perhaps out of memory?
mop-5465fcd69c-whhh4 mop [2025-08-28 18:13:24 +0000] [28] [INFO] Booting worker with pid: 28
mop-5465fcd69c-whhh4 nginx 10.100.18.199 - - [28/Aug/2025:18:13:25 +0000] "POST /accounts/login/?next= HTTP/1.1" 499 0 "https://mop.lco.global/accounts/login/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0" "98.185.248.56"
mop-5465fcd69c-whhh4 mop registering new views: args: ('groups', <class 'tom_common.api_views.GroupViewSet'>, 'groups'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('targets', <class 'tom_targets.api_views.TargetViewSet'>, 'targets'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('targetextra', <class 'tom_targets.api_views.TargetExtraViewSet'>, 'targetextra'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('targetname', <class 'tom_targets.api_views.TargetNameViewSet'>, 'targetname'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('targetlist', <class 'tom_targets.api_views.TargetListViewSet'>, 'targetlist'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('observations', <class 'tom_observations.api_views.ObservationRecordViewSet'>, 'observations'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('dataproducts', <class 'tom_dataproducts.api_views.DataProductViewSet'>, 'dataproducts'), kwargs: {}
mop-5465fcd69c-whhh4 mop registering new views: args: ('reduceddatums', <class 'tom_dataproducts.api_views.ReducedDatumViewSet'>, 'reduceddatums'), kwargs: {}
mop-5465fcd69c-whhh4 mop 127.0.0.1 - - [28/Aug/2025:18:13:34 +0000] "GET /accounts/login/ HTTP/1.1" 200 5574 "https://mop.lco.global/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0"
mop-5465fcd69c-whhh4 mop STARTING TargetDetail page load: 2025-08-28 18:13:34.770768
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop TARGETDETAIL: chk 1, time taken 0:00:00.027418
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop TARGETDETAIL: chk 2, time taken 0:00:00.000411
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop TARGETDETAIL: chk 3, time taken 0:00:00.000180
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop TARGETDETAIL: get took 0:00:00.028009
mop-5465fcd69c-whhh4 mop STARTED GET_CONTEXT 2025-08-28 18:13:34.806746
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop queryTools: Retrieved associated data for 1 Targets
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop queryTools: Time taken: 0:00:00.000487
mop-5465fcd69c-whhh4 mop queryTools: collating data on microlensing event set
mop-5465fcd69c-whhh4 mop queryTools: collated data for target ZTF18acvinru, 0 out of 1
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop queryTools: Collated data for 1 targets in 0:00:00.067084
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop GET_CONTEXT took 0:00:00.081934
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop FINISHED GET_CONTEXT 2025-08-28 18:13:35.173499
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-whhh4 mop MULENS TARGET DATA time taken: 0:00:00.000416
mop-5465fcd69c-whhh4 mop CLASS FORM started at 2025-08-28 18:13:35.187090
mop-5465fcd69c-whhh4 mop CHECKPOINT: N DB connections: 0, memory: 661.88MiB
mop-5465fcd69c-gtxx4 nginx 10.100.18.199 - - [28/Aug/2025:18:13:38 +0000] "GET /favicon.ico HTTP/1.1" 499 0 "https://mop.lco.global/accounts/login/?next=" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0" "98.185.248.56"
mop-5465fcd69c-gtxx4 nginx 10.100.20.210 - - [28/Aug/2025:18:13:43 +0000] "GET /targets/55284/ HTTP/1.1" 499 0 "https://mop.lco.global/targets/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "207.71.224.112"

[phycodurus]

Yes, this is a database deadlock. I'm guessing this is an edge case when deleting a BaseTarget and the custom code target that goes with it. There may be some ORM/database operations that we need to make atomic. I think we could send this issue to TOMToolkit/tom_base

[jchate6]

@rachel3834 what was the name of the target?

[jchate6]

This looks like the delete did eventually work, it just took much longer than expected.
I just made a test object in the MOP DB and removed it. It took ~5 minutes for the delete to actually go through.

[rachel3834]

No I didn't touch it after I posted this issue, so that was just a very long DB operation. For the record the name of the target as
ZTF18acvinru but I confirm that it has finally gone.

It's not clear to me why this is an issue for MOP; we don't delete targets very often. So I wonder if it is an issue for other TOMs?

[Fingel]

Edit, nevermind. It looks like the problem is here:

mop-5465fcd69c-whhh4 mop Total removed orphan object permissions instances: 1524

This is a hook in the Target delete that is supposed to clean up orphaned row level permissions. I'd expect the first target to be deleted after merging TOMToolkit/tom_base#1253 to take some time. But that should only be the case the first run. Perhaps the transaction is failing and attempting to remove all these permissions every time?

I'd recommend manually doing these cleanup via a shell and see if that fixes the issue:

./manage.py clean_orphan_obj_perms

[rachel3834]

Thanks @Fingel. I tried this command and got the following output:

Total removed orphan object permissions instances: 0
Removed 0 object permission entries with no targets

I will try another delete when our runTAP management command isn't running

[Fingel]

Interesting. Is there something in MOP that creates/deletes a lot of database objects on a regular basis?

[rachel3834]

There are management processes that create a lot of objects but not many that delete anything. We rarely delete targets. The one thing we do delete on a regular basis is data products, because our pipeline removes the old lightcurve for a target and uploads the newer one.

…

On Thu, Sep 4, 2025 at 9:43 AM Austin Riba ***@***.***> wrote: *Fingel* left a comment (LCOGT/mop#241) <#241 (comment)> Interesting. Is there something in MOP that creates/deletes a lot of database objects on a regular basis? — Reply to this email directly, view it on GitHub <#241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPJA3AZQCJZMANJYHAYEOT3RBT2HAVCNFSM6AAAAACFCH65WSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENJUGU3DONZQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Fingel]

OK we might be getting somewhere here. Would it be possible to try the ./manage.py clean_orphan_obj_perms management command again after the pipeline runs that deletes DataProducts? I believe DataProducts do create row-level permissions, which would mean that the orphaned_obj_permissions script would be kept busy. If it takes a while and/or you get output about it deleting a bunch of orphaned objects. then I think we've found the issue.

We could potentially even get ahead of it by adding a call toclean_orphan_obj_perms at the end of your pipeline. It is imported from django_guardian:

from guardian.utils import clean_orphan_obj_perms

That way the cleanup script doesn't wait for a target to be deleted to run, it will run as a part of your regular pipeline.

[rachel3834]

I ran the management command again after first confirming that the pipeline had uploaded a new lightcurve, but I got the same output as before: ``` Total removed orphan object permissions instances: 0 Removed 0 object permission entries with no targets ``` It may be worth noting that MOP doesn't really use the permissions controls much - all users have access to all targets and data products. I assume that they are still being set though.

…

On Thu, Sep 4, 2025 at 9:56 AM Austin Riba ***@***.***> wrote: *Fingel* left a comment (LCOGT/mop#241) <#241 (comment)> OK we might be getting somewhere here. Would it be possible to try the ./manage.py clean_orphan_obj_perms management command again after the pipeline runs that deletes DataProducts? I believe DataProducts do create row-level permissions, which would mean that the orphaned_obj_permissions script would be kept busy. If it takes a while and/or you get output about it deleting a bunch of orphaned objects. then I think we've found the issue. We could potentially even get ahead of it by adding a call to clean_orphan_obj_perms at the end of your pipeline. It is imported from django_guardian: from guardian.utils import clean_orphan_obj_perms That way the cleanup script doesn't wait for a target to be deleted to run, it will run as a part of your regular pipeline. — Reply to this email directly, view it on GitHub <#241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPJA3AXZ7SRQDKJZUQ42NL3RBVM3AVCNFSM6AAAAACFCH65WSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENJUGYYDOMJZGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>