IMPR: Deprecate and Remove the `~log` Table.

[dimitri-yatsenko]

Deprecate and Remove the ~log Table

Summary

We propose removing the per-schema ~log table feature from DataJoint. This client-side audit logging mechanism has become obsolete and will be deprecated with no backward compatibility path.

Background

The ~log table was designed to provide an audit trail of schema operations (table declarations, alterations, drops, and deletes) by storing event records directly in each schema's database. While well-intentioned, this approach no longer aligns with modern data management practices.

Rationale for Removal

1. Server-side logging is the modern standard

Database platforms now provide robust, built-in audit capabilities:

• MySQL Enterprise Audit, General Query Log, Binary Log
• PostgreSQL pgaudit
• Cloud-managed database audit logs (AWS RDS, GCP Cloud SQL, Azure)

These server-side solutions capture all database activity regardless of client, providing complete coverage that client-side logging cannot achieve.

2. Incomplete and bypassable

The ~log table only records operations performed through DataJoint. Any direct SQL access, other clients, or administrative tools bypass it entirely—creating a false sense of audit completeness.

3. Fragmented audit trail

Each schema maintains its own ~log table, making centralized audit analysis difficult and providing no cross-schema visibility.

4. Users already have better alternatives

Modern data teams typically rely on:

• Infrastructure observability platforms (Datadog, CloudWatch, Prometheus/Grafana)
• Centralized logging systems (ELK stack, Splunk, Loki)
• Database-native audit logs for compliance requirements

5. Maintenance burden

The log table grows indefinitely with no built-in rotation, cleanup, or retention policies. It adds storage overhead and schema clutter for minimal practical benefit.

6. Python logging module is sufficient

DataJoint already uses Python's standard logging module. We will ensure comprehensive logging coverage for all significant operations, allowing users to route logs to their preferred backends (files, syslog, cloud services, etc.) using standard Python logging configuration.

Migration Path

• The ~log table will be removed with no backward compatibility
• Existing ~log tables will remain in user databases but will no longer be written to or maintained by DataJoint
• Users relying on ~log for audit purposes should migrate to database-level audit logging or external logging infrastructure
• All operations previously logged to ~log will be emitted through Python's logging module at appropriate log levels

Affected Components

• datajoint.table.Log class
• schema.log property
• Internal _log() calls in table declaration, alteration, deletion, and drop operations
• ~log table auto-creation on schema instantiation