0.55.5

Merged PRs

doltgresql

• 2395: information schema sequences
  Adds sequences to information_schema, following the specification outlined by Postgresql here.
• 2394: Implemented the binary wire format
  Fixes:
  • #2317
  • #2347
  • #2348
  • #2360
  • #2362
    In a nutshell, we did not respect a client's wishes when they requested that the results for some columns are returned in their binary format (obtained from their send function). Instead, we had hardcoded that some types, in certain circumstances, would only return their binary format, while others would strictly return their text format. This PR implements the binary wire format for every type that is currently supported. This also fixes numerous bugs that were hidden due to our incorrect behavior.
    In addition to the above fixes, this PR introduces a new test harness for analyzing our compatibility with the wire messages returned directly to Postgres clients. Thousands of lines of new tests have been added that ensure we're returning the exact correct results for a given set of messages (not just queries but the actual client messages), and the test harness allows for quick iteration when comparing what a Postgres server sends versus a Doltgres server, so we can ensure that we're returning the same bytes, not just the same overall structure.
• 2187: #1648: fix VALUES clause type inference

  Summary

  VALUES clause type inference now uses PostgreSQL's common type resolution algorithm instead of using only the first row's type.
  existing issue: SELECT * FROM (VALUES(1),(2.01),(3)) v(n) failed with integer: unhandled type: decimal.Decimal

  Changes

  • Add ResolveValuesTypes analyzer rule that:
  • Collects types from all VALUES rows (not just the first)
  • Uses FindCommonType() to resolve per PostgreSQL's [UNION/CASE type resolution](https://www.postgresql.org/docs/15/typeconv-union-case.html; per FindCommonType doc comment)
  • Applies implicit casts to convert values to the common type
  • Updates GetField expressions in parent nodes (handles aggregates like SUM)
  • Add UnknownCoercion expression for unknown to target type coercion
  • add go and bats tests

  Questions for Reviewers

  1. Analyzer rule approach: Is adding this as an analyzer rule (after TypeSanitizer) the right approach? I considered alternatives but this seemed cleanest for handling the two-pass transformation needed for GetField updates. Open to feedback if there's a better pattern.
  2. PostgreSQL version: The code references PostgreSQL 15 docs. Should this stay as-is since doltgresql targets PG 15, or should it use /docs/current/?
     Fixes: #1648

Closed Issues

• 2362: COUNT(*) returns wrong value via binary protocol (extended query)
• 2360: Extended protocol sends JSONB as raw JSON text instead of binary JSONB format (missing 0x01 version prefix)
• 2348: Extended protocol sends incorrect binary format for ENUM columns, causing client-side decode panic
• 2347: Extended protocol sends empty binary payload for NUMERIC/DECIMAL columns, causing client panic
• 2317: Extended protocol SELECT fails with 'permission denied for table pg_type'
• 1648: the first value in VALUES should not determine the column type

0.55.4

Merged PRs

doltgresql

• 2353: allow cte in CREATE VIEW
  Also fixes:
  • string_agg with separator cast with text type
  • fix parse procedure and sequence name in GRANT privileges ON statement
• 2350: add plpgsql-dynexecute statement for CREATE FUNCTION
  Also fixes:
  • type resolution in DECLARE
  • add support for TG_OP special variable usage for triggers
• 2349: Checking for table relation respects temporary tables
  Fixes #2326
  When checking for whether a root has a relation, we should respect temporary tables in addition to our existing checks for regular tables.
• 2346: Added test framework for testing wire data
  This is a precursor for:
  • #2317
    Currently, we have no in-engine way of ensuring that the proper data is received on the wire for any given sequence of messages that were sent. This means that we must rely on client tests to ensure that the correct data is sent, with the expectation that they'll fail if we change something. This isn't conducive to our standard debugging workflow, so this adds the ability to perform wire tests and verification from within the Go testing environment, which is resistant to change (client tests could fail due to a library update that is outside of our control) and easily debuggable.
    For most of these tests, I'll be using WireShark to ensure that our output is exactly the same as a standard Postgres server.

Closed Issues

• 2326: Cannot create keyless temporary tables with serial columns

0.55.3

Merged PRs

doltgresql

• 2327: bugfix: support records containing non-DoltgresType values
• 2324: resolve type with empty search path

Closed Issues

0.55.2

Merged PRs

doltgresql

• 2323: Fixed issues 2316 and 2318
  This fixes:
  • #2316
  • #2318
    For 2316, clients may only send a single format code that should apply to multiple values, but we assumed that their lengths always matched. This is fixed by extending the format code slice length to the same as the value slice if there is a single format code provided.
    For 2318, it appears that Rust wants these values in binary format, so now we're returning them in binary format. I'm not sure why Rust doesn't accept text format like everything else does, but perhaps it's the default in Postgres so some ORMs only work with the default.
    It's worth mentioning that tests for these exist only within the client tests, as I could not replicate the errors within our standard testing framework. As mentioned with the binary format, it seems that Rust isn't as permissive as other ORMs, so we have to test with Rust specifically.

Closed Issues

• 2318: Extended protocol sends TIMESTAMPTZ and DATE as text instead of binary format
• 2316: Extended protocol INSERT/UPDATE/DELETE crashes with 'index out of range [1] with length 1'

0.55.1

Merged PRs

doltgresql

• 2308: Fixed issues 2299, 2301, and 2307
  This fixes:
  • #2299
  • #2307
    For 2299, we were only resolving the column type, and not the types within default values, so we ended up with an unresolved type that was persisted in the schema. Since we assume that every deserialized type is resolved, this would cause a panic when fields were missing.
    For 2307, we return the schema from the unanalyzed node obtained right after parsing the node. Our analysis passes handle type conversions between GMS types (with EXISTS being a node defined in GMS), and therefore we were returning a GMS type (which was being automatically converted to the wrong type). We now attempt an analysis pass before determining the schema, which should work in most cases, but may fail for some bindings. In those cases, we ignore the analysis error and return the original schema. This isn't a complete solution, but it at least gets us further in all other cases.
    This most likely fixes:
  • #2301
    2301 seems related to 2299. We don't appear to have any intrinsic problem with many CREATE TABLE statements within Rust's sqlx library (as used in the issue filed, 30 statements work fine), so it appears tied to the other issue. It's possible that there are further issues masked by the aforementioned one, but that cannot be determined without the user's migration data.
• 2291: add nightly ORM test

Closed Issues

• 2301: Panic when executing large CREATE TABLE migration via extended query protocol (SQLx)
• 2307: SELECT EXISTS(...) returns INT2 instead of BOOL, breaking typed PostgreSQL clients
• 2299: Panic: nil pointer dereference when using DEFAULT on ENUM column
• 2206: Support for LOWER() function (ERROR: at or near "(": syntax error)

0.55.0

Merged PRs

doltgresql

• 2289: Table types are handled by all relevant ALTER TABLE statements
  This adds the remaining ALTER TABLE statements that have special hooks for table types.
• 2272: Correctly support foreign key constraint violation checks during merge
  This PR is mostly tests. See dolthub/dolt#10418 for most of the logic.
• 2270: add information_schema.constraint_column_usage view
  Depends on dolthub/go-mysql-server#3416
  • adds information_schema.constraint_column_usage system view as table
  • fix parsing schema table with double quotes in functions
  • allow int to be casted in regtype as int4
• 2268: sqlAlchemy integration tests
  Integration tests for sqlAlchemy.
  Pretty similar to our standard dolt sqlAlchemy tests, with a couple of differences. I used the newer version of sqlAlchemy (2.0.46) and had to make some setup changes using features that were introduced in 2.x builds.
  psycopg2 seems to convert row results into python lists as well so the expected results had to be changed. I couldn't find confirmation but it seems like that would be the expected behavior (from what claude found).
• 2254: Issue #2197 Part 2
  This fixes the other issues brought up in:
  • #2197
    This also adds support for column access, which is used by the UPDATE statement when a new column is added.
• 2235: Fix windows binary name in build script
  I found that starting from version 0.54.4 (after #2061), binaries for Windows are built with empty names. This PR should fix that.
• 2231: Add client tests for psycopg2 python client
  Adds regression tests using psycopg2 for fix from #2216
• 2227: Fixed issue 2206
  This is a fix for:
  • #2206
    I've added a skipped test to verify that the syntax error portion is fixed, however we don't yet support expression index attributes, so the test overall fails. However, the issue only covers the syntax portion which we're addressing here.
• 2226: Bump Dolt version
  I accidentally merged #2196 while it contained a replace directive pointing to a fork of Dolt.
  This PR contains the actual dolt version bump.
• 2223: put pg_catalog implicitly on the search_path when it isn't there already
  pg_catalog is always implicitly the first element on the search_path, unless it's explicitly included elsewhere:
  https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-CATALOG
  We weren't implementing this behavior, causing us to fail to resolve ::regclass expressions in some cases with a "no such relation" error for all pg_catalog tables.
  This fix isn't complete, because the same logic should also apply to all table lookups. i.e. users should be able to create a table named pg_class, and what the name pg_class resolves to depends on whether the pg_catalog schema is explicitly present in the search_path and where it is relative to the user's other schemas. That's a pretty obscure bug and is left unaddressed for now.
  Fixes #2217
• 2219: additional tests for merging foreign keys and constraint violations
• 2216: Bug fix: don't send RowDescription message with each spooled batch of rows
  Ensure that RowDescription messages are sent only once, at the beginning of query results. Sending multiple times, at the start of each batch of spooled rows, breaks some clients, such as psycopg2.
  Note that this wasn't causing a problem in the other clients we use for testing. To properly regression test this, we should follow up with adding a test suite for psycopg2.
  Related to #2143
• 2198: Support for UDFs with RETURNS TABLE
• 2196: dolthub/dolt#10227: add dolt_status_ignored dolt system table

  Summary

  Add adapter and test expectations for the new dolt_status_ignored system table from dolthub/dolt#10227.
  This table provides the same information as dolt_status plus an additional ignored boolean column.

  Changes

  • Add DoltgresDoltStatusIgnoredTableAdapter in server/tables/dtables/status_ignored.go
  • Register adapter in server/tables/dtables/init.go
  • Add new table to test expectations in:
  • testing/go/pgcatalog_test.go
  • testing/go/sequences_test.go

  Dependencies

  • Requires dolthub/dolt#10227

Closed Issues

• 2197: Bugs with table composite types
• 1465: SQLAlchemy unable to fetch type information
• 2143: psycopg2 support
• 2217: filter with 'pg_class'::regclass produces relation "pg_class" does not exist error when Eval is called during simplifyFilters
• 2175: Error: only a single statement at a time is currently supported (transaction support?)
• 2174: Error: DISTINCT ON is not yet supported

0.54.10

Merged PRs

doltgresql

• 2199: Support multiple statements
  This allows for handling multiple statements within a single Query message. This fixes:
  • #2175
    This also tests the contents of the messages returned, which were verified using Wireshark on a local Postgres instance as the source-of-truth.
• 2190: Initial support for UDFs with RETURNS SETOF
  Adds support for user-defined functions that use RETURNS SETOF to return values of a composite type.
  Support for UDFs that use RETURNS TABLE to return an anonymous composite type will be added as a follow up.

Closed Issues

• 2174: Error: DISTINCT ON is not yet supported

0.54.9

Merged PRs

doltgresql

• 2181: Added DISTINCT ON handling
  This adds support for:
  • #2174
    Related PRs:
  • dolthub/vitess#446
  • dolthub/go-mysql-server#3369

Closed Issues

0.54.8

Merged PRs

doltgresql

• 2172: drop schema support
  Fixes #2159
• 2158: Added bit and bit varying types
  Fixes #2149
• 2152: Bug fix: allow bpchar(N) casting
  Fixes: #2145
• 2151: Correctly support empty messages
  Fixes #2147
• 2139: #1863: validate database exists before CREATE SCHEMA

  Summary

  Adds a ValidateCreateSchema analyzer rule to ensure CREATE SCHEMA fails that appropriately when executed against a non-existent or invalid database context.
  • Adds analyzer rule that checks if the current database has a valid root before allowing CREATE SCHEMA to proceed
  • Ensures PostgreSQL-compliant behavior where schemas must be created within an existing database
  • Adds integration and bats tests

  Note

  I wasn't 100% sure if adding an analyzer rule was the right approach here, but it seemed like the cleanest way to validate database context before schema creation. The rule follows the same pattern as existing validation rules like ValidateCreateTable.
  The connection-level validation (rejecting connections to non-existent databases) already handles most cases, but this analyzer rule acts as an additional safety net for edge cases where a query might run with an invalid database context (also not sure how realistic this is 😕 ).
  Open to feedback on whether there's a better approach!

  Testing

  • Integrationt: TestCreateSchemaWithNonExistentDatabase
  • Bats: bats --filter "non-existent database|CREATE SCHEMA works" testing/bats/doltgres.bats
  • Existing schema tests pass: go test ./testing/go/... -run TestSchemas -v
    Closes: #1863
• 2138: #1361: unskip dolt_clean zero-argument tests

  Summary

  • Unskips the SELECT DOLT_CLEAN() zero-argument smoke tests that were added in #1373
  • I believe the underlying issue was resolved in #1763 which added Function0 registration for all Dolt procedures
  • Reorders test assertions so DOLT_CLEAN('t1') runs before DOLT_CLEAN() to properly test both
    Note: #1373 also added skipped tests in prepared_statement_test.go, but those were already unskipped in commit 62569c7.

  Tests to verify

  • Run go test ./testing/go/... -run "TestDoltClean" to verify existing dolt_clean tests pass
  • Run go test ./testing/go/... -run "TestDoltFunctionSmokeTests/smoke_test_select_dolt_clean" to verify unskipped smoke tests pass
    Fixes #1361
• 2137: #1868: remove outdated skip list entries for OR index tests

  Summary

  Remove outdated skip list entries for tests that were previously panicking due to an OR condition index lookup bug. The underlying issue was fixed in PR #2123.

  What was the problem?

  Issue #1868 reported a panic when executing queries like:

  SELECT * FROM test WHERE pk1 = 1 OR i = 1;

  where the first condition matches an index (primary key) and the second doesn't.

  What fixed it?

  I believe PR #2123 fixed this issue. In the PR it implemented a LogicTreeWalker for DoltgreSQL that properly handles doltgres-specific expression nodes when analyzing filters. This allows the query optimizer to correctly handle OR expressions with mixed indexed/non-indexed columns.

  Changes in this PR

  Removed 3 entries from the engine test skip list:
  • Complex Filter Index Scan #2
  • Complex Filter Index Scan #3
  • complicated range tree
    All three tests now pass.

  Testing

  • Verified all three previously-skipped tests now pass
  • Verified the exact scenario from issue #1868 works without panicking
    Fixes #1868
• 2136: #1057 allow unquoted STATUS keyword in DESCRIBE statements

  IMPORTANT

  Below is the summary of the cause of the bug in #1057 and the simple fix, but I want to clarify this is the desired fix.

  Summary

  • Fixes DESCRIBE dolt.status failing with "at or near 'status': syntax error"
  • Adds STATUS to the simple_ident grammar rule in sql.y, following the pattern established for PUBLIC in PR #828
  • Enables the previously skipped test for this functionality

  Background

  The simple_ident rule is used for parsing table name components in DESCRIBE statements. It only allowed IDENT tokens and the explicitly added PUBLIC keyword. Since STATUS is an unreserved keyword (not a regular identifier), the parser rejected it unless quoted.
  Before: DESCRIBE dolt.status fails, DESCRIBE dolt."status" works
  After: Both work

  Test Plan

  DESCRIBE dolt.status test now passes (TestUserSpaceDoltTables/dolt_status)
  Fixes #1057
• 2126: Add tests for SHOW TABLES bug
  Adds a test for a bug in SHOW TABLES behavior that caused sequences to be included in returned results.
  Depends on: dolthub/dolt#10220
  Fixes: #1743

Closed Issues

• 2149: bit type is not supported
• 2159: Please complete the implementation for dropping a schema in Doltgres
• 2099: [RFC] Exploring feasibility of Dolt or Doltgres as backend fo Matrix homeserver
• 1743: SHOW TABLES should not show sequences
• 2145: PostgreSQL compat: select '1'::bpchar(1) fails
• 2147: PostgreSQL compatibility: status field for empty query should be EMPTY
• 1359: Error adding a column with a check constraint to table with data
• 1361: dolt_clean procedure does not work if no arguments provided
• 1863: Handle PostgreSQL CREATE SCHEMA error when no database is selected
• 1057: describe dolt.status doesn't work
• 1868: Panic on SELECT ... WHERE ... OR
• 2127: JSONB SELECT Panic for large objects ~2KB

0.54.7

Merged PRs

doltgresql

• 2132: Fixed JSONB extended value error
  Fixes the issue:
  • #2127
• 2128: Added some hooks for DROP TABLE
  This adds support for ensuring that dropping a table while using table types doesn't outright put the database in an unusable state. There are many more hooks that need to be implemented across the board, but these are relatively straightforward ones for a common case of dropping a table, function, or procedure that uses them.
  This also adds support for DROP PROCEDURE, which I apparently overlooked when adding procedure support in general. Lastly, this adds support for the serialization and deserialization of values within columns that use table types. While it was previously possible to create tables that use table types, it wasn't possible to read from those tables (and writing to them just wrote useless data that couldn't be removed).
  You'll notice that error messages involving procedures return "function" instead of "procedure". Postgres calls them functions in their error messages, so we're replicating that.
• 2125: Changed configuration file for function passing
  In a nutshell, our internal repository for handling Hosted Doltgres only needs to include the configuration file, however recent changes added more packages that were referenced from our servercfg package. As our internal repository wasn't configured to handle these additional packages (which essentially requires building the entirety of Doltgres), it broke the existing workflow. To workaround this, I've moved the "details" of the configuration to its own package (named cfgdetails), and that will be referenced by our internal repository. Elsewhere in Doltgres, we still refer to servercfg, which has a wrapper struct around our details.
  The wrapper is needed as we cannot store a field that contains the overrides on the struct itself. Our YAML deserialization function does not honor ignored fields (which are supposed to work using yaml:"-") and is therefore panicking. Instead, the wrapper declares its own Overrides() function that overrides the function within cfgdetails.
• 2123: Bug fix: ensure OR expressions on a single table use an index when possible
  This fixes the performance regression in select_random_ranges, which uses BETWEEN .. OR BETWEEN .. in its filters.
  Requires dolthub/go-mysql-server#3347

Closed Issues

• 2097: auth.db, unable to configure path for "authFileName" file
• 2100: DOLTGRES_DB environment variable isn't respected