diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml index 625dbba..9cd0e0e 100644 --- a/.quickstart/quickstart.yml +++ b/.quickstart/quickstart.yml @@ -16,6 +16,15 @@ table_variables: github__using_requested_reviewer_history: - requested_reviewer_history + _fivetran_quickstart_issue_closed_history: + - issue_closed_history + _fivetran_quickstart_issue_comment: + - issue_comment + _fivetran_quickstart_issue_merged: + - issue_merged + _fivetran_quickstart_pull_request_review: + - pull_request_review + destination_configurations: databricks: dispatch: diff --git a/CHANGELOG.md b/CHANGELOG.md index 27d4134..39ba817 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,16 @@ +# dbt_github v1.3.0 + +[PR #76](https://github.com/fivetran/dbt_github/pull/76) includes the following updates: + +## Documentation +- Updates README with standardized Fivetran formatting + +## Under the Hood +- In the `.quickstart.yml` file: + - Adds `table_variables` for relevant sources to prevent missing sources from blocking downstream Quickstart models. + - Adds `supported_vars` for Quickstart UI customization, +- Adds the `partition_by_source_relation` macro to improve window function partition handling and prevent conflicts when the unioned data feature is disabled. + # dbt_github v1.2.0 [PR #75](https://github.com/fivetran/dbt_github/pull/75) includes the following updates: diff --git a/README.md b/README.md index 5c8133f..942c1fe 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -# GitHub dbt Package ([Docs](https://fivetran.github.io/dbt_github/)) + +# Github dbt Package

+This dbt package transforms data from Fivetran's Github connector into analytics-ready tables. + +## Resources + +- Number of materialized models¹: 34 +- Connector documentation + - [Github connector documentation](https://fivetran.com/docs/connectors/applications/github) + - [Github ERD](https://fivetran.com/docs/connectors/applications/github#schemainformation) +- dbt package documentation + - [GitHub repository](https://github.com/fivetran/dbt_github) + - [dbt Docs](https://fivetran.github.io/dbt_github/#!/overview) + - [DAG](https://fivetran.github.io/dbt_github/#!/overview?g_v=1) + - [Changelog](https://github.com/fivetran/dbt_github/blob/main/CHANGELOG.md) + ## What does this dbt package do? -- Produces modeled tables that leverage GitHub data from [Fivetran's connector](https://fivetran.com/docs/applications/github) in the format described by [this ERD](https://fivetran.com/docs/applications/github#schemainformation). -- Provides insight into GitHub issues and pull requests by enhancing these core objects with commonly used metrics. -- Produces metrics tables, which increase understanding of your team's velocity over time. Metrics are available on a daily, weekly, monthly, and quarterly level. -- Generates a comprehensive data dictionary of your source and modeled github data through the [dbt docs site](https://fivetran.github.io/dbt_github/). - - -The following table provides a detailed list of all tables materialized within this package by default. -> TIP: See more details about these tables in the package's [dbt docs site](https://fivetran.github.io/dbt_github/#!/overview?g_v=1&g_e=seeds). - -| **Table** | **Description** | -| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [github__issues](https://fivetran.github.io/dbt_github/#!/model/model.github.github__issues) | Each record represents a GitHub issue, enriched with data about its assignees, milestones, and time comparisons. | -| [github__pull_requests](https://fivetran.github.io/dbt_github/#!/model/model.github.github__pull_requests) | Each record represents a GitHub pull request, enriched with data about its repository, reviewers, and durations between review requests, merges and reviews. | -| [github__daily_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__daily_metrics) | Each record represents a single day and repository, enriched with metrics about PRs and issues that were created and closed during that period. | -| [github__weekly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__weekly_metrics) | Each record represents a single week and repository, enriched with metrics about PRs and issues that were created and closed during that period. | -| [github__monthly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__monthly_metrics) | Each record represents a single month and repository, enriched with metrics about PRs and issues that were created and closed during that period. | -| [github__quarterly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__quarterly_metrics) | Each record represents a single quarter and repository, enriched with metrics about PRs and issues that were created and closed during that period. | - -### Materialized Models -Each Quickstart transformation job run materializes 34 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`. - +This package enables you to analyze GitHub issues and pull requests, enhance core objects with commonly used metrics, and produce velocity metrics over time. It creates enriched models with metrics focused on issue and pull request tracking, team performance, and repository activity. -## How do I use the dbt package? -### Step 1: Prerequisites +### Output schema +Final output tables are generated in the following target schema: + +``` +._github +``` + +### Final output tables + +By default, this package materializes the following final tables: + +| Table | Description | +| :---- | :---- | +| [github__issues](https://fivetran.github.io/dbt_github/#!/model/model.github.github__issues) | Tracks all GitHub issues with creator information, labels, lifecycle metrics, and comment activity to monitor issue resolution times, contributor engagement, and project health.

**Example Analytics Questions:**| +| [github__pull_requests](https://fivetran.github.io/dbt_github/#!/model/model.github.github__pull_requests) | Provides comprehensive pull request data including reviewers, approval status, merge times, changed files, and review cycles to analyze code review efficiency and development velocity.

**Example Analytics Questions:**| +| [github__daily_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__daily_metrics) | Tracks daily repository activity including pull requests and issues created and closed to monitor development velocity and project health on a day-by-day basis.

**Example Analytics Questions:**| +| [github__weekly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__weekly_metrics) | Aggregates weekly repository activity to analyze sprint-level productivity, track week-over-week trends, and understand development patterns at the weekly cadence.

**Example Analytics Questions:**| +| [github__monthly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__monthly_metrics) | Summarizes monthly repository activity to track long-term development trends, measure team productivity over time, and identify seasonal patterns in contribution activity.

**Example Analytics Questions:**| +| [github__quarterly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__quarterly_metrics) | Provides quarterly repository performance metrics to support strategic planning, measure progress against OKRs, and understand high-level development trends by quarter.

**Example Analytics Questions:**| + +¹ Each Quickstart transformation job run materializes these models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`. + +--- + +## Prerequisites To use this dbt package, you must have the following: -- At least one Fivetran GitHub connection syncing data into your destination. +- At least one Fivetran Github connection syncing data into your destination. - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination. -#### Databricks Dispatch Configuration -If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively. -```yml -dispatch: - - macro_namespace: dbt_utils - search_order: ['spark_utils', 'dbt_utils'] -``` +## How do I use the dbt package? +You can either add this dbt package in the Fivetran dashboard or import it into your dbt project: + +- To add the package in the Fivetran dashboard, follow our [Quickstart guide](https://fivetran.com/docs/transformations/dbt). +- To add the package to your dbt project, follow the setup instructions in the dbt package's [README file](https://github.com/fivetran/dbt_github/blob/main/README.md#how-do-i-use-the-dbt-package) to use this package. -### Step 2: Install the package + + +### Install the package Include the following github package version in your `packages.yml` file. > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages. ```yaml packages: - package: fivetran/github - version: [">=1.2.0", "<1.3.0"] # we recommend using ranges to capture non-breaking changes automatically + version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically ``` > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/github_source` in your `packages.yml` since this package has been deprecated. -### Step 3: Define database and schema variables +#### Databricks Dispatch Configuration +If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively. +```yml +dispatch: + - macro_namespace: dbt_utils + search_order: ['spark_utils', 'dbt_utils'] +``` + +### Define database and schema variables #### Option A: Single connection By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `github` schema. If this is not where your GitHub data is (for example, if your github schema is named `github_fivetran`), add the following configuration to your root `dbt_project.yml` file: @@ -103,7 +129,7 @@ vars: By default, this package defines one single-connection source, called `github`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your GitHub sources, though the package will run successfully. To properly incorporate all of your GitHub connections into your project's DAG: -1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_github.yml` [file](https://github.com/fivetran/dbt_github/blob/main/models/staging/src_github.yml). +1. Define each of your sources in a `.yml` file in the `models` directory of your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_github.yml` [file](https://github.com/fivetran/dbt_github/blob/main/models/staging/src_github.yml). ```yml # a .yml file in your root project @@ -124,7 +150,7 @@ sources: tables: # copy and paste from github/models/staging/src_github.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once ``` -> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_github?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`. +> **Note**: If there are source tables you do not have (see [Disable models for non-existent sources](https://github.com/fivetran/dbt_github?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`. 2. Set the `has_defined_sources` variable (scoped to the `github` package) to `True`, like such: ```yml @@ -134,7 +160,7 @@ vars: has_defined_sources: true ``` -### Step 4: Disable models for non-existent sources +### Disable models for non-existent sources Your GitHub connection might not sync every table that this package expects. If your syncs exclude certain tables, it is because you either don't use that functionality in GitHub or have actively excluded some tables from your syncs. If you do not have the `TEAM`, `REPO_TEAM`, `ISSUE_ASSIGNEE`, `ISSUE_LABEL`, `LABEL`, or `REQUESTED_REVIEWER_HISTORY` tables synced and are not running the package via Fivetran Quickstart, add the following variables to your `dbt_project.yml` file: @@ -150,7 +176,7 @@ vars: *Note: This package only integrates the above variables. If you'd like to disable other models, please create an [issue](https://github.com/fivetran/dbt_github/issues) specifying which ones.* -### (Optional) Step 5: Additional configurations +### (Optional) Additional configurations
Expand/collapse configurations @@ -175,7 +201,7 @@ vars: ```
-### (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™ +### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
Expand for more details Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core setup guides](https://fivetran.com/docs/transformations/dbt#setupguide). @@ -197,15 +223,18 @@ packages: - package: dbt-labs/spark_utils version: [">=0.3.0", "<0.4.0"] ``` + ## How is this package maintained and can I contribute? ### Package Maintenance -The Fivetran team maintaining this package _only_ maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/github/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_github/blob/main/CHANGELOG.md) and release notes for more information on changes across versions. +The Fivetran team maintaining this package only maintains the [latest version](https://hub.getdbt.com/fivetran/github/latest/) of the package. We highly recommend you stay consistent with the latest version of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_github/blob/main/CHANGELOG.md) and release notes for more information on changes across versions. ### Contributions A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions. -We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package. +We highly encourage and welcome contributions to this package. Learn how to contribute to a package in dbt's [Contributing to an external dbt package article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657). + + ## Are there any resources available? - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_github/issues/new/choose) section to find the right avenue of support for you. -- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW). +- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW). \ No newline at end of file diff --git a/dbt_project.yml b/dbt_project.yml index 9d1c4e2..ff9164e 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -1,6 +1,6 @@ config-version: 2 name: 'github' -version: '1.2.0' +version: '1.3.0' require-dbt-version: [">=1.3.0", "<3.0.0"] models: github: diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 8188557..46082e0 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -1,5 +1,5 @@ name: 'github_integration_tests' -version: '1.2.0' +version: '1.3.0' config-version: 2 profile: 'integration_tests' vars: diff --git a/macros/union/partition_by_source_relation.sql b/macros/union/partition_by_source_relation.sql new file mode 100644 index 0000000..b233d91 --- /dev/null +++ b/macros/union/partition_by_source_relation.sql @@ -0,0 +1,15 @@ +{%- macro partition_by_source_relation(has_other_partitions='yes', alias=None) -%} + +{{ adapter.dispatch('partition_by_source_relation', 'github') () }} + +{%- endmacro %} + +{% macro default__partition_by_source_relation(has_other_partitions, alias) -%} + {% set prefix = '' if alias is none else alias ~ '.' %} + + {%- if has_other_partitions == 'no' -%} + {{ 'partition by ' ~ prefix ~ 'source_relation' if var('github_sources', [])|length > 1 }} + {%- else -%} + {{ ', ' ~ prefix ~ 'source_relation' if var('github_sources', [])|length > 1 }} + {%- endif -%} +{%- endmacro -%} \ No newline at end of file diff --git a/models/intermediate/int_github__issue_open_length.sql b/models/intermediate/int_github__issue_open_length.sql index 4b77a93..05dc526 100644 --- a/models/intermediate/int_github__issue_open_length.sql +++ b/models/intermediate/int_github__issue_open_length.sql @@ -29,7 +29,7 @@ close_events_with_timestamps as ( source_relation, issue_id, updated_at as valid_starting, - coalesce(lead(updated_at) over (partition by issue_id, source_relation order by updated_at), {{ dbt.current_timestamp() }}) as valid_until, + coalesce(lead(updated_at) over (partition by issue_id {{ github.partition_by_source_relation() }} order by updated_at), {{ dbt.current_timestamp() }}) as valid_until, is_closed from close_events_stacked )