Source properties

Related documentation

• Using sources
• Declaring resource properties

Overview

Source properties can be declared in .yml files in your models/ directory (as defined by the source-paths config).

You can name these files whatever_you_want.yml, and nest them arbitrarily deeply in subfolders within the models/ directory.

models/<filename>.yml

version: 2

sources:
  - name: <string> # required
    description: <markdown_string>
    database: <database_name>
    schema: <schema_name>
    loader: <string>
    loaded_at_field: <column_name>
    meta: {<dictionary>}
    tags: [<string>]

    overrides: <string>

    freshness:
      warn_after:
        count: <positive_integer>
        period: minute | hour | day
      error_after:
        count: <positive_integer>
        period: minute | hour | day
      filter: <where-condition>

    quoting:
      database: true | false
      schema: true | false
      identifier: true | false

    tables:
      - name: <string> #required
        description: <markdown_string>
        meta: {<dictionary>}
        identifier: <table_name>
        loaded_at_field: <column_name>
        tests:
          - <test>
          - ... # declare additional tests
        tags: [<string>]
        freshness:
          warn_after:
            count: <positive_integer>
            period: minute | hour | day
          error_after:
            count: <positive_integer>
            period: minute | hour | day
          filter: <where-condition>

        quoting:
          database: true | false
          schema: true | false
          identifier: true | false
        external: {<dictionary>}
        columns:
          - name: <column_name> # required
            description: <markdown_string>
            meta: {<dictionary>}
            quote: true | false
            tests:
              - <test>
              - ... # declare additional tests
            tags: [<string>]
          - name: ... # declare properties of additional columns

      - name: ... # declare properties of additional source tables

  - name: ... # declare properties of additional sources

Example

models/<filename>.yml

version: 2

sources:
  - name: jaffle_shop
    database: raw
    schema: public
    loader: emr # informational only (free text)
    loaded_at_field: _loaded_at # configure for all sources

    # meta fields are rendered in auto-generated documentation
    meta:
      contains_pii: true
      owner: "@alice"

    # Add tags to this source
    tags:
      - ecom
      - pii

    quoting:
      database: false
      schema: false
      identifier: false

    tables:
      - name: orders
        identifier: Orders_
        loaded_at_field: updated_at # override source defaults
        columns:
          - name: id
            tests:
              - unique

          - name: price_in_usd
            tests:
              - not_null

      - name: customers
        quoting:
          identifier: true # override source defaults
        columns:
            tests:
              - unique