{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "article-000",
      "metadata": {},
      "source": [
        "# Spark SQL That Reads Like PySpark\n",
        "\n",
        "In Microsoft Fabric Runtime 2.0, we bring Spark SQL pipe syntax to developers working with Apache Spark 4.1. It lets you write SQL as a sequence of transformations: start with data, filter it, add a column, aggregate, and keep going.\n",
        "\n",
        "If you already think in PySpark DataFrame chains, this should feel familiar. Instead of working backward from a final `SELECT` or adding another subquery every time you need a new stage, you can write each step in the order you want to reason about it.\n",
        "\n",
        "**The benefit is easier query composition and debugging, not an automatic performance boost.** Pipe SQL still uses Spark's query planner and execution engine.\n",
        "\n",
        "In this tutorial, we will use eight sales rows to go from a simple filter to a regional revenue analysis. You will see conventional Spark SQL and PySpark alongside pipe SQL, inspect expected results, and run an equivalence check.\n",
        "\n",
        "\n",
        "**What is new, and where to run it**\n",
        "\n",
        "Pipe syntax was introduced in Apache Spark 4.0 and is available in Spark 4.1, the Spark version used by Fabric Runtime 2.0.\n",
        "\n",
        "\n",
        "Run these examples in a **Fabric Spark notebook**, not the Lakehouse SQL analytics endpoint or a Warehouse T-SQL editor. No pipe-specific configuration flag is needed.\n",
        "\n",
        "\n",
        "## What you will learn\n",
        "\n",
        "1. [Set up a self-contained notebook](#1-set-up-a-runtime-20-notebook).\n",
        "2. [Think in transformations](#2-the-mental-model-each-step-receives-a-table).\n",
        "3. [Filter, project, and sort](#3-filter-project-and-sort).\n",
        "4. [Add, replace, and remove columns](#4-add-replace-and-remove-columns).\n",
        "5. [Aggregate and filter groups](#5-aggregate-then-filter-without-having).\n",
        "6. [Join and rank results](#6-join-and-rank-a-complete-analysis).\n",
        "7. [Combine pipes with existing SQL and PySpark](#7-use-pipes-with-existing-sql-and-pyspark).\n",
        "8. [Try the translation exercise](#try-it-yourself).\n",
        "\n",
        "Then explore the optional [validation and execution plans](#optional-validate-results-and-inspect-plans), [additional syntax tips](#optional-two-more-useful-patterns), and [troubleshooting guide](#debugging-and-common-mistakes).\n",
        "\n",
        "## Download and import the companion notebook\n",
        "\n",
        "[Download the Pipe Syntax tutorial notebook (.ipynb)](https://arshadali.dev/assets/downloads/fabric-pipe-syntax/pipe-syntax-runtime-2-0.ipynb)\n",
        "\n",
        "The notebook contains the article's executable examples in order, explanations, expected-result tables, and validation cells. It has no saved execution outputs, credentials, workspace bindings, or attached Lakehouse. You can also follow the tutorial by copying the cells below.\n",
        "\n",
        "1. **Download** the notebook. If your browser displays JSON instead, save the linked file with the `.ipynb` extension, not `.txt`.\n",
        "2. In a **development Fabric workspace**, use the workspace toolbar's **Import notebook** option and select the downloaded file. Fabric imports standard Jupyter `.ipynb` files as notebook items.\n",
        "3. Open the imported notebook and confirm its default language is **PySpark**. SQL cells use `%%sql` on their first line.\n",
        "4. Select a published **Runtime 2.0 Environment**, as described below. No Lakehouse attachment or additional packages are needed.\n",
        "5. Review the code, start a **fresh session**, and run cells from top to bottom. **Run all** also runs the optional checks. The temporary views remain available for experimentation until the session ends.\n",
        "6. Compare the outputs with the expected tables. The optional validation cell should print `Passed: schemas, row multiplicities, and expected values match.` Stop on any error rather than skipping ahead.\n",
        "\n",
        "Starting a Spark session consumes Fabric capacity. When finished, save your notebook and stop the session. To record a reproducible result, retain the printed Spark/Python versions and UTC timestamp, note the selected Environment, release channel and Native Execution Engine setting, and repeat **Run all** after restarting the session.\n",
        "\n",
        "## 1. Set up a Runtime 2.0 notebook\n",
        "\n",
        "Use a notebook attached to a Fabric Environment configured for **Runtime 2.0**. Save and publish the Environment changes, select it in the notebook, and start a new session if an older session is running. Alternatively, use the workspace Runtime 2.0 default if your notebook has no Environment override.\n",
        "\n",
        "Run this in a **PySpark cell**:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-001",
      "metadata": {},
      "source": [
        "import platform\n",
        "from datetime import datetime, timezone\n",
        "\n",
        "print(\"Run started (UTC):\", datetime.now(timezone.utc).isoformat())\n",
        "print(f\"Spark version: {spark.version}\")\n",
        "print(\"Python version:\", platform.python_version())\n",
        "assert tuple(map(int, spark.version.split(\".\")[:2])) == (4, 1), (\n",
        "    \"This tutorial targets Spark 4.1. Select Fabric Runtime 2.0 \"\n",
        "    \"and start a new notebook session.\"\n",
        ")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-002",
      "metadata": {},
      "source": [
        "The examples below use session-scoped temporary views. For the purpose of the examples in this blog, no Lakehouse attachment, external data, package installation, or persistent table creation is required. You only need permission to run the notebook's Spark session.\n",
        "\n",
        "### Create the sales data\n",
        "\n",
        "Copy each block into its own notebook cell. SQL blocks start with `%%sql`, so you can paste them directly into cells in a PySpark notebook. Keep `%%sql` on the first line; do not put it inside a `spark.sql()` string."
      ]
    },
    {
      "cell_type": "code",
      "id": "article-003",
      "metadata": {},
      "source": [
        "%%sql\n",
        "CREATE OR REPLACE TEMP VIEW pipe_sales AS\n",
        "SELECT\n",
        "    order_id,\n",
        "    city,\n",
        "    product,\n",
        "    category,\n",
        "    CAST(price AS DECIMAL(10, 2)) AS price,\n",
        "    quantity\n",
        "FROM VALUES\n",
        "    (1, 'Seattle',  'Laptop',  'Electronics', 1200, 2),\n",
        "    (2, 'Seattle',  'Monitor', 'Electronics',  400, 3),\n",
        "    (3, 'New York', 'Laptop',  'Electronics', 1300, 4),\n",
        "    (4, 'New York', 'Desk',    'Furniture',    500, 2),\n",
        "    (5, 'Chicago',  'Chair',   'Furniture',    250, 6),\n",
        "    (6, 'Chicago',  'Laptop',  'Electronics', 1250, 1),\n",
        "    (7, 'Austin',   'Desk',    'Furniture',    550, 5),\n",
        "    (8, 'Austin',   'Monitor', 'Electronics',  450, 4)\n",
        "AS source(order_id, city, product, category, price, quantity);\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-004",
      "metadata": {},
      "source": [
        "Each row is one order line. Revenue means `price * quantity`; we are not modeling tax, discounts, returns, or currency conversion. We use a decimal price to avoid floating-point rounding surprises in monetary examples.\n",
        "\n",
        "Create a small lookup view in the next cell:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-005",
      "metadata": {},
      "source": [
        "%%sql\n",
        "CREATE OR REPLACE TEMP VIEW pipe_city_info AS\n",
        "SELECT *\n",
        "FROM VALUES\n",
        "    ('Seattle',  'West'),\n",
        "    ('New York', 'East'),\n",
        "    ('Chicago',  'Central'),\n",
        "    ('Austin',   'South')\n",
        "AS source(city, region);\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-006",
      "metadata": {},
      "source": [
        "These temporary views are scoped to your session. Rerunning setup replaces only these two views.\n",
        "\n",
        "Check the input:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-007",
      "metadata": {},
      "source": [
        "%%sql\n",
        "SELECT *\n",
        "FROM pipe_sales\n",
        "ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-008",
      "metadata": {},
      "source": [
        "You should see eight rows. Run the setup cells again after a session restart; temporary views do not survive it.\n",
        "\n",
        "Finally, initialize the shared Python imports and DataFrames used throughout the comparisons:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-009",
      "metadata": {},
      "source": [
        "from pyspark.sql import functions as F\n",
        "from pyspark.sql.window import Window\n",
        "\n",
        "sales = spark.table(\"pipe_sales\")\n",
        "city_info = spark.table(\"pipe_city_info\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-010",
      "metadata": {},
      "source": [
        "## 2. The mental model: each step receives a table\n",
        "\n",
        "A pipe query starts with a relation and passes its output to the next operator:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-011",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE price > 500\n",
        "|> SELECT city, product, price\n",
        "|> ORDER BY price DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-012",
      "metadata": {},
      "source": [
        "Read it top to bottom:\n",
        "\n",
        "**Start with sales > keep prices above 500 > choose three columns > sort.**\n",
        "\n",
        "`FROM pipe_sales` is itself a valid query. Each `|>` introduces an operator that consumes the previous result and produces another relation.\n",
        "\n",
        "Operators can repeat and can be composed in different orders, provided each operator receives the columns and input it needs. Reordering is not a promise of an equivalent result: filtering before aggregation can answer a different question from filtering afterward.\n",
        "\n",
        "This resembles a DataFrame chain, but it is **not** a command to execute and materialize every step separately. Spark analyzes and optimizes the complete query.\n",
        "\n",
        "| What you do in PySpark | Pipe SQL equivalent |\n",
        "| --- | --- |\n",
        "| `spark.table(\"pipe_sales\")` | `FROM pipe_sales` |\n",
        "| `.filter(...)` or `.where(...)` | `\\|> WHERE ...` |\n",
        "| `.select(...)` | `\\|> SELECT ...` |\n",
        "| `.withColumn(...)` with a new name | `\\|> EXTEND expression AS new_name` |\n",
        "| `.withColumn(...)` replacing an existing name | `\\|> SET existing_name = expression` |\n",
        "| `.drop(...)` | `\\|> DROP ...` |\n",
        "| `.groupBy(...).agg(...)` | `\\|> AGGREGATE ... GROUP BY ...` |\n",
        "| `.join(...)` | `\\|> JOIN ...` |\n",
        "| `.orderBy(...)` | `\\|> ORDER BY ...` |\n",
        "| `.limit(...)` | `\\|> LIMIT ...` |\n",
        "\n",
        "This is a mental-model guide, not a claim that every DataFrame method has identical semantics to a pipe operator. For example, PySpark's `withColumn` both adds and replaces columns; pipe SQL provides separate operators for those intentions.\n",
        "\n",
        "## 3. Filter, project, and sort\n",
        "\n",
        "Let's find order lines with at least three units, keeping just the columns we want.\n",
        "\n",
        "### Build the query one step at a time\n",
        "\n",
        "Run each of these four cells separately. Each cell is a complete query against the same view, not a fragment that depends on the previous cell's result.\n",
        "\n",
        "**Step 1: Start with the source.** This returns all eight rows and all six columns:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-013",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-014",
      "metadata": {},
      "source": [
        "**Step 2: Keep order lines with at least three units.** The result now has five rows, still with all six columns:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-015",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE quantity >= 3;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-016",
      "metadata": {},
      "source": [
        "**Step 3: Choose the output columns.** We still have five rows, but now only four columns:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-017",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE quantity >= 3\n",
        "|> SELECT order_id, city, product, quantity;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-018",
      "metadata": {},
      "source": [
        "**Step 4: Sort the result.** Add a final operation without changing the earlier steps:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-019",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE quantity >= 3\n",
        "|> SELECT order_id, city, product, quantity\n",
        "|> ORDER BY quantity DESC, order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-020",
      "metadata": {},
      "source": [
        "Before the final step, the displayed row order can vary. The row counts and available columns are the checkpoints; sorting makes the final output predictable.\n",
        "\n",
        "### The same query in PySpark\n",
        "\n",
        "Use the `sales` DataFrame and `F` alias from setup:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-021",
      "metadata": {},
      "source": [
        "quantity_result = (\n",
        "    sales\n",
        "    .filter(F.col(\"quantity\") >= 3)\n",
        "    .select(\"order_id\", \"city\", \"product\", \"quantity\")\n",
        "    .orderBy(F.col(\"quantity\").desc(), \"order_id\")\n",
        ")\n",
        "quantity_result.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-022",
      "metadata": {},
      "source": [
        "### The same query in conventional Spark SQL"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-023",
      "metadata": {},
      "source": [
        "%%sql\n",
        "SELECT order_id, city, product, quantity\n",
        "FROM pipe_sales\n",
        "WHERE quantity >= 3\n",
        "ORDER BY quantity DESC, order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-024",
      "metadata": {},
      "source": [
        "The final pipe query, PySpark chain, and conventional SQL query all return:\n",
        "\n",
        "| order_id | city | product | quantity |\n",
        "| --- | --- | --- | --- |\n",
        "| 5 | Chicago | Chair | 6 |\n",
        "| 7 | Austin | Desk | 5 |\n",
        "| 3 | New York | Laptop | 4 |\n",
        "| 8 | Austin | Monitor | 4 |\n",
        "| 2 | Seattle | Monitor | 3 |\n",
        "\n",
        "The second sort key makes tied quantities deterministic. Without an explicit final sort, Spark does not promise a particular display order.\n",
        "\n",
        "For a short query, conventional SQL is already clear. Pipes become more useful as the number of dependent transformations grows.\n",
        "\n",
        "## 4. Add, replace, and remove columns\n",
        "\n",
        "### Add a column with EXTEND\n",
        "\n",
        "In PySpark, the revenue calculation looks like this:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-025",
      "metadata": {},
      "source": [
        "revenue_lines = sales.withColumn(\n",
        "    \"revenue\", F.col(\"price\") * F.col(\"quantity\")\n",
        ")\n",
        "revenue_lines.orderBy(\"order_id\").show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-026",
      "metadata": {},
      "source": [
        "The pipe equivalent is:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-027",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-028",
      "metadata": {},
      "source": [
        "`EXTEND` keeps the input columns and appends the calculated column. The next operator can use `revenue` immediately:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-029",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> WHERE revenue > 1500\n",
        "|> SELECT order_id, city, product, revenue\n",
        "|> ORDER BY revenue DESC, order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-030",
      "metadata": {},
      "source": [
        "Expected result:\n",
        "\n",
        "| order_id | city | product | revenue |\n",
        "| --- | --- | --- | --- |\n",
        "| 3 | New York | Laptop | 5200.00 |\n",
        "| 7 | Austin | Desk | 2750.00 |\n",
        "| 1 | Seattle | Laptop | 2400.00 |\n",
        "| 8 | Austin | Monitor | 1800.00 |\n",
        "\n",
        "When a calculation depends on another calculated column, use separate stages to make the dependency clear:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-031",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> EXTEND CAST(revenue * 0.90 AS DECIMAL(14, 2)) AS discounted_revenue\n",
        "|> SELECT order_id, revenue, discounted_revenue\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-032",
      "metadata": {},
      "source": [
        "There is no need to depend on same-list alias resolution: `revenue` is already an input column to the second `EXTEND`.\n",
        "\n",
        "### Replace an existing column with SET\n",
        "\n",
        "Suppose you want to model a 10% price increase:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-033",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> SET price = CAST(price * 1.10 AS DECIMAL(10, 2))\n",
        "|> SELECT order_id, product, price\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-034",
      "metadata": {},
      "source": [
        "Order 1 now has a projected price of `1320.00`.\n",
        "\n",
        "**This does not update the source view or any stored data.** Pipe `SET` transforms the current query result; it is not an `UPDATE` statement. A later query against `pipe_sales` still sees `1200.00` for order 1.\n",
        "\n",
        "Use `EXTEND` for a new column and `SET` when replacing an existing one. Do not use `EXTEND` as a replacement operation: duplicate names can make later references ambiguous.\n",
        "\n",
        "### Remove columns with DROP"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-035",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> DROP category, product\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-036",
      "metadata": {},
      "source": [
        "The result contains `order_id`, `city`, `price`, and `quantity`. This is a query projection, not `ALTER TABLE ... DROP COLUMN`.\n",
        "\n",
        "## 5. Aggregate, then filter without HAVING\n",
        "\n",
        "Our next question is: **Which cities generated more than 4,000 in total revenue?**\n",
        "\n",
        "### PySpark"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-037",
      "metadata": {},
      "source": [
        "city_revenue = (\n",
        "    sales\n",
        "    .withColumn(\"revenue\", F.col(\"price\") * F.col(\"quantity\"))\n",
        "    .groupBy(\"city\")\n",
        "    .agg(F.sum(\"revenue\").alias(\"total_revenue\"))\n",
        "    .filter(F.col(\"total_revenue\") > 4000)\n",
        "    .orderBy(F.col(\"total_revenue\").desc(), \"city\")\n",
        ")\n",
        "city_revenue.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-038",
      "metadata": {},
      "source": [
        "### Conventional Spark SQL"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-039",
      "metadata": {},
      "source": [
        "%%sql\n",
        "SELECT city, SUM(price * quantity) AS total_revenue\n",
        "FROM pipe_sales\n",
        "GROUP BY city\n",
        "HAVING SUM(price * quantity) > 4000\n",
        "ORDER BY total_revenue DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-040",
      "metadata": {},
      "source": [
        "### Pipe SQL"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-041",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> AGGREGATE SUM(revenue) AS total_revenue GROUP BY city\n",
        "|> WHERE total_revenue > 4000\n",
        "|> ORDER BY total_revenue DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-042",
      "metadata": {},
      "source": [
        "Expected result:\n",
        "\n",
        "| city | total_revenue |\n",
        "| --- | --- |\n",
        "| New York | 6200.00 |\n",
        "| Austin | 4550.00 |\n",
        "\n",
        "Aggregation produces four city totals; the filter keeps two. Seattle (`3600.00`) and Chicago (`2750.00`) do not pass the threshold.\n",
        "\n",
        "### Position gives WHERE its meaning\n",
        "\n",
        "With pipes, `WHERE` filters the relation at that point in the chain:\n",
        "\n",
        "- **Before `AGGREGATE`:** filter input rows, like conventional `WHERE`.\n",
        "- **After `AGGREGATE`:** filter grouped results, like conventional `HAVING`.\n",
        "\n",
        "Here is both in one query:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-043",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE category = 'Electronics'\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> AGGREGATE SUM(revenue) AS total_revenue GROUP BY city\n",
        "|> WHERE total_revenue > 2000\n",
        "|> ORDER BY total_revenue DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-044",
      "metadata": {},
      "source": [
        "This time, the answer is New York (`5200.00`) and Seattle (`3600.00`). Austin has only `1800.00` of Electronics revenue; Chicago has `1250.00`.\n",
        "\n",
        "\n",
        "**Aggregation changes the available columns**\n",
        "\n",
        "After `AGGREGATE`, the output contains the grouping columns followed by the aggregate columns. In the example above, only `city` and `total_revenue` remain. A later operator cannot reference `category`, `price`, or `quantity` unless you retained them through an appropriate grouping or aggregation.\n",
        "\n",
        "\n",
        "Use the pipe `AGGREGATE` operator for aggregate functions, not a pipe `SELECT SUM(...)`. Pipe `SELECT` is for projection, including window expressions, rather than grouped aggregation.\n",
        "\n",
        "\n",
        "### Aggregate the whole input\n",
        "\n",
        "Without `GROUP BY`, `AGGREGATE` returns a single summary row:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-045",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> AGGREGATE\n",
        "     COUNT(*) AS order_lines,\n",
        "     SUM(price * quantity) AS total_revenue;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-046",
      "metadata": {},
      "source": [
        "Expected result: `order_lines = 8`, `total_revenue = 17100.00`.\n",
        "\n",
        "## 6. Join and rank: a complete analysis\n",
        "\n",
        "Before combining joins and windows, let's learn each operation on its own.\n",
        "\n",
        "### Rank city revenue without a join\n",
        "\n",
        "Start with a question that needs no lookup data: **Which three cities have the highest revenue across all categories?**"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-047",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> AGGREGATE SUM(revenue) AS total_revenue GROUP BY city\n",
        "|> EXTEND ROW_NUMBER() OVER (\n",
        "     ORDER BY total_revenue DESC, city\n",
        "   ) AS revenue_rank\n",
        "|> WHERE revenue_rank <= 3\n",
        "|> SELECT revenue_rank, city, total_revenue\n",
        "|> ORDER BY revenue_rank;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-048",
      "metadata": {},
      "source": [
        "Expected result:\n",
        "\n",
        "| revenue_rank | city | total_revenue |\n",
        "| --- | --- | --- |\n",
        "| 1 | New York | 6200.00 |\n",
        "| 2 | Austin | 4550.00 |\n",
        "| 3 | Seattle | 3600.00 |\n",
        "\n",
        "Follow the three stages after the revenue calculation:\n",
        "\n",
        "1. `AGGREGATE` produces four city totals.\n",
        "2. `EXTEND ROW_NUMBER() OVER (...)` adds a rank to those totals.\n",
        "3. The following `WHERE` keeps ranks 1 through 3.\n",
        "\n",
        "The window result is an input column to the next operator, so filtering it does not require an outer query. The `city` tie-breaker makes rank assignment deterministic, and the final sort controls display order. We will use the same pattern on regional totals shortly.\n",
        "\n",
        "Here is the same analysis in conventional SQL. A CTE names the ranked result so the outer query can filter it:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-049",
      "metadata": {},
      "source": [
        "%%sql\n",
        "WITH ranked_cities AS (\n",
        "    SELECT\n",
        "        city,\n",
        "        SUM(price * quantity) AS total_revenue,\n",
        "        ROW_NUMBER() OVER (\n",
        "            ORDER BY SUM(price * quantity) DESC, city\n",
        "        ) AS revenue_rank\n",
        "    FROM pipe_sales\n",
        "    GROUP BY city\n",
        ")\n",
        "SELECT revenue_rank, city, total_revenue\n",
        "FROM ranked_cities\n",
        "WHERE revenue_rank <= 3\n",
        "ORDER BY revenue_rank;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-050",
      "metadata": {},
      "source": [
        "Both versions are valid. The pipe version makes the transition from aggregation to ranking to filtering explicit without introducing a named query layer.\n",
        "\n",
        "Choose the ranking function to match the question:\n",
        "\n",
        "- `ROW_NUMBER() <= 3` returns exactly three rows when at least three are available; include a deterministic tie-breaker.\n",
        "- `RANK() <= 3`, ordered by revenue alone, keeps the first three competition-ranking positions, including ties, with gaps after ties.\n",
        "- `DENSE_RANK() <= 3`, ordered by revenue alone, keeps the three highest distinct revenue values, including all ties.\n",
        "\n",
        "An unpartitioned window requires a global window stage. Applying it to four aggregate rows is reasonable; applying it to millions of detail rows can create a bottleneck.\n",
        "\n",
        "### Start with a simple join\n",
        "\n",
        "Now enrich the original order lines with their regions, without aggregating or ranking anything:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-051",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> LEFT JOIN pipe_city_info USING (city)\n",
        "|> SELECT order_id, city, region\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-052",
      "metadata": {},
      "source": [
        "Expected result:\n",
        "\n",
        "| order_id | city | region |\n",
        "| --- | --- | --- |\n",
        "| 1 | Seattle | West |\n",
        "| 2 | Seattle | West |\n",
        "| 3 | New York | East |\n",
        "| 4 | New York | East |\n",
        "| 5 | Chicago | Central |\n",
        "| 6 | Chicago | Central |\n",
        "| 7 | Austin | South |\n",
        "| 8 | Austin | South |\n",
        "\n",
        "There are still eight rows: our lookup has one matching row per city. We have added regional context, not calculated regional totals. Inspecting this enriched result first makes it easier to spot lookup problems before adding aggregation.\n",
        "\n",
        "`USING (city)` joins on equal city values and exposes one join-key column. In real data, duplicate lookup keys can multiply sales rows and inflate totals. A left join preserves unmatched sales rows with a null `region`; those rows form a null group during aggregation unless you handle them.\n",
        "\n",
        "### Combine the steps into a regional analysis\n",
        "\n",
        "Now let's answer a more realistic question:\n",
        "\n",
        "**For Electronics, what are the top three regions by revenue, and how many units did they sell?**\n",
        "\n",
        "The shape of the query tells the story:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "article-053",
      "metadata": {},
      "source": [
        "```text\n",
        "Sales -> filter Electronics -> calculate revenue -> join region\n",
        "      -> aggregate revenue and units -> rank -> keep top three\n",
        "      -> select output columns -> sort for display\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "article-054",
      "metadata": {},
      "source": [
        "Define the complete analysis as a temporary view so we can also read it from PySpark later. The view stores the query definition, not a materialized or cached result:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-055",
      "metadata": {},
      "source": [
        "%%sql\n",
        "CREATE OR REPLACE TEMP VIEW pipe_top_regions AS\n",
        "FROM pipe_sales\n",
        "|> WHERE category = 'Electronics'\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> LEFT JOIN pipe_city_info USING (city)\n",
        "|> AGGREGATE\n",
        "     SUM(revenue) AS total_revenue,\n",
        "     SUM(quantity) AS units_sold\n",
        "   GROUP BY region\n",
        "|> EXTEND ROW_NUMBER() OVER (\n",
        "     ORDER BY total_revenue DESC, region\n",
        "   ) AS revenue_rank\n",
        "|> WHERE revenue_rank <= 3\n",
        "|> SELECT revenue_rank, region, units_sold, total_revenue;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-056",
      "metadata": {},
      "source": [
        "Read and sort the view to display the result:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-057",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_top_regions\n",
        "|> ORDER BY revenue_rank;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-058",
      "metadata": {},
      "source": [
        "Expected result:\n",
        "\n",
        "| revenue_rank | region | units_sold | total_revenue |\n",
        "| --- | --- | --- | --- |\n",
        "| 1 | East | 4 | 5200.00 |\n",
        "| 2 | West | 5 | 3600.00 |\n",
        "| 3 | South | 4 | 1800.00 |\n",
        "\n",
        "### The same complete analysis in PySpark\n",
        "\n",
        "This creates `pyspark_result` for the optional validation section, using the imports and DataFrames from setup:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-059",
      "metadata": {},
      "source": [
        "rank_window = Window.orderBy(F.col(\"total_revenue\").desc(), \"region\")\n",
        "\n",
        "pyspark_result = (\n",
        "    sales\n",
        "    .filter(F.col(\"category\") == \"Electronics\")\n",
        "    .withColumn(\"revenue\", F.col(\"price\") * F.col(\"quantity\"))\n",
        "    .join(city_info, on=\"city\", how=\"left\")\n",
        "    .groupBy(\"region\")\n",
        "    .agg(\n",
        "        F.sum(\"revenue\").alias(\"total_revenue\"),\n",
        "        F.sum(\"quantity\").alias(\"units_sold\"),\n",
        "    )\n",
        "    .withColumn(\"revenue_rank\", F.row_number().over(rank_window))\n",
        "    .filter(F.col(\"revenue_rank\") <= 3)\n",
        "    .select(\"revenue_rank\", \"region\", \"units_sold\", \"total_revenue\")\n",
        "    .orderBy(\"revenue_rank\")\n",
        ")\n",
        "pyspark_result.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-060",
      "metadata": {},
      "source": [
        "The correspondence is the point: pipes let the SQL version follow the same sequence of transformations without changing the business logic.\n",
        "\n",
        "## 7. Use pipes with existing SQL and PySpark\n",
        "\n",
        "You do not have to choose between all conventional SQL and all pipe SQL.\n",
        "\n",
        "### Append a pipe to a conventional query\n",
        "\n",
        "Suppose an existing query already produces city totals. Keep it and add the next transformation:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-061",
      "metadata": {},
      "source": [
        "%%sql\n",
        "SELECT city, SUM(price * quantity) AS total_revenue\n",
        "FROM pipe_sales\n",
        "GROUP BY city\n",
        "|> WHERE total_revenue > 4000\n",
        "|> ORDER BY total_revenue DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-062",
      "metadata": {},
      "source": [
        "Expected result: New York (`6200.00`) and Austin (`4550.00`).\n",
        "\n",
        "Put the semicolon at the **end of the complete chain**, not before `|>`.\n",
        "\n",
        "### Read the pipe view from PySpark\n",
        "\n",
        "Read the view created in section 6 and keep the same final ordering as the PySpark comparison:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-063",
      "metadata": {},
      "source": [
        "pipe_result = spark.table(\"pipe_top_regions\").orderBy(\"revenue_rank\")\n",
        "pipe_result.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-064",
      "metadata": {},
      "source": [
        "Sorting when reading the view makes the presentation requirement explicit; a view does not guarantee row order.\n",
        "\n",
        "You can also pass pipe SQL directly to `spark.sql()`:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-065",
      "metadata": {},
      "source": [
        "spark.sql(\"\"\"\n",
        "    FROM pipe_sales\n",
        "    |> WHERE category = 'Electronics'\n",
        "    |> AGGREGATE SUM(price * quantity) AS total_revenue GROUP BY city\n",
        "    |> ORDER BY total_revenue DESC, city\n",
        "\"\"\").show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-066",
      "metadata": {},
      "source": [
        "This is ordinary Spark SQL submitted from Python, not a separate PySpark pipe API.\n",
        "\n",
        "## Try it yourself\n",
        "\n",
        "Translate this PySpark chain before looking at the answer:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-067",
      "metadata": {},
      "source": [
        "exercise_result = (\n",
        "    sales\n",
        "    .filter(F.col(\"quantity\") >= 2)\n",
        "    .withColumn(\"revenue\", F.col(\"price\") * F.col(\"quantity\"))\n",
        "    .groupBy(\"city\")\n",
        "    .agg(F.sum(\"revenue\").alias(\"total_revenue\"))\n",
        "    .filter(F.col(\"total_revenue\") > 1500)\n",
        "    .orderBy(F.col(\"total_revenue\").desc(), \"city\")\n",
        ")\n",
        "exercise_result.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-068",
      "metadata": {},
      "source": [
        "One pipe SQL translation is:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-069",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE quantity >= 2\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> AGGREGATE SUM(revenue) AS total_revenue GROUP BY city\n",
        "|> WHERE total_revenue > 1500\n",
        "|> ORDER BY total_revenue DESC, city;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-070",
      "metadata": {},
      "source": [
        "Expected answer: New York (`6200.00`), Austin (`4550.00`), and Seattle (`3600.00`). Chicago's one-unit laptop line is filtered out before aggregation; the remaining `1500.00` does not satisfy `> 1500`.\n",
        "\n",
        "You have now covered the core tutorial. The remaining sections are optional checks and reference material.\n",
        "\n",
        "## Optional: validate results and inspect plans\n",
        "\n",
        "### Check both equivalence and the expected answer\n",
        "\n",
        "Run this PySpark cell after creating `pyspark_result` and `pipe_result` above:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-071",
      "metadata": {},
      "source": [
        "from decimal import Decimal\n",
        "\n",
        "assert pipe_result.columns == pyspark_result.columns, \"Column order differs\"\n",
        "assert pipe_result.schema == pyspark_result.schema, \"Output schemas differ\"\n",
        "\n",
        "assert pipe_result.exceptAll(pyspark_result).count() == 0, (\n",
        "    \"Pipe SQL produced unexpected rows\"\n",
        ")\n",
        "assert pyspark_result.exceptAll(pipe_result).count() == 0, (\n",
        "    \"Pipe SQL is missing expected rows\"\n",
        ")\n",
        "\n",
        "expected = [\n",
        "    (1, \"East\", 4, Decimal(\"5200.00\")),\n",
        "    (2, \"West\", 5, Decimal(\"3600.00\")),\n",
        "    (3, \"South\", 4, Decimal(\"1800.00\")),\n",
        "]\n",
        "actual = [\n",
        "    tuple(row)\n",
        "    for row in pipe_result.orderBy(\"revenue_rank\").collect()\n",
        "]\n",
        "assert actual == expected, f\"Unexpected result: {actual}\"\n",
        "print(\"Passed: schemas, row multiplicities, and expected values match.\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-072",
      "metadata": {},
      "source": [
        "`exceptAll` preserves duplicates and compares columns by position, which is why we explicitly align the output columns. The two directions catch both extra and missing rows. The fixed expected answer also protects against making the same mistake in both implementations.\n",
        "\n",
        "We use `collect()` only for this tiny three-row tutorial result. Do not collect an unbounded production dataset to the driver.\n",
        "\n",
        "### Understand what changes in execution\n",
        "\n",
        "Pipe syntax is a different way to express relational operations. Spark analyzes those operations, optimizes the logical plan, and selects physical operators using the same planning machinery as other Spark SQL queries.\n",
        "\n",
        "For example:\n",
        "\n",
        "- `WHERE` creates a filter that the optimizer may push closer to the source when semantically safe.\n",
        "- `EXTEND`, `SET`, and `SELECT` express projections; they are not automatically separate materialization points.\n",
        "- `AGGREGATE` may require a shuffle to bring grouping keys together.\n",
        "- Joins still depend on data size, statistics, join conditions, and available execution strategies.\n",
        "- A global sort or window still has its normal distributed-execution cost.\n",
        "\n",
        "Both result DataFrames include the same final ordering. Inspect their plans:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-073",
      "metadata": {},
      "source": [
        "pipe_result.explain(mode=\"formatted\")\n",
        "pyspark_result.explain(mode=\"formatted\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-074",
      "metadata": {},
      "source": [
        "Compare joins, exchanges, aggregates, windows, and sorts rather than expecting identical plan text. **Eight rows demonstrate syntax and correctness, not performance.** For performance testing, use representative data with the same input snapshot, compute, runtime, cache state, and execution settings.\n",
        "\n",
        "### Confirm the source was not modified\n",
        "\n",
        "The earlier `SET` and `DROP` examples only changed their query results. This check verifies the original price and columns remain:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-075",
      "metadata": {},
      "source": [
        "assert spark.table(\"pipe_sales\").columns == [\n",
        "    \"order_id\", \"city\", \"product\", \"category\", \"price\", \"quantity\"\n",
        "], \"The source schema changed\"\n",
        "assert spark.table(\"pipe_sales\").where(\"order_id = 1\").first().price == Decimal(\"1200.00\"), (\n",
        "    \"The source price changed\"\n",
        ")\n",
        "print(\"Passed: the original source columns and price are unchanged.\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-076",
      "metadata": {},
      "source": [
        "## Optional: two more useful patterns\n",
        "\n",
        "### Name the current relation with AS\n",
        "\n",
        "For an `ON` join, use `AS` after a transformation to name the current relation, including its new columns:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-077",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> AS s\n",
        "|> LEFT JOIN pipe_city_info AS c ON s.city = c.city\n",
        "|> SELECT s.order_id, s.city, c.region, s.revenue\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-078",
      "metadata": {},
      "source": [
        "Explicit projection avoids duplicate column names after an `ON` join.\n",
        "\n",
        "### Use LIMIT when you do not need a rank column\n",
        "\n",
        "For the top three Electronics cities without rank values:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-079",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE category = 'Electronics'\n",
        "|> AGGREGATE SUM(price * quantity) AS total_revenue GROUP BY city\n",
        "|> ORDER BY total_revenue DESC, city\n",
        "|> LIMIT 3;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-080",
      "metadata": {},
      "source": [
        "## Debugging and common mistakes\n",
        "\n",
        "### Run a complete prefix of the chain\n",
        "\n",
        "If the final total is wrong, inspect the input to aggregation first:"
      ]
    },
    {
      "cell_type": "code",
      "id": "article-081",
      "metadata": {},
      "source": [
        "%%sql\n",
        "FROM pipe_sales\n",
        "|> WHERE category = 'Electronics'\n",
        "|> EXTEND price * quantity AS revenue\n",
        "|> LEFT JOIN pipe_city_info USING (city)\n",
        "|> SELECT order_id, city, region, revenue\n",
        "|> ORDER BY order_id;\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "article-082",
      "metadata": {},
      "source": [
        "You should see five rows, with revenues `2400.00`, `1200.00`, `5200.00`, `1250.00`, and `1800.00` in order-ID order. Then add aggregation back.\n",
        "\n",
        "Do not run a bare `|> EXTEND ...` fragment in a new cell: a pipe operator needs its preceding query. Query prefixes are inspection checkpoints, not saved execution checkpoints.\n",
        "\n",
        "| Symptom or trap | What to check |\n",
        "| --- | --- |\n",
        "| Parser error at `\\|>` | Verify the active Spark version and that this is a Spark notebook, not a T-SQL endpoint. |\n",
        "| View not found | Run the setup in the current session; temporary views disappear after restart. |\n",
        "| Unresolved column after projection | `SELECT` removed a column that a later operator still needs. |\n",
        "| Unresolved column after aggregation | Only grouping and aggregate outputs remain. Filter detail columns earlier. |\n",
        "| Ambiguous column | Use `SET` for replacement, and qualify/project duplicate columns after an `ON` join. |\n",
        "| Totals increase after a join | Check lookup-key uniqueness and join cardinality. Pipes do not prevent fan-out. |\n",
        "| Top-N results vary across runs | Add deterministic tie-breakers and a final `ORDER BY`. |\n",
        "| A query became slower after reordering | Operator order still changes semantics and can change the physical plan. |\n",
        "\n",
        "`WHERE` still uses SQL null semantics: only predicates evaluating to true keep a row. Pipe syntax does not change null handling, join semantics, ANSI behavior, or the need to understand your data.\n",
        "\n",
        "Your SQL formatter, linter, or external editor may also need an update to recognize `|>`, even when the Spark runtime supports it.\n",
        "\n",
        "## Final takeaway: write the next transformation\n",
        "\n",
        "Pipe syntax does not replace PySpark or conventional Spark SQL. It gives you another way to compose SQL, especially when your analysis needs a sequence of dependent stages.\n",
        "\n",
        "The useful question is not \"Can I make this query shorter?\" It is **\"Can the next developer follow what happens next?\"**\n",
        "\n",
        "I recommend starting with one familiar DataFrame chain or one SQL query with several nested stages. Express it as pipes, compare the answers, and inspect the plans. Keep the version that makes the transformation clearest without changing its meaning.\n",
        "\n",
        "## Further reading\n",
        "\n",
        "- [Apache Spark 4.1: SQL pipe syntax and supported operators](https://spark.apache.org/docs/4.1.0/sql-pipe-syntax.html)\n",
        "- [Apache Spark 4.0: SQL pipe syntax](https://spark.apache.org/docs/4.0.0/sql-pipe-syntax.html), for the earlier availability of the feature\n",
        "- [Microsoft Fabric Runtime 2.0 and runtime selection](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0)\n",
        "- [Fabric notebook development and language magics](https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook)\n",
        "- [Import and use notebooks in Fabric](https://learn.microsoft.com/en-us/fabric/data-engineering/how-to-use-notebook#import-existing-notebooks)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Synapse PySpark",
      "language": "python",
      "name": "synapse_pyspark"
    },
    "language_info": {
      "name": "python"
    },
    "tutorial": {
      "source": "_posts/2026-10-19-pipe-syntax-spark-4-1-fabric-runtime-2-0.md",
      "source_sha256": "338264452572cd11537358215b323b1f472f0a63aa326fab1edfdf4ef49e04e6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}