Fabric Notebooks: implementing CI/CD ribboning and testing for consistent, modular code

As Senior Data Engineers, we want a pipeline that is deterministic, fast and understandable. No magic, no surprises. The goal is a workflow that:

• Enforce consistency
  No more discussions about line lengths or formatting.

• Modularity encourages
  Code that is not in a function or class is hardly testable. This enforces better habits.

• Readability improves
  Most of your time is spent reading code. Modular, lined code is simply cheaper to maintain.

• Independent remains
  No API calls to external services. Only pure Python on your build agent.

By controlling this in your own Git environment (Azure DevOps, GitHub, GitLab), you maintain complete control.
Compared to AI agents, “boring” tools also have distinct advantages:

• No hallucinations – the linter follows PEP8, no guesswork.

• No API fees – you use CPU cycles you are already paying for.

• No governance nightmare – sensitive data stays within your trusted Git environment.

Fabric notebooks are full of magic commands (%pip, %sql, %run). Useful for speed, disastrous for standard linters like Ruff – they crash immediately.

The solution: the preamble strategy

We hide magic commands temporarily, without forgetting where they were.
We do this by prefixing lines beginning with % with a unique comment marker, for example:

Why those three pipes (|||)?

When Ruff adds or deletes rules, we need a watertight way to recover our magic commands. With a unique preamble, we can always recover them exactly after ribboning.

Implementation in Python:

About multi-line magics

IPython technically allows magic commands to span multiple lines with backslashes.

But honestly: if you do that, you build a maintenance problem. Keep your magics simple and your Python logic modular. Time wasted on unreadable code is more expensive than any compute.

The pipeline flow

1. Transform – % → #||| %

2. Ribbons & fixes – ruff check --fix file.py

3. Recover – #||| % → %

4. Committing – if there are changes, the pipeline commits them back

Sample output:

Pro-tip: don’t trust a bot to commit directly?
Have the pipeline create a separate branch (fix/linting-XYZ) and automatically open a PR. A human still clicks on merge – secure AND scalable.

That way your pipeline doesn’t whine about technical debt.
He solves it.

Notebook testing is tricky. An import often runs the entire notebook.
If that includes a heavy Spark job, you may wait 20 minutes for your test to fail – for one helper function.

The solution: the AST (Abstract Syntax Tree)

Instead of executing the file, we parse it with Python’s ast-module.
We are looking for targeted:

• functions
• classes
• imports

We detach those from the notebook and load them into the test scope, without triggering top-level code.

What does this yield?

• The “modularity root“
  Only code in functions and classes is tested. Loose clutter at the top does not count.

• No side-effects
  No if __name__ == "__main__"-constructions or hacks to prevent execution.

Real Spark, local speed

No heavy mocks.
We start a single-node Spark session locally on the build agent (Azure DevOps or GitHub Actions).

Spark + Delta installs surprisingly quickly on modern agents.
You test real transformations and schema logic, without using Fabric capability.

Tests

With access to Spark and your notebook features, you can write tests – even in notebook form, as you are used to in Fabric.

What you test, and how to do it maintainably, deserves a separate blog.