cloning:run

Execute a cloning transfer: validate the .cloning.yaml configuration, verify connectivity, and transfer data from the source database to the target with all configured anonymization transformations applied. Produces a signed audit log and a structured run log delivered to the configured channels.

Usage

clonio cloning:run <file> [options]

Arguments

Options

--skip-tables and --only-tables are mutually exclusive. Verbosity flags -v / -vv / -vvv are also supported (see Output Modes).

--break-on-failure

By default, Clonio continues processing all tables even when one fails — every table gets a chance to transfer and the full results are reported at the end. Pass --break-on-failure to abort the run immediately the first time a table fails (either at schema creation or at data transfer).

The audit log is always written, even on early abort.

Prerequisites

1. Run clonio init to set up APP_KEY
2. Add connections with clonio connection:add
3. Generate a configuration file with clonio cloning:dump
4. Review and adjust the generated .cloning.yaml

Examples

Basic transfer

clonio cloning:run production-db.cloning.yaml --target local-dev

Dry run (no data transferred)

Validates the config, tests connectivity, estimates row counts, and shows the schema diff between source and target — without moving any data:

clonio cloning:run production-db.cloning.yaml --target staging --dry-run

  Dry-run: production-db

  Schema diff: source → target

  Missing tables (1):   audit_logs
  Modified table:        users: +1 cols (phone)

  Table                   Rows (est.)   Strategy   Transformations
  ─────────────────────────────────────────────────────────────────
  users                    12 340       last 5000   email, first_name, password
  orders                   48 201       full        shipping_address
  audit_logs              NOT FOUND     —           —
  product_catalog             923       full        (none)
  sessions                      0       schema      —
  ─────────────────────────────────────────────────────────────────
  Total                    61 464 rows across 4 tables (1 not found, will be skipped)

  No data will be transferred. Run without --dry-run to execute.

When source and target schemas are identical, the diff block shows:

  Schema diff: target matches source

Skip specific tables

clonio cloning:run production-db.cloning.yaml --target staging --skip-tables=audit_logs,sessions

Transfer only specific tables

clonio cloning:run production-db.cloning.yaml --target staging --only-tables=users,orders

CI mode

clonio cloning:run production-db.cloning.yaml --target staging --ci

Optional CI step (always exits 0)

# GitHub Actions example
- name: Sync staging (optional)
  run: clonio cloning:run prod.cloning.yaml --target staging --ci --allow-failure

Dump to a SQL-file archive

When --target names a dump connection, no live target database is touched. The source is read and anonymized as usual, but schema and data are written to a dialect-correct .sql file and compressed into a ZIP archive in the current directory:

clonio cloning:run production-db.cloning.yaml --target staging-dump

  Tables: 24/24  Rows: 18432  Duration: 4.1s

  Dump: app_20260615_143022.zip  (2.3 MB, AES-256)

The DDL/DML respect options.drop_unknown_tables, options.chunk_size, and options.disable_foreign_key_checks from the YAML, the same way a live target does. The source connection must be a real database. See dump Connection Type for archive format, cross-dialect type mapping, and encryption details.

Schema Synchronization

Before transferring data, Clonio synchronizes the target schema to match the source. Six options in the options: block of the YAML file control this behaviour and the data transfer. These values are set interactively by cloning:dump and can be overridden at run time via CLI flags.

options:
  chunk_size: 1000
  enforce_column_types: true   # add missing columns
  drop_extra_columns: true     # remove extra columns
  drop_unknown_tables: false   # keep extra tables
  disable_foreign_key_checks: true
  faker_locale: en_US

All schema-sync options are applied during Phase 4 — Schema Replication, which runs before any data is transferred. Schema replication is skipped entirely when --skip-schema is passed on the command line.

Table creation

Missing tables are always created, regardless of any option setting. If a table exists in the source but not on the target, Clonio creates it during Phase 4. No option needs to be enabled for this — it is unconditional.

The following options control what happens to columns and tables that diverge from the source:

Generated columns

Generated (computed) columns are never copied. Clonio leaves a column out of the transfer when it is generated on the source or on the target, or when it uses the exclude strategy.

Clonio does not recreate generation expressions. Only a MySQL/MariaDB source copied to a target of the same type keeps them, through the native table definition. Transfers between different database systems, columns added later and SQL dumps get a plain, nullable column that stays empty.

Each affected table gets a note, for example:

Note: bookings: Generated columns not copied and left empty: day. The generation expression was not replicated to the target.

The notes appear on the console, in the process log and in the Notes section of the audit log. See Generated columns for details.

Schema-only tables

Tables with rows.strategy: schema are created and synchronized like any other table, but no rows are copied. rows.clear still runs, so clear: delete empties an existing target table. The dry-run shows them with 0 estimated rows.

When a copied table has a foreign key to a schema-only table, its foreign key values point to rows that do not exist on the target. The table result gets a note:

Note: members: References tables copied without data ('schema' strategy): teams. Foreign key values may point to missing rows, or inserts fail when foreign key checks are enabled.

Use skip instead when the referencing tables should be left out too. See Row strategies.

Overriding options at run time

CLI flags override the YAML options: values for a single run without modifying the file:

# Force table-creation safety net even if YAML says false
clonio cloning:run prod.cloning.yaml --target staging --enforce-column-types

# Tear down stale tables on a throwaway CI database
clonio cloning:run prod.cloning.yaml --target ci-db --drop-unknown-tables --drop-extra-columns

# Rebuild drifted target tables from the source schema (e.g. a column that is too narrow)
clonio cloning:run prod.cloning.yaml --target staging --recreate-tables

# Disable a destructive option set in the YAML for this run only
clonio cloning:run prod.cloning.yaml --target staging --no-drop-unknown-tables

Both the --<option> and --no-<option> variants are always available, so you can force either direction regardless of what the YAML contains.

Schema diff in dry-run

--dry-run always inspects both the source and target schemas and prints a diff summary before the per-table row counts, so you can see exactly what Phase 4 will change before committing to a full run.

Caution: drop_extra_columns

Dropping columns is irreversible and will destroy any data stored in those columns on the target. Only enable drop_extra_columns: true when:

• The target environment is ephemeral (e.g. a fresh CI database) or regularly rebuilt, or
• You have confirmed that the extra columns on the target are safe to remove.

Caution: recreate_tables

recreate_tables: true drops every selected table that already exists on the target and re-creates it from the source schema before the data transfer. Use it when the target schema has drifted from the source (for example, a varchar column on the target is shorter than on the source, which makes inserts fail with "Data too long").

• All target-only structures on those tables are lost: indexes, triggers and extra columns.
• Foreign keys from the source are not re-created on the target.
• On PostgreSQL the table is dropped with CASCADE, and on SQL Server foreign keys that reference the table are dropped first. Foreign-key constraints on other target tables that point at a recreated table are therefore removed. On MySQL/MariaDB and SQLite, foreign-key checks are disabled for the schema session so the drop succeeds and those constraints stay.
• Only tables selected for transfer are recreated. Tables in skip: or not listed under tables: stay untouched.
• It has no effect with --skip-schema, because Phase 4 does not run.
• If a drop fails (e.g. missing privileges), the table is reported as a schema failure and skipped.

Key Remapping

Key remapping assigns new IDs to transferred primary keys and rewrites all foreign key references that point to those IDs, preventing ID collisions on the target environment.

Defining remapping inline (recommended)

Add strategy: remapping to any column in a table's columns block. The arguments list specifies the strategy and, crucially, which other columns hold foreign keys to this one:

tables:
  users:
    rows:
      strategy: full
    columns:
      id:
        strategy: remapping
        arguments:
          - use: random_integer
          - foreign_keys:
              - table: orders
                column: user_id
              - table: employees
                column: manager_id
                self_referential: true   # employees.manager_id → employees.id
      email:
        strategy: fake
        faker_method: safeEmail
        faker_arguments: []

  orders:
    rows:
      strategy: full
    columns:
      id:
        strategy: remapping
        arguments:
          - use: random_integer
          - foreign_keys:
              - table: order_items
                column: order_id
    # orders.user_id is rewritten automatically because users.id declares it as a FK

  order_items:
    rows:
      strategy: full
    # no PII detected — no columns listed; all kept as-is

  employees:
    rows:
      strategy: full
    columns:
      id:
        strategy: remapping
        arguments:
          - use: random_integer
          - foreign_keys:
              - table: employees
                column: manager_id
                self_referential: true

How FK rewriting works:

• orders.user_id is rewritten to the new value for whichever users.id matched the original.
• order_items.order_id is rewritten to the new orders.id.
• employees.manager_id is self-referential: on first pass it is set to null, then updated in a second pass once all employees are inserted.

Argument reference

Each foreign_keys entry:

Disabling remapping

Key remapping can be turned off without removing the remapping definition — either for a single run or persistently in the YAML:

clonio cloning:run production-db.cloning.yaml --target staging --no-remap-keys

options:
  remap_keys: false

Both bypass all key mapping generation and PK/FK rewriting even when the YAML has strategy: remapping columns defined, so primary and foreign key values are copied from the source unchanged. Outside --ci, the run prints a notice when this happens. --skip-remapping-keys is kept as an alias of --no-remap-keys.

When the YAML sets remap_keys: false, --remap-keys re-enables remapping for a single run. If both --remap-keys and --no-remap-keys are passed, --no-remap-keys wins.

Memory management for large databases

By default, all PK mappings are held in RAM simultaneously (one entry per row per remapped table). For most databases this is fast and adequate. For very large datasets two flags are available:

Both flags can be used together if needed. When remapping is disabled (--no-remap-keys or remap_keys: false), both flags are silently ignored.

# Remove memory cap (dataset fits in RAM, just over the PHP limit)
clonio cloning:run prod.cloning.yaml --target staging --no-memory-limit

# Encrypt mappings to disk (dataset too large for RAM)
clonio cloning:run prod.cloning.yaml --target staging --file-based

Legacy format (still supported)

The original top-level key_remapping: section is still parsed. Existing YAML files do not need to be updated. When both formats are present in the same file, the inline column format takes priority.

Recovering from "key remapping exhausted"

When the column type cannot host the source row count (for example a TINYINT primary key that already holds 250 rows but the source has 300), the run aborts in Phase 5b with a key remapping exhausted error.

• Interactive mode — Clonio prints a prose summary (column type, ceiling, rows requested, slots available) and prompts:

  Switch users.id strategy to 'keep' in production-db.cloning.yaml? (yes/no)
  

  Accepting the prompt invokes cloning:column:edit ... --strategy=keep in-process, which rewrites the YAML so the offending column is no longer remapped. The command exits with code 0 and prints the original cloning:run invocation so you can re-run it. Declining the prompt also exits with code 0 and leaves the YAML untouched.

• CI mode (--ci) — Clonio prints the same prose summary plus a hint, including the exact cloning:column:edit command needed to patch the configuration, then exits with code 1 (GeneralError). No interactive prompts are shown.

If widening the column type is the right fix instead of switching to keep, run a schema migration on the source and retry — Clonio will pick up the new ceiling automatically on the next cloning:run.

The 8-Phase Pipeline

Phase 1  — YAML Validation
Phase 2  — Connection Checks
Phase 3  — Dry-run               (only if --dry-run; exits here)
Phase 4  — Schema Replication    (skipped if --skip-schema)
Phase 5  — Dependency Resolution
Phase 5b — Key Mapping Generation (when remapping columns are defined)
Phase 6  — Data Transfer
Phase 7  — Key Mapping Cleanup   (when remapping columns are defined)
Phase 8  — Audit Log & Process Log
Phase 9  — Summary

Each phase must complete before the next begins.

Phase 1 — YAML Validation

Loads and validates the YAML file against the Clonio schema. All validation errors are reported at once. Logical constraints are also checked (e.g. first/last row strategy requires a limit; hash requires a supported algorithm).

Phase 2 — Connection Checks

Resolves the source connection (from the YAML connection field) and the target connection (from --target or interactive selection). Tests both with a real PDO ping. Verifies that source and target are not the same connection.

Phase 3 — Dry-run

Only runs when --dry-run is set. Fetches both the source and target schemas, computes the schema diff, counts estimated rows per table, and prints a summary. Tables not found in the source are shown as NOT FOUND. Exits with code 0 after printing the summary.

Phase 4 — Schema Replication

Synchronizes the target schema with the source. Controlled by five options in the YAML options block (see Schema Synchronization):

• Always creates tables that exist in the source but not in the target.
• enforce_column_types: true — adds missing columns to existing target tables.
• drop_extra_columns: true — drops columns from existing target tables that are absent in the source.
• drop_unknown_tables: true — drops tables from the target that are not present in the source.
• recreate_tables: true — drops selected tables that already exist on the target and re-creates them from the source schema.

Each option can be overridden for a single run via CLI flags (--enforce-column-types, --drop-unknown-tables, --drop-extra-columns, --recreate-tables, --disable-fk-checks, --remap-keys and their --no-* counterparts) without modifying the YAML file.

Skipped entirely when --skip-schema is set.

Phase 5 — Dependency Resolution

Builds a foreign key dependency graph and topologically sorts tables in parent-first order to avoid FK violations during insert. Tables excluded by --skip-tables or --only-tables — and all tables with a FK dependency on them — are removed from the working set (see Table Filtering).

Phase 6 — Data Transfer

For each table (in resolved order):

1. If options.disable_foreign_key_checks is true, foreign-key constraints are disabled on the target connection.
2. If rows.clear is truncate or delete, the target table is emptied before any rows are inserted (see "Clearing Tables" below).
3. Tables with rows.strategy: schema stop here: no rows are copied.
4. Rows are fetched from the source in chunks and the configured column transformations are applied before inserting into the target.
5. On unique or FK constraint violations, falls back to row-by-row insert and records skipped rows.

Clearing Tables

The rows.clear setting in the YAML config controls whether the target table is emptied before transfer:

Clearing happens after FK checks are disabled, so TRUNCATE does not fail on FK-constrained tables (on databases that enforce this restriction at the statement level).

Phase 7 — Audit Log & Process Log

Assembles and signs the HTML audit log and JSONL process log, then delivers them to the configured channels according to each channel's delivery settings. Skipped silently when audit is absent from clonio.json.

Phase 8 — Summary

Prints the run summary. Tables not found in the source are always listed, regardless of verbosity level.

Table Filtering

--skip-tables and --only-tables

Both flags accept a comma-separated list of exact table names. Glob patterns and regex are not supported. Spaces around commas are trimmed.

# Skip two tables
clonio cloning:run prod.cloning.yaml --target staging --skip-tables=audit_logs,sessions

# Transfer only these tables
clonio cloning:run prod.cloning.yaml --target staging --only-tables=users,orders

Using both flags together is a ValidationError (4).

FK-Dependent Cascade

When a table is excluded, all tables with a foreign key dependency on it — directly or transitively — are also excluded automatically:

  Skipped tables:
    users        (--skip-tables)
    orders       (cascaded: FK dependency on users)
    order_items  (cascaded: FK dependency on orders)

Cascaded tables appear in the audit log with status skipped_by_cascade.

Skipping tables in YAML

Instead of passing --skip-tables on every invocation, you can declare tables to skip permanently inside cloning.yaml. Two syntaxes are supported and can be combined:

Top-level skip: list — for tables that need no anonymisation config:

skip:
  - audit_logs
  - telescope_entries
  - failed_jobs

rows.strategy: skip — for tables already present in tables::

tables:
  audit_logs:
    rows:
      strategy: skip
  users:
    rows:
      strategy: full
    columns:
      email:
        strategy: fake
        faker_method: safeEmail
        faker_arguments: []

YAML-level skips and --skip-tables are additive: both lists are merged at runtime. The same cascade rules apply — tables with FK dependencies on a skipped table are also skipped automatically.

Table Run Statuses

Each table in a run is recorded with one of the following statuses in the audit and process logs:

Output Modes

--ci / -q (quiet)

No stdout. Errors written to stderr with [ERROR] prefix. Exit code only.

Default (no flags)

Dot-style progress per table, with a summary at the end:

....F..........

  Warning: 1 table not found in source (audit_logs)
  Run failed: table "orders" — connection reset (120 rows skipped)

  Tables: 14/15  Rows: 48 302  Duration: 2m 14s
  Audit log: production-db_staging_2026-04-01T14-32-00Z_audit.html  ✓ delivered

Dot indicators:

• . — table transferred successfully
• F — table transferred with skipped rows
• E — table transfer failed
• ? — table not found in source
• S — skipped due to schema replication failure

-v (verbose)

One line per table with status and row count:

  ✓  Validating YAML ...
  ✓  Connecting to production-db and staging-db ...
  ✓  Replicating schema ...
  ✓  Resolving table order ...
  ✓  users     (12 340 rows)
  ✓  orders    (48 201 rows)
  ✗  audit_logs  — not found in source, skipped
  ✓  Generating audit log ...
  ✓  Delivering via local ...

-vv (very verbose)

Live streaming of run log events to stderr:

  ✓  users           12 340 rows   0 skipped
  ✓  orders          48 201 rows   0 skipped
  ✗  audit_logs      — not found in source, skipped
  ⚠  sessions        3 910 rows  120 skipped  (FK violation × 89, unique conflict × 31)

-vvv (debug)

Full debug output: schema diff, per-table progress bars with rows/sec and ETA, chunk-level events, skipped row details, and delivery channel responses:

  Transferring table: orders
  ████████████████████░░░░░  18 000 / 48 201  (~312 rows/sec, ~1m 31s left)

Progress bars are suppressed when --ci is active regardless of verbosity level.

Exit Codes

Audit & Process Logs

At the end of every successful run, Clonio produces two types of artefacts:

• Audit log (*_audit.html + *_audit.sig) — a signed, human-readable document listing all tables, transformations, and transfer counts. Suitable for compliance auditors.
• Process log (*_process.jsonl) — a structured, machine-readable JSONL execution log with per-table, per-chunk, and per-skipped-row events recorded during the run.

Files are named using the pattern:

{source}_{target}_{timestamp}_{type}.{ext}

For example:

production-db_staging_2026-04-01T14-32-00Z_audit.html
production-db_staging_2026-04-01T14-32-00Z_audit.sig
production-db_staging_2026-04-01T14-32-00Z_process.jsonl

What each channel delivers

Each channel delivers artefacts based on its type and any per-channel overrides in clonio.json:

Override the defaults for any individual channel with two optional boolean keys in its clonio.json entry:

{
  "audit": {
    "default": "local",
    "channels": {
      "local": {
        "type": "local",
        "path": "./",
        "delivers_audit": true,
        "delivers_process_log": false
      }
    }
  }
}

Delivery channels are configured in the audit block of clonio.json. The audit.use array (list of channel names) selects which channels actively deliver artefacts. Use --audit-channel=<list> to override audit.use for a single run. If audit is absent from clonio.json, all delivery is silently skipped.

Verify the integrity of a stored audit log with clonio cloning:verify-audit.

Related Commands

• cloning:dump — inspect a database and generate the .cloning.yaml file
• cloning:verify-audit — verify the integrity of a Clonio audit log
• matchers:init — export the baseline PII matchers for customisation
• connection:add — add a database connection