Uniform Siphon - Sitecore data download

This guide walks you through the complete process of downloading content, media, and presentation details from Sitecore CMS for further migration to the Uniform platform.

  1. Intro
  2. Prepare work directory
  3. Prepare Sitecore
  4. Configure Siphon
  5. Download data
  6. Troubleshooting
  7. Next steps

The full migration process is described in the detailed guide here.

This guide only touches the first phase.

  • Ensure sufficient disk space for downloaded content
  • Be able to deploy a file to every Sitecore CD instance — that is where the migration service goes, and where item and media extraction runs
  • Have administrator access to the Sitecore CM server — download-presentation renders the Experience Editor there

Download `en` first — even if your content is not in English

Whatever languages your content lives in, the first download-items pass must be en, and it must cover at least:

  • /sitecore/templates — the templates Siphon turns into Uniform content types and components
  • /sitecore/system/Languages — the language definition items Siphon turns into Uniform locales

en is also the default value of --defaultLanguage — the context language Siphon uses to fetch the Sitecore language list and to resolve media items — so a download without en fights the defaults everywhere.

See Download data for the recommended pass order.

The wizard will create .env file with your settings that will be used by all Siphon commands.

Siphon comes with a built-in serilog logger. It creates several log files in the current directory:

C:\migration\ ├── ... ├── .siphon.log.debug.txt ├── .siphon.log.info.txt ├── .siphon.log.warn.txt ├── .siphon.log.error.txt └── ... other files

The log level is configured with SIPHON_LOGLEVEL (or --logLevel parameter) that takes these values:

  • DEBUG
  • INFO
  • WARN
  • ERROR

It is recommended to use the default INFO and switch to DEBUG only if something goes wrong.

Create a work directory for all files will be downloaded and processed, for example: C:\migration\.

Open a new PowerShell session and navigate there:

PS C:\> cd C:\migration PS C:\migration>

Before downloading data, you must deploy the migration service to your Sitecore instance.

  1. Rename C:\siphon\{ver}\UniformMigrationService.txt to UniformMigrationService.aspx
  2. Open UniformMigrationService.aspx
  3. Set a secure secret and valid until on Lines 5-7:
    // the default secret will not work, you must change it const string Secret = "(your-complex-secret-phrase-here)"; const string ValidUntil = "(date-which-is-several-month-in-the-future)"
  4. Save changes

Deploy the service to your CD instances — that is where item and media extraction should run.

  1. Upload the modified file to a subfolder like /layouts on your Sitecore CD instance
  2. Deploy it to every load-balanced CD instance. The load balancer then spreads Siphon's --threads workers across the whole farm instead of one box, which is the single biggest lever on extraction speed. A service reachable on only some instances is worse than useless: requests land on a node without the file and fail intermittently, so the run fails in a way that looks like a network problem
  3. Verify the page works by accessing it via the browser — it is expected to see a SecurityException

CD for extraction, CM for the Experience Editor

The two halves of a download go to different roles, and it is worth being deliberate about which is which:

commandinstancewhy
download-items, download-mediaCD--uniformServiceUrl. These read the web database, which a CD serves, and they are throughput-bound — spreading them over the CD farm is what makes them fast, and it keeps the load off the box your editors are using
download-presentationCM--host. Experience Editor markup (?sc_mode=edit) only renders on a Content Management instance — a CD has no editing pipelines and will return a server error for every page

Two consequences:

  • A run that pulls unpublished / draft content with --database master must point --uniformServiceUrl at the CM, because a CD has no master database. This is one more reason the recommended path is to publish everything before the content freeze and migrate from web only.
  • download-presentation's --productionHost is a CD host as well, but a different setting — it is the public site used for HTML comparison, not the service endpoint.

Important: Some folders may restrict *.aspx execution.

Also, in rare cases the IIS configuration may prohibit runtime compilation of *.aspx files. To overcome this, it needs to be deployed alongside with the application during normal deployment phase.

Uploading the file

On a CD instance, deploy the file the way you deploy anything else to that farm — your normal release pipeline, a file copy to each node, or a web deploy package. This also settles the runtime-compilation caveat above, since the file arrives with the application.

If you are deploying to a CM instead (a master pass, or a single-instance setup where CM and CD are the same box), the quickest route is the Import Language dialog in the Control panel:

https://www.loom.com/share/5dc22bf86cc244d48f207291e9c72692

In the current terminal window run configuration wizard and follow the instructions:

PS C:\migration> siphon configure-download

Here's example output:

======================================================================= Siphon Download Configuration Wizard ======================================================================= This wizard will help you configure the Siphon Download phase. It will generate a .env file with all necessary settings. Press Enter to accept values shown in [brackets]. Previous configuration loaded from: .env ----------------------------------------------------------------------- Work Directory ----------------------------------------------------------------------- Base directory for all downloaded files. Subdirectories will be created: data\items, data\media, data\presentation > Work Directory [default: C:\migration]: ----------------------------------------------------------------------- Common Settings (used by all download commands) ----------------------------------------------------------------------- The Sitecore database to download from (e.g., 'web' or 'master'). > Database [default: web]: Full URL to the deployed UniformMigrationService.aspx on your Sitecore CD. Deploy it to every load-balanced CD instance - that is what makes extraction fast. Example: https://cd.example.com/layouts/UniformMigrationService.aspx > Uniform Service URL: https://cd.example.com/layouts/UniformMigrationService.aspx The secret phrase configured in UniformMigrationService.aspx (Line 5). > Secret: ***** ----------------------------------------------------------------------- 1. Download Items Settings ----------------------------------------------------------------------- Root item GUID to start downloading from (with all descendants). It is typically the Home item. In a multi-site solutions there are 2 options: - Download sites one by one - Download everything by using root ID: 11111111-1111-1111-1111-111111111111 > Root Item ID: ITEM-ID-OF-YOUR-HOMEPAGE Languages to download (comma-separated, e.g., 'en,ja-JP,da'). Leave empty to download all registered languages. > Languages: Number of parallel download threads (1-16). > Threads [default: 4]: Download rendering datasource items outside of root? Unsure or in doubt? Use the default option. > Download Renderings Datasources [Y/n]: Download field-referenced items outside of root? Unsure or in doubt? Use the default option. > Download Fields References [Y/n]: ----------------------------------------------------------------------- 2. Download Media Settings ----------------------------------------------------------------------- Number of parallel download threads for media (1-16). > Threads [default: 4]: ----------------------------------------------------------------------- 3. Download Presentation Settings (Optional) ----------------------------------------------------------------------- Presentation download captures page layouts using Playwright. This is optional - skip if you only need content migration. Unsure or in doubt? Use the default option. > Configure presentation download? [Y/n]: Sitecore CM URL - the Experience Editor renders there, a CD cannot. Example: https://cm.example.com/ > Sitecore Host: https://cm.example.com Production host (CD instance) for production HTML comparison. Example: When host is https://cm.example.com then production host is https://www.example.com > Production Host: https://www.example.com Path to the site root item in Sitecore. Example: /sitecore/content/Habitat/Home > Start Path: /sitecore/content/example/Home Virtual path to download a sub-set of the website (optional). Example: /about-habitat (leave empty for entire site) > Virtual Path: Sitecore login page URL for authentication. Example: https://sc.dev/sitecore/login > Login URL: https://identity.example.com/sitecore/login Sitecore username for presentation download. > Username: admin Sitecore password for presentation download. > Password: * Sitecore <site> name from <sites> section of showconfig.aspx (or corresponding Site configuration item in SXA). Example: website > Site Name: example-site Number of parallel download threads (1-8). > Threads [default: 4]: Unsure or in doubt? Use the default option. > Download referenced assets (styles, static images etc.)? [Y/n]: [OK] Configuration saved to: .env ======================================================================= Configuration Summary ======================================================================= Work Directory: C:\migration\data Common: Database: web Service URL: https://cd.example.com/layouts/UniformMigrationService.aspx Secret: ******** Download Items: Root: ITEM-ID-OF-YOUR-HOMEPAGE Output: C:\migration\data\items Threads: 4 Download Media: Input: C:\migration\data\items Output: C:\migration\data\media Threads: 4 Download Presentation: Host: https://cm.example.com Prod Host: https://www.example.com Start Path: /sitecore/content/example/Home Login URL: https://identity.example.com/sitecore/login Threads: 4 ======================================================================= Next Steps: ======================================================================= 1. Copy the .env file to your migration working directory 2. Run the download commands: siphon download-items siphon download-media siphon download-presentation

The Languages prompt

Leaving Languages empty downloads all registered languages, which includes en — that is the safe answer. If you do list languages explicitly, en must be one of them, and it must be downloaded first. See Always download en first.

First, download content items:

PS C:\migration> siphon download-items

Verify the command returned no critical errors and exit code is 0. If the error code is not 0, halt and contact Uniform migration team.

All download commands support resuming downloading: if it failed after some time of running well, it won't try to re-download what's already there.

Every ten seconds each download command — items, media and presentation alike — prints what it is achieving: what the last window transferred, what the run has averaged, and how many workers are busy:

[stats] last 10s: 152 items, 673.8 KB, 336.2 KB/s, 76 items/s | total 00:16: 725 items, 3.3 MB, 213.5 KB/s, 45 items/s | threads 4

If it is too slow, or the Sitecore instance is visibly struggling, you don't have to stop and restart with a different --threads. Put a settings-override.json in the folder you run Siphon from and the thread count changes within ten seconds:

{ "threads": 8 }

Or add --autotune and let Siphon search for the number itself, keeping each step only when it actually made the download faster.

The same file can put the download on hold — useful when Sitecore needs to be restarted, or when a publish is running and you want to stop hammering it:

{ "pause": true }

Workers finish what they are fetching and park; nothing new is requested until you take the key back out (checked every second), and the queue and everything already downloaded are kept. The log keeps saying still PAUSED every ten seconds while it holds, so a paused run never looks like a stuck one. All three are described in monitoring and tuning a running download.

Skipping page-local datasource folders#

Many Sitecore solutions keep the datasources that belong to a single page in a folder under that page — usually called _local, sometimes data. On a long-lived site those folders are often the majority of the item count, and much of what is in them is no longer pointed at by anything: datasources of components that were removed from the page, leftovers of old designs, experiment variants.

--skipLocalDatasources stops the crawl from walking into them:

PS C:\migration> siphon download-items --skipLocalDatasources

Only a child of a page is skipped — an item that has a layout. That distinction matters: Data is also the usual name of a shared, site-wide data folder such as /sitecore/content/MySite/Data, and that one holds content the migration needs, so it keeps being downloaded. If your solution names its page-local folders something else, list them with --localDatasourceFolderNames (the value replaces _local and data rather than adding to them).

Each language pass reports what it left out, so a pruned download never looks like a download that lost items:

--skipLocalDatasources in en: 1483 local datasource folder(s) (_local, data) under a page, and everything below them, were not downloaded.

Pair it with `--downloadRenderingsDatasources`

A page-local datasource holds the text and images of a component on that page — dropping one silently empties that component after migration. The switch only prunes the tree walk, so with --downloadRenderingsDatasources --downloadFieldsReferences on (the recommended setting anyway) every datasource a live rendering or a reference field actually points at is still fetched by id, and what is left behind is the part of the folder nothing points at. Without those two switches, --skipLocalDatasources will lose page content.

If you left the Languages prompt empty, every registered language — including en — is downloaded in one pass and you are done: skip to the media download.

If you restrict languages (with --languages or SIPHON_LANGUAGES), run en first and only then the other languages:

# 1. en — always first PS C:\migration> siphon download-items --languages en # 2. the languages your content actually lives in PS C:\migration> siphon download-items --languages "da,ja-JP" --skipClearingFolders

Note --skipClearingFolders on every pass after the first — without it each run clears the output folder and throws away what the previous pass downloaded.

The en pass must cover /sitecore/templates and /sitecore/system/Languages. Downloading from the all-content root 11111111-1111-1111-1111-111111111111 already includes them. If instead you download site by site starting at a Home item, add a dedicated en pass with --root for each of those two subtrees before the content passes — their standard Sitecore IDs are 3C1715FE-6A13-4FCF-845F-DE308BA9741D (/sitecore/templates) and 64C4F646-A3FA-4205-B98A-4405DD8C1743 (/sitecore/system/Languages), and you can confirm them in the Sitecore Content Editor.

Verify the `en` pass landed

After step 1, check that data\items\content\en\ and data\items\core\en\ exist — or data\items\en\ on an export that predates --itemFolders — and that data\items\_languages.json lists every language you expect to migrate. An empty or missing en means the later phases have no templates and no locales to work from — fix it before downloading anything else.

Next, run two remaining commands.

PS C:\migration> siphon download-media PS C:\migration> siphon download-presentation

download-media is a download-items pass before it is a binary fetch, and with --itemFolders it writes only the media folder and its index — it no longer rewrites the index the content download produced. So the content tree can be packaged or published while media is still downloading.

download-presentation is a different matter: it reads the item export to work out which items are pages, and it merges every folder's index to do so. Run it after download-media has finished, or give it a copy of the item export taken before the media download started (--inputDir).

When everything is complete, review the downloaded files:

C:\migration\ ├── items\ │ ├── _languages.json # Available languages │ ├── .folders.json # The --itemFolders rule set this export was written with │ └── {folder}\ # content | media | core | other — see --itemFolders │ └── {language}\ │ ├── {0-f}\ │ | └── {0-f}\ │ | └── {item-id}.json # Individual items │ ├── _index.json # This folder's index, for this language │ └── _map.json # Path → id lookup for the same ├── media\ │ └── ...Binary media files └── presentation\ ├── test-results # For azure devops reporting tool └── {language}\ ├── {first-level-children}\ │ ├── {second-level-children}\ │ │ ├── .... │ │ ├── index.html # page raw output via Experience Editor │ │ ├── index.json # page structure parsed from index.html │ │ └── prod.html # page raw output from production website │ ├── index.html │ ├── index.json │ └── prod.html ├── index.html ├── index.json └── prod.html

Each item folder keeps its own index, so the commands that write different trees no longer contend over one file — see --itemFolders for the rules, and for how to read or convert an export that predates the split.

An export downloaded before --itemFolders existed has no {folder} level — items sit directly under items\{language}\ — and is still read exactly as it was. Nothing needs converting.

  1. UniformMigrationService.aspx not accessible

    • Check IIS configuration
    • Verify file permissions
    • Behind a load balancer, confirm the file is deployed to every CD node — if one is missing it, requests fail only when the balancer routes there, which looks like an intermittent network fault rather than a missing file

  2. Download interruptions

    • Use resume functionality
    • Check network connectivity
    • Run outside of peak hours of content editing
    • Reduce thread count if experiencing timeouts — without restarting the run

  3. download-presentation returns a server error page for every page

    • Check --host points at the CM. Experience Editor markup does not render on a CD, so every page comes back as an error page and is reported as "has no placeholders and renderings detected"
    • Set --siteName to the Sitecore <site> name. Without it the site is resolved from the hostname, which on a multi-site solution may resolve to a site that does not contain your start path

  4. Authentication failures in presentation download

    • Verify Sitecore credentials
    • Check login URL accessibility
    • Ensure Playwright is properly installed

For additional support, contact Uniform migration team with DEBUG log file.

When downloading is completed, provide the C:\migration\ directory to Uniform migration team.

Alternatively, you can complete migration yourself:

Last modified
Next
Walkthough