# Uniform Siphon - Sitecore data download > Understanding the Sitecore data download process Source: https://docs.uniform.app/docs/guides/migration/sitecore/download This guide walks you through the complete process of downloading content, media, and presentation details from Sitecore CMS for further migration to the Uniform platform. ## Table of Contents 1. [Intro](#intro) 2. [Prepare work directory](#prepare-work-directory) 3. [Prepare Sitecore](#prepare-sitecore) 4. [Configure Siphon](#configure-siphon) 5. [Download data](#download-data) 6. [Troubleshooting](#troubleshooting) 7. [Next steps](#next-steps) --- ## Intro The full migration process is described in the [detailed guide here](./sitecore/walkthrough). This guide only touches the first phase. ### Before you begin - Ensure sufficient disk space for downloaded content - Be able to deploy a file to every Sitecore **CD** instance — that is where the migration service goes, and where item and media extraction runs - Have administrator access to the Sitecore **CM** server — `download-presentation` renders the Experience Editor there ### Always download `en` first > **Download `en` first — even if your content is not in English:** > > Whatever languages your content lives in, the **first** `download-items` pass must be `en`, and it must cover at least: > > - `/sitecore/templates` — the templates Siphon turns into Uniform content types and components > - `/sitecore/system/Languages` — the language definition items Siphon turns into Uniform locales `en` is also the default value of [`--defaultLanguage`](https://docs.uniform.app/docs/guides/migration/sitecore/common-reference#languages) — the context language Siphon uses to fetch the Sitecore language list and to resolve media items — so a download without `en` fights the defaults everywhere. See [Download data](#download-data) for the recommended pass order. ### Environment variables The wizard will create `.env` file with your settings that will be used by all Siphon commands. ### Logging Siphon comes with a built-in serilog logger. It creates several log files in the current directory: ``` C:\migration\ ├── ... ├── .siphon.log.debug.txt ├── .siphon.log.info.txt ├── .siphon.log.warn.txt ├── .siphon.log.error.txt └── ... other files ``` The log level is configured with `SIPHON_LOGLEVEL` (or `--logLevel` parameter) that takes these values: - DEBUG - INFO - WARN - ERROR It is recommended to use the default `INFO` and switch to `DEBUG` only if something goes wrong. ## Prepare work directory Create a work directory for all files will be downloaded and processed, for example: `C:\migration\`. Open a new PowerShell session and navigate there: ```powershell PS C:\> cd C:\migration PS C:\migration> ``` ## Prepare Sitecore Before downloading data, you must deploy the migration service to your Sitecore instance. ### Locate the Service File 1. Rename `C:\siphon\{ver}\UniformMigrationService.txt` to `UniformMigrationService.aspx` 2. Open `UniformMigrationService.aspx` 3. Set a secure secret and valid until on Lines 5-7: ```csharp // the default secret will not work, you must change it const string Secret = "(your-complex-secret-phrase-here)"; const string ValidUntil = "(date-which-is-several-month-in-the-future)" ``` 4. Save changes ### Deploy to Sitecore Deploy the service to your **CD** instances — that is where item and media extraction should run. 1. Upload the modified file to a subfolder like `/layouts` on your Sitecore CD instance 2. **Deploy it to every load-balanced CD instance.** The load balancer then spreads Siphon's `--threads` workers across the whole farm instead of one box, which is the single biggest lever on extraction speed. A service reachable on only some instances is worse than useless: requests land on a node without the file and fail intermittently, so the run fails in a way that looks like a network problem 3. Verify the page works by accessing it via the browser — it is expected to see a SecurityException > **CD for extraction, CM for the Experience Editor:** > > The two halves of a download go to different roles, and it is worth being deliberate about which is which: > > | command | instance | why | > | --- | --- | --- | > | `download-items`, `download-media` | **CD** | `--uniformServiceUrl`. These read the **`web`** database, which a CD serves, and they are throughput-bound — spreading them over the CD farm is what makes them fast, and it keeps the load off the box your editors are using | > | `download-presentation` | **CM** | `--host`. Experience Editor markup (`?sc_mode=edit`) only renders on a Content Management instance — a CD has no editing pipelines and will return a server error for every page | > > Two consequences: > > - A run that pulls **unpublished / draft** content with `--database master` must point `--uniformServiceUrl` at the **CM**, because a CD has no `master` database. This is one more reason the [recommended path](https://docs.uniform.app/docs/guides/migration/sitecore/download-reference#download-items) is to publish everything before the content freeze and migrate from `web` only. > - `download-presentation`'s `--productionHost` is a CD host as well, but a _different_ setting — it is the public site used for HTML comparison, not the service endpoint. > **Important**: Some folders may restrict `*.aspx` execution. > > Also, in rare cases the IIS configuration may prohibit runtime compilation of *.aspx files. To overcome this, it needs to be deployed alongside with the application during normal deployment phase. > **Uploading the file:** > > On a CD instance, deploy the file the way you deploy anything else to that farm — your normal release > pipeline, a file copy to each node, or a web deploy package. This also settles the runtime-compilation > caveat above, since the file arrives with the application. > > If you are deploying to a **CM** instead (a `master` pass, or a single-instance setup where CM and CD > are the same box), the quickest route is the Import Language dialog in the Control panel: > > https://www.loom.com/share/5dc22bf86cc244d48f207291e9c72692 ## Configure Siphon In the current terminal window run configuration wizard and follow the instructions: ```powershell PS C:\migration> siphon configure-download ``` Here's example output: ```powershell ======================================================================= Siphon Download Configuration Wizard ======================================================================= This wizard will help you configure the Siphon Download phase. It will generate a .env file with all necessary settings. Press Enter to accept values shown in [brackets]. Previous configuration loaded from: .env ----------------------------------------------------------------------- Work Directory ----------------------------------------------------------------------- Base directory for all downloaded files. Subdirectories will be created: data\items, data\media, data\presentation > Work Directory [default: C:\migration]: ----------------------------------------------------------------------- Common Settings (used by all download commands) ----------------------------------------------------------------------- The Sitecore database to download from (e.g., 'web' or 'master'). > Database [default: web]: Full URL to the deployed UniformMigrationService.aspx on your Sitecore CD. Deploy it to every load-balanced CD instance - that is what makes extraction fast. Example: https://cd.example.com/layouts/UniformMigrationService.aspx > Uniform Service URL: https://cd.example.com/layouts/UniformMigrationService.aspx The secret phrase configured in UniformMigrationService.aspx (Line 5). > Secret: ***** ----------------------------------------------------------------------- 1. Download Items Settings ----------------------------------------------------------------------- Root item GUID to start downloading from (with all descendants). It is typically the Home item. In a multi-site solutions there are 2 options: - Download sites one by one - Download everything by using root ID: 11111111-1111-1111-1111-111111111111 > Root Item ID: ITEM-ID-OF-YOUR-HOMEPAGE Languages to download (comma-separated, e.g., 'en,ja-JP,da'). Leave empty to download all registered languages. > Languages: Number of parallel download threads (1-16). > Threads [default: 4]: Download rendering datasource items outside of root? Unsure or in doubt? Use the default option. > Download Renderings Datasources [Y/n]: Download field-referenced items outside of root? Unsure or in doubt? Use the default option. > Download Fields References [Y/n]: ----------------------------------------------------------------------- 2. Download Media Settings ----------------------------------------------------------------------- Number of parallel download threads for media (1-16). > Threads [default: 4]: ----------------------------------------------------------------------- 3. Download Presentation Settings (Optional) ----------------------------------------------------------------------- Presentation download captures page layouts using Playwright. This is optional - skip if you only need content migration. Unsure or in doubt? Use the default option. > Configure presentation download? [Y/n]: Sitecore CM URL - the Experience Editor renders there, a CD cannot. Example: https://cm.example.com/ > Sitecore Host: https://cm.example.com Production host (CD instance) for production HTML comparison. Example: When host is https://cm.example.com then production host is https://www.example.com > Production Host: https://www.example.com Path to the site root item in Sitecore. Example: /sitecore/content/Habitat/Home > Start Path: /sitecore/content/example/Home Virtual path to download a sub-set of the website (optional). Example: /about-habitat (leave empty for entire site) > Virtual Path: Sitecore login page URL for authentication. Example: https://sc.dev/sitecore/login > Login URL: https://identity.example.com/sitecore/login Sitecore username for presentation download. > Username: admin Sitecore password for presentation download. > Password: * Sitecore name from section of showconfig.aspx (or corresponding Site configuration item in SXA). Example: website > Site Name: example-site Number of parallel download threads (1-8). > Threads [default: 4]: Unsure or in doubt? Use the default option. > Download referenced assets (styles, static images etc.)? [Y/n]: [OK] Configuration saved to: .env ======================================================================= Configuration Summary ======================================================================= Work Directory: C:\migration\data Common: Database: web Service URL: https://cd.example.com/layouts/UniformMigrationService.aspx Secret: ******** Download Items: Root: ITEM-ID-OF-YOUR-HOMEPAGE Output: C:\migration\data\items Threads: 4 Download Media: Input: C:\migration\data\items Output: C:\migration\data\media Threads: 4 Download Presentation: Host: https://cm.example.com Prod Host: https://www.example.com Start Path: /sitecore/content/example/Home Login URL: https://identity.example.com/sitecore/login Threads: 4 ======================================================================= Next Steps: ======================================================================= 1. Copy the .env file to your migration working directory 2. Run the download commands: siphon download-items siphon download-media siphon download-presentation ``` > **The Languages prompt:** > > Leaving **Languages** empty downloads all registered languages, which includes `en` — that is the safe answer. If you do list languages explicitly, `en` must be one of them, and it must be downloaded first. See [Always download `en` first](#always-download-en-first). ## Download data First, download content items: ```powershell PS C:\migration> siphon download-items ``` Verify the command returned no critical errors and exit code is `0`. If the error code is not `0`, halt and contact Uniform migration team. All download commands support resuming downloading: if it failed after some time of running well, it won't try to re-download what's already there. ### Watch the download while it runs Every ten seconds each download command — items, media and presentation alike — prints what it is achieving: what the last window transferred, what the run has averaged, and how many workers are busy: ``` [stats] last 10s: 152 items, 673.8 KB, 336.2 KB/s, 76 items/s | total 00:16: 725 items, 3.3 MB, 213.5 KB/s, 45 items/s | threads 4 ``` If it is too slow, or the Sitecore instance is visibly struggling, you don't have to stop and restart with a different `--threads`. Put a `settings-override.json` in the folder you run Siphon from and the thread count changes within ten seconds: ```json { "threads": 8 } ``` Or add `--autotune` and let Siphon search for the number itself, keeping each step only when it actually made the download faster. The same file can put the download on hold — useful when Sitecore needs to be restarted, or when a publish is running and you want to stop hammering it: ```json { "pause": true } ``` Workers finish what they are fetching and park; nothing new is requested until you take the key back out (checked every second), and the queue and everything already downloaded are kept. The log keeps saying `still PAUSED` every ten seconds while it holds, so a paused run never looks like a stuck one. All three are described in [monitoring and tuning a running download](https://docs.uniform.app/docs/guides/migration/sitecore/download-reference#monitoring-and-tuning-a-running-download). ### Skipping page-local datasource folders Many Sitecore solutions keep the datasources that belong to a single page in a folder under that page — usually called `_local`, sometimes `data`. On a long-lived site those folders are often the majority of the item count, and much of what is in them is no longer pointed at by anything: datasources of components that were removed from the page, leftovers of old designs, experiment variants. `--skipLocalDatasources` stops the crawl from walking into them: ```powershell PS C:\migration> siphon download-items --skipLocalDatasources ``` Only a child of a **page** is skipped — an item that has a layout. That distinction matters: `Data` is also the usual name of a shared, site-wide data folder such as `/sitecore/content/MySite/Data`, and that one holds content the migration needs, so it keeps being downloaded. If your solution names its page-local folders something else, list them with `--localDatasourceFolderNames` (the value replaces `_local` and `data` rather than adding to them). Each language pass reports what it left out, so a pruned download never looks like a download that lost items: ``` --skipLocalDatasources in en: 1483 local datasource folder(s) (_local, data) under a page, and everything below them, were not downloaded. ``` > **Pair it with `--downloadRenderingsDatasources`:** > > A page-local datasource holds the text and images of a component on that page — dropping one silently empties that component after migration. The switch only prunes the **tree walk**, so with `--downloadRenderingsDatasources --downloadFieldsReferences` on (the recommended setting anyway) every datasource a live rendering or a reference field actually points at is still fetched by id, and what is left behind is the part of the folder nothing points at. Without those two switches, `--skipLocalDatasources` will lose page content. ### Language pass order If you left the **Languages** prompt empty, every registered language — including `en` — is downloaded in one pass and you are done: skip to the media download. If you restrict languages (with `--languages` or `SIPHON_LANGUAGES`), run `en` **first** and only then the other languages: ```powershell # 1. en — always first PS C:\migration> siphon download-items --languages en # 2. the languages your content actually lives in PS C:\migration> siphon download-items --languages "da,ja-JP" --skipClearingFolders ``` Note `--skipClearingFolders` on every pass after the first — without it each run clears the output folder and throws away what the previous pass downloaded. The `en` pass must **cover** `/sitecore/templates` and `/sitecore/system/Languages`. Downloading from the all-content root `11111111-1111-1111-1111-111111111111` already includes them. If instead you download site by site starting at a **Home** item, add a dedicated `en` pass with `--root` for each of those two subtrees before the content passes — their standard Sitecore IDs are `3C1715FE-6A13-4FCF-845F-DE308BA9741D` (`/sitecore/templates`) and `64C4F646-A3FA-4205-B98A-4405DD8C1743` (`/sitecore/system/Languages`), and you can confirm them in the Sitecore Content Editor. > **Verify the `en` pass landed:** > > After step 1, check that `data\items\content\en\` and `data\items\core\en\` exist — or `data\items\en\` on an export that predates [`--itemFolders`](https://docs.uniform.app/docs/guides/migration/sitecore/common-reference#item-folders) — and that `data\items\_languages.json` lists every language you expect to migrate. An empty or missing `en` means the later phases have no templates and no locales to work from — fix it before downloading anything else. ### Remaining downloads Next, run two remaining commands. ```powershell PS C:\migration> siphon download-media PS C:\migration> siphon download-presentation ``` `download-media` is a `download-items` pass before it is a binary fetch, and with [`--itemFolders`](https://docs.uniform.app/docs/guides/migration/sitecore/common-reference#item-folders) it writes only the `media` folder and its index — it no longer rewrites the index the content download produced. So the content tree can be packaged or published while media is still downloading. `download-presentation` is a different matter: it reads the item export to work out which items are pages, and it merges **every** folder's index to do so. Run it after `download-media` has finished, or give it a copy of the item export taken before the media download started (`--inputDir`). When everything is complete, review the downloaded files: ``` C:\migration\ ├── items\ │ ├── _languages.json # Available languages │ ├── .folders.json # The --itemFolders rule set this export was written with │ └── {folder}\ # content | media | core | other — see --itemFolders │ └── {language}\ │ ├── {0-f}\ │ | └── {0-f}\ │ | └── {item-id}.json # Individual items │ ├── _index.json # This folder's index, for this language │ └── _map.json # Path → id lookup for the same ├── media\ │ └── ...Binary media files └── presentation\ ├── test-results # For azure devops reporting tool └── {language}\ ├── {first-level-children}\ │ ├── {second-level-children}\ │ │ ├── .... │ │ ├── index.html # page raw output via Experience Editor │ │ ├── index.json # page structure parsed from index.html │ │ └── prod.html # page raw output from production website │ ├── index.html │ ├── index.json │ └── prod.html ├── index.html ├── index.json └── prod.html ``` Each item folder keeps **its own index**, so the commands that write different trees no longer contend over one file — see [`--itemFolders`](https://docs.uniform.app/docs/guides/migration/sitecore/common-reference#item-folders) for the rules, and for how to read or convert an export that predates the split. An export downloaded before `--itemFolders` existed has no `{folder}` level — items sit directly under `items\{language}\` — and is still read exactly as it was. Nothing needs converting. --- ## Troubleshooting ### Common Issues 1. **UniformMigrationService.aspx not accessible** - Check IIS configuration - Verify file permissions - Behind a load balancer, confirm the file is deployed to **every** CD node — if one is missing it, requests fail only when the balancer routes there, which looks like an intermittent network fault rather than a missing file 2. **Download interruptions** - Use resume functionality - Check network connectivity - Run outside of peak hours of content editing - Reduce thread count if experiencing timeouts — [without restarting the run](#watch-the-download-while-it-runs) 3. **`download-presentation` returns a server error page for every page** - Check `--host` points at the **CM**. Experience Editor markup does not render on a CD, so every page comes back as an error page and is reported as _"has no placeholders and renderings detected"_ - Set `--siteName` to the Sitecore `` name. Without it the site is resolved from the hostname, which on a multi-site solution may resolve to a site that does not contain your start path 4. **Authentication failures in presentation download** - Verify Sitecore credentials - Check login URL accessibility - Ensure Playwright is properly installed For additional support, contact Uniform migration team with `DEBUG` log file. ## Next Steps When downloading is completed, provide the `C:\migration\` directory to Uniform migration team. Alternatively, you can complete migration yourself: - [Detailed guide here](https://docs.uniform.app/docs/guides/migration/sitecore/walkthrough)