Handling Large File Transfers in SharePoint Online Migration using PnP PowerShell

Migrating large files to SharePoint Online can be challenging due to file size limitations, throttling, and connectivity issues. Using PnP PowerShell, you can efficiently transfer large files while ensuring data integrity, performance optimization, and error handling.

Key Challenges in Large File Transfers

✔ File Size Limits – SharePoint Online supports files up to 250 GB, but handling such large files requires special methods.
✔ Throttling Issues – Microsoft limits API calls, affecting upload speed.
✔ Interrupted Transfers – Network failures can cause file corruption or incomplete uploads.
✔ Metadata Preservation – Ensuring metadata like timestamps and authorship is retained.

Step 1: Connect to SharePoint Online

Before transferring files, establish a connection to SharePoint Online.

powershellCopyEdit$siteUrl = "https://yourtenant.sharepoint.com/sites/YourSite"
Connect-PnPOnline -Url $siteUrl -Interactive

✔ Ensures authentication for uploading large files.

Step 2: Prepare the Large File for Upload

If your file is smaller than 250 MB, you can upload it normally:

$localFile = "C:\LargeFiles\Report.pdf"
$library = "Documents"

Add-PnPFile -Path $localFile -Folder $library

✔ Works for small to medium files.

Step 3: Upload Large Files in Chunks (Over 250 MB)

For files larger than 250 MB, use chunked uploads.

$largeFile = "C:\LargeFiles\BigData.zip"
$targetLibrary = "Documents"

Add-PnPFile -Path $largeFile -Folder $targetLibrary -ChunkSize 10MB

✔ Splits files into 10MB chunks for smooth upload.
✔ Prevents timeout and API throttling issues.

Step 4: Monitor Upload Progress and Handle Failures

Create a robust upload script with error handling.

$files = Get-ChildItem "C:\LargeFiles"

foreach ($file in $files) {
    try {
        Write-Host "Uploading: $($file.Name)..."
        Add-PnPFile -Path $file.FullName -Folder "Documents" -ChunkSize 10MB
        Write-Host "Upload Successful: $($file.Name)"
    } catch {
        Write-Host "Upload Failed: $($file.Name) - Error: $_"
    }
}

✔ Logs progress and handles failed uploads gracefully.

Step 5: Validate File Integrity After Upload

Compare the file size of uploaded files with local files.

$uploadedFile = Get-PnPFile -Url "/sites/YourSite/Shared Documents/BigData.zip" -AsFile -Path "C:\Validation\BigData.zip"

if ((Get-Item "C:\LargeFiles\BigData.zip").Length -eq (Get-Item "C:\Validation\BigData.zip").Length) {
    Write-Host "File Transfer Verified: BigData.zip"
} else {
    Write-Host "File Transfer Failed: Size Mismatch!"
}

✔ Ensures file integrity after upload.

Step 6: Optimize Migration for Large Files

To avoid throttling, adjust migration settings:

Set-PnPTraceLog -On -Level Debug  # Enable logging
$ProgressPreference = "SilentlyContinue"  # Speed up transfer by suppressing progress messages

✔ Prevents API rate limits and performance slowdowns.

Step 7: Automate Large File Migration

Schedule migration using Task Scheduler for non-business hours.

$taskAction = New-ScheduledTaskAction -Execute "PowerShell.exe" -Argument "C:\Scripts\UploadLargeFiles.ps1"
Register-ScheduledTask -TaskName "LargeFileMigration" -Action $taskAction -Trigger (New-ScheduledTaskTrigger -Daily -At 2AM) -User "YourAdminAccount" -RunLevel Highest

✔ Automates large file transfers during off-peak hours.