How to Smoothly Migrate Files to Office 365 SharePoint Online

Many clients ask me what they can do to make their Office 365 migration into SharePoint Online go as smoothly as possible. Often, they're sincerely looking for a way to reduce the costs for everyone involved.

In part 1 of this series, I'll focus mainly on the technical complexities of moving files from traditional shared folders into SharePoint Online. In future episodes (that's what my shrink calls them) we'll take a look at things from the architectural and strategic perspectives.

First, let me say that Office 365 customers come from all sorts and sizes of business. Some have IT staff supporting hundreds of users. Others are just a handful of folks with one or two office administrators - who often work double-duty as the IT help desk. Whether you're big or small, IT savvy or technophobic, on the bleeding edge or upgrading from Windows 3.1, planning to use commercially available migration tools or making your unpaid intern Carl copy all the files by hand, there's something here for you in the lessons we've learned.

Communication Plans
They say that communication is key. Like any other project, this is absolutely true with a major migration like moving into SharePoint Online. There are lots of things that probably should go into every communication plan. In a future part of this series, we'll tackle the fuzzy stuff. For now, let's focus on some of the hard technical details that many project managers forget to include when they coordinate an Office 365 migration.

Locked Files
It is important for users to understand that files which are open are likely to interfere with the migration procedure. This can even include folks who have a network share open in Windows Explorer on their desktop. While it's possible to work around these issues, during critical times, it might be a good idea to encourage users to close unused folders and documents in Office.

Note this is especially true for files open in MS Office, as it not only locks the file but also creates an auto-recovery file starting with ~$ that can't and shouldn't be migrated into SharePoint Online.

Backup Window
Often users will want access to their documents right up until the last minute. Whether you're using ZIP files, Acronis, or some other strategy to do a last minute backup - really any backup method, it is important to let the users know after what point their files will no longer have a backup.

File Move Timing and Locations
If you're doing everything under cover of darkness, then one day folks will come into the office to find all their files have been moved. If you're migrating during business hours (my personal preference is to not stay away all night babysitting a file move) then some users will still be working up until the last minute. Either way, you need to communicate to the users when the files will move and where they are going. This is especially true if you're doing some reorganization - whether elective or forced by Office 365 limitations.

Remember to Include in Your Communication Plans:
When will the move be taking place, including regular updates
Instructions about closing windows or office apps - or logging out completely
Advice on when files will no longer be backed up
Information about who to reach out to if there seem to be problems after the move


Technical Gotchas

Permissions


File permissions are one of those things that we've found can create a lot of problems when migrating to Office 365. Even though you may be using an administrator account with high level access, there may still be the occasional file buried deep in a folder structure that maybe was copied from a user's desktop and dragged very limited permissions along with it. Or, there may be entire folders that somebody locked down - sometimes with good reasons long forgotten about.

Obviously, you can't do much with these files if you can't get to them. It can be even more frustrating to find this kind of issue late in the game, when they'll blow up a move operation or a ZIP archive that's only halfway done.

Illegal Characters in File Names
One of those things that can be really vexing about moving to Office 365 is the limitations on the characters you can use in file names. The restrictions on Windows file names are not as stringent as they are on the web. So many users will have files with ampersands, hash-signs, and other "illegal" characters. But, I've also seen really exotic characters such as the registered trademark and copyright symbols, long-dash, and other unusual stuff.

Here's something to consider. When you get into Unicode and upper ASCII, the list of invalid characters can be quite long. If you're screening against the list of known illegal characters, you might let one slip through. Consider checking against known legal characters rather than known illegal ones.

Why is it that Office 365 has these restrictions?
Well, a lot of it has to do with the way the web actually works. For example, characters like “#”, “&”, and “?” All have special meanings when they are used in URLs. When you upload a file to SharePoint using the web site, these characters get converted to special codes that are safe in URLs; this process is called URL encoding. However, the encoded filenames look like unreadable gibberish when they're viewed through Explorer View (WebDAV) and other places, so we try to avoid using them at all.

Still other special characters may be using something called Unicode that allows the display of things like Chinese and Japanese characters alongside English. Unfortunately, web addresses use older standards called ANSI and ASCII which only allows English characters (and a few extended characters that are implemented so inconsistently across all the different computer systems that we really can't rely on them anyway).
Oh yeah, and these restrictions count for folders too! And while we're at it, did I also mention that there are certain names you're not allowed to give to a file in SharePoint, and certain words you cannot end the file name with? 

Renaming these files by hand is extremely tedious; it's so tedious that I actually made my son do this as a rite of passage when he first joined us here as an intern at Liquid Mercury Solutions. Needless to say he was very excited to have software that would do this instead.

Fortunately, you can use PowerShell, download a tool, or buy any of several products that will take care of this step for you.  You'll want to have some input into how files ought to be renamed, and the configurability will vary from solution to solution.

If you hire IT folks to do your migration for you, then they should have at least one of these methods.  Here at LMS, we generally use a combination of PowerShell and products from MetaLogix and ShareGate to complete Office 365 migrations. Other good choices come from MetaVis, Dell Software (formerly Quest Software), AvePoint and others.

File Name and Folder Structure
But, file names alone are not the only problem. There are additional limits.

For example, ZIP files have a limit in how long any given file name can be, so good luck backing your files up if you go past this threshold. (Some of this can be avoided by using WinRAR or newer versions of WinZip instead of the built in Windows ZIP file utility.)

Windows also has a slightly longer limit of no more than 248 characters in a file name and no more than 260 characters for the full file path including folders. This can be problematic if the file was copied to a network location using a mapped drive, but you need to access it using the longer UNC file share name.

SharePoint itself has a limit that the URL pointing to a file, including the DNS name of the server and path to the document library, can't be longer than 255 characters. So woe to you with lots and lots of sub-sites - or long domain names like liquidmercurysolutions.com.

Blocked File Types
There are a number of file types that are blocked in SharePoint Online. This is done for security reasons, including DLLs and executable files (EXEs).

It's worth noting that Microsoft has sole discretion on what file types are blocked. I was recently very surprised during a demo I tried to upload an MSI file to show that it would get blocked - and surprisingly it worked! So check to see what the latest list of block files is before you start; this can be done from the Office 365 management portal.

Miscellaneous Poor Fit for SharePoint
In the course of evaluating content, you're likely to encounter files or even whole folders that just aren't a good fit for the cloud. For example, thumbs.db and desktop.ini files are great for making Windows UI more useful, but they serve no real purpose in SharePoint - and they can lock the file system which can complicate moves. Likewise, Office recovery files (those that start with ~$) or their analog in WordPerfect (*.TMP) can be a nuisance since they're marked as hidden and system files.

Large files like AVIs, ISOs, and ZIPs may be a poor fit for SharePoint. While technically you can put up to a 2GB file into SharePoint Online, some folks just don't have the bandwidth to put the files into the cloud and also have them be useful. Even in circumstances where you're just archiving the file, you should consider that you might spend a very long time uploading it only to find that the operation will time out and the effort will be wasted. You best bet is to test your bandwidth, then decide where to draw the line.

A folder full of executable programs - and probably all the non-executable stuff in that folder too - is a poor fit for SharePoint. Use your own judgment, and don't be afraid to exclude certain parts of the folder structure altogether if they seem to support running software.

  • Before You Start: File Structure Check
  • Permissions
    • Take ownership of all the files you need to archive and/or move.
    • Make sure you have permissions to all the files by overwriting the current permissions.
  • Illegal Characters
    • Avoid &, ?, #, and others - get the complete list here
    • What tools are you using to rename files?
    • Check folders as well as files
    • Legal characters or illegal characters
  • File Types / Filename Extensions
    • All files must have an extension
  • Check for blocked file types - get the complete list here
    Length of Individual File Names
    • Beware any single file with a name longer than 60 characters; here there be dragons!
  • Best bet: try to keep them 50 characters or less
    Length of Folder Paths
    • Can your backup solution handle long file paths?
    • Check the windows path of all files to ensure it doesn't go over 248 characters
    • Check the intended destination URL for each file to ensure it isn't longer than 255 characters.
    • Best bet: try to keep them under 200 characters
  • Misc. "poor fit" for SharePoint
    • Delete desktop.ini, and thumbs.db files
    • Move or otherwise deal with any ~$*.* or *.TMP recovery files
    • Test your bandwidth to see if it supports very large files
    • Consider what to do with very large files such as AVI, ISO, ZIP, etc.
  • Optional, if using ZIP with legacy support:
    • Make sure your archives will not exceed the 4GB limit.
    • Make sure your archives will not have more than 65,535 files/folders total.

Office 365 and SharePoint Limitations

As if all this were not enough to consider, you're going to have a number of limits and options on top of this when moving into the SharePoint Online platform.

A lot of these impact choices about site and information architecture, which is where Liquid Mercury Solutions comes in to help our customers. Even if you're using custom tools, it's often best to have a SharePoint specialist who can help you choose the right way to organize your sites and files.

Here are a few of the not-so-fuzzy limitations that we often have to work around.

Begin Your Start: Check for SharePoint Limitations

  • User quotas - if you're not a site collection admin
  • No single file can be > 2 GB
  • No more than 5000 items in a single folder
  • No more than 100GB in a single site collection ("supported limit" is 200GB)
  • No more than 30,000,000 items total in a single document library, including all documents and folders
  • SkyDrive Pro has its own weird limits which are different


Other Best Practices


There are lots of other best practices to take into account when moving to SharePoint Online - so many that we'll have to save them for another episode in this series. So, tune in next time!

Meanwhile, here's some more information for you. We recently saw a very thorough presentation on this topic given to the Baltimore SharePoint User's Group by my friend Mark McGovern from MetaLogix. And, our colleagues over at ShareGate have written a good article that starts to scratch the surface on some of the complexities involved.

If you'd like to purchase tools to help with your Office 365 migration, or if you want a professional consultant's advice please contact us and we'll be happy to help you. If you find these tips helpful, and want to give migrating to SharePoint Online a try on your own, please consider joining our free support network. Whatever path you decide to take, we wish you the best of luck!

--------------------------------------------------------------------------------------------------------------------------------------------