CloudPrep 2014 Development Update

I wanted to take a few minutes today to talk about what we've been doing since late January in regards to CloupPrep and the PowerShell commands for file migration and management of SharePoint Online.

First thing I can say is that one of our most difficult choices was in choosing an e-commerce platform and licensing API to use for our product. Even though we plan to keep our licensing fairly simply, we wanted to have options for future products and well as many of the items we also sell through our partners.

This turned out to be more challenging than I imagined, but we have settled down on using Fast Spring and LogicNP Crypto License. Perhaps in some future post I will talk about those more from a software developer's perspective. What I can say today is that it will be at least a couple weeks before we can get a working prototype of the licensing server and the store online, and so we have had to push back release closer to the end of March or early April, mostly for that reason.

Meanwhile, we have been developing features for the different editions of CloudPrep 2014. Progress on that front continues at a rapid pace and I am pretty satisfied with the way our tools are maturing.

When we decided to produce this software, we planned to release the lite and standard editions first and follow up with premium and professional features later this spring. I was a bit surprised to see that where we are putting our development efforts, probably all four editions of CloudPrep will be available at one time.

Now for the geeky stuff. Here's some of what's been happening as we've been building.

Features we've essentially completed:

  • Upload an entire folder or specific set files to document library
    We've tested that these commands will work against network drives and UNC paths. Take that, OneDrive!
  • Preserve metadata about the local file system that the document was uploaded from
  • Create and Modified dates on files are preserved, though we did find that with larger files there are limits to what we can accomplish here
  • You can specify the content type for uploaded files, root folders, and sub-folders - including Document Set and its child content types
  • A bunch of other random stuff including commands for manipulating SharePoint lists and reports to make sure that file uploads won't exceed SharePoint limits


We noticed that Office 365 throws us a lot of connectivity errors that we don't normally see in on-premises SharePoint environments. If you've been trying to copy files using their standard UI or using OneDrive, some of these errors might be hidden from you. However, they're readily apparent if you're using Web Folders (WebDAV) or Client Side Object Model to connect. We see unexpected dropped connections quite often, and certain upload methods will time out on files that are too big and required some fun workarounds. There are different methods needed for files under 2MB, under 35MB, and larger.

Our path was also complicated by the fact that on certain Office 365 sites, our rights come from delegated admin privileges. This is the preferred way that consultants get their rights to help clients manage SharePoint Online, so we figure a lot of folks who are interested in CloudPrep are seeing this phenomenon as well. When you log in with delegated admin to a client's Office 365 site using the credentials from your own Office 365 account, you sometimes see the access denied page; login again a few seconds later, everything is fine. Our code had to expect and handle this contingency.

Another thing that we did not expect is that we're seeing some reasonable evidence that Office 365 uploads are being throttled. Most of the time, file transfers seem to be limited to about 300KB/sec; there are days when the transfer speed is even slower than that, sometimes by half. As such, it is difficult for us to estimate file upload times, and we're having to improve our algorithms to take these fluctuations and sea changes into account.

As for the cause, we can't say if this is something Microsoft is doing, or if it comes from the erosion of net neutrality. We do wonder if Comcast or other providers may be limiting traffic to Office 365 in order to give their own offerings a competitive advantage or just to control their own costs. I expect we'll be doing some tests in the near future, and we've been kicking around some ways to circumvent these bandwidth caps - at least partially. One test we did in January showed that if we took half our files to a different physical location, we were able to upload them to SharePoint Online in about half the time it would have taken if we'd uploaded them all from one server.

One thing that became clear during early development was that the disconnected nature of cloud storage was going to introduce multiple random problems along the way. As a result, in any large set of documents to be moved to the cloud, there would be some which for one reason or another may not be successfully copied. We started by trying to get this failure rate as low as possible, down to less that 0.25% of files in most cases. We did a lot of work in early February to improve the code and reach this threshold.

Even so, we needed to be able to easily run multiple passes on any file copy operation and track the results. Our first prototypes had to crawl the Document Library in SharePoint one folder and file at a time. This proved to be incredibly slow, and it quickly became apparent that we needed to be able to gather status information for thousands of files at a time if we wanted to hone in on only those which required an update from the local copy. This is something we added to the code base about a week ago, and we're now in the process of replacing some of our early code to use the new file comparison analytics logic.

As a side note, a bevy of SharePoint management features found their way into our PowerShell library simply because we had customers who needed them in short order. For example, we now have the ability to take a View from any SharePoint List and make a copy of it in the same List or a different one even on another SharePoint Site. Of course, one must be very careful with this kind of power, since creating Views with field references that don't exist in the List will certainly break the View if not the entire List itself. When we've added sufficient safety checks, we'll open the capability up as part of the CloudPrep product.

This week, we introduced the concept of using a hash algorithm to test whether files in SharePoint match those on our local drive. Use of a hash in addition to checking the file size and date stamps of a document ensures that the document has been uploaded into SharePoint and that it has not been corrupted in the process. We developed this ability in order to add credibility to Office 365 migrations where we may be moving hundreds of thousands or even millions of files, and we need to establish that the migration process has been completed satisfactorily. This capability can also be used to perform duplicate file detection, and we may develop a follow on product or feature to do just that later on.

Next week, we're planning to work on some important features that we feel are a must for getting this product to where we want it to be.

The first is the make sure that we can translate between Active Directory permissions on the local file system and users in SharePoint. The primary purpose here is to preserve meaningful data for Created By and Modified By fields in SharePoint; this is something we can't do yet. As part of this process, we'll be introducing PowerShell commands to add new users into SharePoint sites and manage groups. For most customers, this is probably of limited use. However, those with several hundred users or groups to manage will find it much easier to deal with these via PowerShell instead of using the SharePoint admin web pages. For consultants, it will make migrations faster by speeding up the time it takes to implement the security configuration. Our goal here is to lower the cost of our migration services.

The next things we do after that will be:

  • Download documents from SharePoint to the local drive
  • Assign metadata from CSV file as you upload documents
  • Flatten a folder structure as you upload it.


These are harder to do than you might think. I'll post more on this in coming weeks, including our challenges and progress updates.

Anouncing CloudPrep 2014 Migration Toolkit for SharePoint Online

We do a lot of Office 365 migrations. Most of these are for businesses with fewer than 50 employees. This should surprise nobody except maybe Microsoft, who seemed to be slow to realize that their cloud platform would have the most appeal to companies with limited budgets – or that most jobs in the US are provided by small businesses. Go figure.

Over the years, I’ve written several times about the challenges of moving from a conventional file store to Office 365. Fact is, it’s just not simple to do. It really makes sense to have an experienced IT professional help you make the move. I like helping customers make the switch, but doing so has presented interesting challenges for my business that I’m sure other SharePoint consultants share too.

Firstly, there are great third party tools out there for migrating files. We often use ShareGate and Content Matrix from MetaLogix. MetaVis is another great company that has great tools with lots of features. Fact is that even though these tools are great, they are also quite expensive. They’re feature rich, so really knowing the tool is a skillset of its own – and it makes good IT people hard to find when I need them to do a job. We also run up against serious limitations when trying to use these tools; sometimes we cannot find a way to use the tools to migrate the files in exactly the way we want to.

Second, some of my client already have a part-time IT person or managed services company that helps them service their PCs and on premises servers. Traditionally, we’re a SharePoint consultancy and we never set out to try and replace other IT folks; they need work too. They have the relationship with my customer, and the local presence needed for that on-site work. Over the years, I’ve seen that customers prefer to have their own local IT provider for most small requests. We needed to find a way to coexist with these other businesses in a way that would benefit us both.

Back in 2012, at the behest of a marketing consultant (who gave me lots of advice that was either bad or I couldn’t follow it at the time) I created a small tool called CloudPrep. This tool wasn’t much; I never had much confidence in it and so I never really promoted it. But, it did the work of renaming files that SharePoint didn’t like, and combined with WebDAV it was enough to make getting 20 to 50 GB of customer files into the cloud in a few days’ time. I released it into the wild, and CloudPrep has been getting downloaded a few times a week – mostly by other Office 365 consultants to my chagrin. Lesson learned and another checkmark for finding a way to compete with other IT providers; there are more of you than there are of me!

One problem I’ve noticed is that Office 365 migration budgets are small – I mean really tiny! That’s weird when you consider that for a 25 person company the ROI could be hundreds of thousands of bucks. But, we have been in an economic slump for something like 5 years now. I guess that takes its toll; even if you knew it would make you a thousand dollars next month, you can’t spend $100 today unless you have it to spare. Some companies are reluctant to spend even a few thousand to plan and execute.

There are a few tools that are in the “beer money” range. I tried FilesToGo once – and only once. It lacked some features that seems obvious to me, but made my client extremely angry. It didn’t have a lot of options either, one size fits all. I won’t discourage anyone from using it if it meets your needs, but I’m not going to risk my relationship with my clients on it. I am honestly surprised that after all this time, there’s nothing else in its price range.

I guess you could say that I’ve gotten fed up with this situation. Yet another migration we had to do where the current tools on the market couldn’t meet our needs for the client’s budget. That story gets old.

So, the boys in the lab and I finally built our own!

Announcing CloudPrep 2014! Forget everything you ever knew about that crappy tool we made back in 2012, because this is completely something at a whole new level.

CloudPrep 2014 is not one of those big expensive tools with a fancy GUI. It’s a set of PowerShell command-lets that work with SharePoint Online and your local file system. These commands and the sample scripts provided with them are designed to empower IT people and make migrating files to and from SharePoint Online a piece of cake.

These tools don’t replace an IT person or their experience. You’ll still need an experienced consultant to tell you how to organize your files, use metadata, overcome or avoid SharePoint Online limitations, and of course actually use the tools. You needed all that before anyway. The difference is that now much of this can be provided by your own experienced IT staff; or if you’re an IT consultant yourself, you can use our tool and make your small-business and small-budget migrations a breeze instead of a quagmire.

Our commands fall into basic categories: planning, preparation, file migration, and SharePoint management. We’re still putting the finishing touches on the product now. We’re hoping to have the Lite and Standard editions released to market sometime in February, with the Premium and Professional versions available as soon as March or April.

In the meantime, please take a look at our feature matrix and proposed pricing structure. There’s still time to collect some feedback. So, if you have a feature you’d like to see that isn’t here, then leave us a comment and let us know. Even if you don’t add a feature by the launch date, we’re planning to add even more features later. We’ll entertain any reasonable suggestion – except charging more for the product.

Like what you see and can’t wait to try it out? Contact us and I’ll give you a 15% discount if you purchase during the early access period.

Edition->Feature        Lite   standard Premium Professional
Release Date   Feb  Feb  March  April
Proposed Price Free $285 $576

$1,092

+$300 Per Tenant>2

Number of Office 365 Tenants Unlimited Unlimited unlimited Unlimited
Numbre of Site collections Unlimited Unlimited Unlimited Unlimited
Requires powershell 2.0 or higher Yes Yes Yes Yes
Requires Sharepoint client connectivity Yes Yes Yes

Yes

1 year support and Updates

(renewable Annually)

  Yes Yes Yes
Supported OS: Windows server 2008 or 2008 R2 N/A Yes Yes

Yes

Supported OS: Windows XP N/A   ?? ??
Supported OS: Windows Server 2003 N/A   ?? ??
Planning and Reporting        
Sizes and Numbers of items by folder, extention, ect. Yes Yes Yes

Yes

 

Check for Potentially Illegal file types   Yes Yes

 Yes

Folder and File Path Length Checking   Yes Yes

Yes

Permissions Checking for Local Files     Yes

Yes

Target URL Length Check Report     Yes

Yes

Upload Time Estimates      

Yes

File Preparation        
File Renaming for Illegal charaters Yes Yes Yes Yes

File Renaming for Illegal Paths

(_files,_forms)

Yes Yes Yes

Yes

Preserve Author and Editor for uploaded Files

  Yes Yes

Yes

Check for and Automatically ZIP files with illegal extentions (EXEs, Ect.)

    Yes

Yes

Check for and Automatically ZIP "_files" Folders

  Yes Yes

Yes

Migrate and Manage Files

       

Supports Network Mapped Drives

yes yes yes yes

Supports Network UNC Paths

yes yes yes

yes

 

Upload Entire Folder to Document Library

Yes Yes Yes

Yes

Upload Specific File to Document library

  yes Yes

Yes

Download Document Library to Folder

  Yes Yes

Yes

Download Specific File

  Yes Yes

Yes

Warns if Source Exceeds 5,000 items

  yes Yes

Yes

Warns if Target URL length Too Long

  yes Yes

Yes

Specify Content Type for Uploaded Documents

  Yes Yes

Yes

Specify Content Type for Top Level Folder

    Yes

Yes

Specify Content Type for Sub-Folders

    yes

Yes

Support for Documents Sets

     

Yes

Flatten Folder Structure with duplicate filename handing

    Yes

Yes

Flatten Folder Structure at 1 or more levels deep

     

Yes

Convert Folder Names to Metadata Fields

    Yes

Yes

Create Source URL Field for Uploaded Files

    Yes

Yes

Create MD5 Hash Field for Uploaded Files

     

Yes

Export Metadata to CSV File when Downloading Files

     

Yes

Synchronize of Local and Cloud files using File Modified Time

    Yes

Yes

Synchronize of Local and Cloud Files using File Modified Time+ MD5 Hash

     

Yes

Automation Features

       

Powershell command-lets

Yes Yes Yes Yes

Unattended Execution

  Yes Yes Yes

Sharepoint Management &

Development      

Create and Edit SharePoint Users

  Yes Yes Yes

Set Common Properties for Lists and Document Librarys

  Yes Yes Yes

Create and Edit Columns in Lists and Document Libraries

  Yes Yes Yes

Create and Edit Views Lists and Document Libraries

    Yes Yes

Copy a view to same or Different Document Library or list and site

    Yes Yes

Import and Export Site Columns

    Yes Yes

Import and Export Content Types

    Yes Yes

Import and export views

    Yes Yes

Add, Remove users and Groups, Permission Sets

    Yes Yes

 

CloudPrep Lite
This edition is a good fit for small file migration needs and try-before-you-buy. You can use it to do basic reporting on the structure of your files, rename files that are known to cause problems during migration, and upload folder structures to your SharePoint Online document libraries. In most cases it has a 99.7% or better success rate, and it produces a handy report so that your remaining files can be uploaded manually.

CloudPrep Standard
This edition includes a standard set of features designed to help you move files into Office 365 with a minimum amount of difficulty. You can upload and download large file collections without having to stand by the computer, perform multiple upload/download passes, and specify a default content type for files. Run it from anywhere, including various versions of Windows Server. We also include some additional pre-migration reporting tools that help to identify problems before you migrate your files.

CloudPrep Premium
For the seasoned SharePoint admin or IT professional, this edition includes features that will help you get the most out of Office 365 in the cloud. We include even more reports to give you a 360 degree view into any potential file migration issues. The file upload tool includes a variety of features for setting metadata and flattening folder structures.

CloudPrep Professional
This edition enables the true Office 365 IT professional to handle migrations for multiple clients. All the features of the Premium Edition plus advanced content type features including support for Document Sets. It also includes the ability to create MD5 Hash file uploaded files, which helps in detecting duplicate files and in determining that if two files are not the same even when their date stamps match.