Office 365 Security and You - Access Control

YouTuber JackSkepticEye plays Papers, Please. What does this have to do with Office 365 security, read on and find out! This is the second part of a series on cloud security topics. In the first part, I discussed the threat that has Ransomware over people and companies. I started this series to book-end around my appearance as part of the SharePoint security panel at this week's Federal IT Security Conference. Since that conversation unfolded, I think we'll do a Part 3 next week to cover the topics discussed at the panel, which were very different that I imagined they would be.

PART 2: 12 Ways to Control When and Where People Access Office 365

Recently, many of our customers who are interested in migrating to Office 365 have been asking us questions about whether it's possible to control when, how, and where their employees can access their data.

While there are some technical approaches that may work, the unfortunate news is that there's no "silver bullet", at least as far as we've been able to find - yet. Many possible solution feel like kludgy work-arounds, temporary half-measures, partial solutions, or something created only for larger organizations.

I thought I'd take the opportunity to put together a list of possible ways to tackle this challenge. Even though no option is a complete answer, it's possible that some of these may be a good fit for your specific circumstances. I'll do my best to go over the pros and cons of each option.

Is this necessary? Depending on who you are and what you do, maybe not. Overkill? A bit heavy handed? Perhaps. Thus the graphic above, which (for those of you who may not be gamers) parodies an Arstotzkan border guard from the dystopian job simulator game classic "Papers, Please". Understand though that in some cases it may be reasonable, since many industries are subject to regulatory compliance requirements that might not always be perfectly aligned with a cloud based IT strategy.

Fair warning, this is a pretty complex topic. Hopefully everyone has gotten over their election night hangover and is ready to dig in. So, without any more fanfare, let's check out some methods to implement extreme vetting in Office 365.

Access Control - The Basics

When we think about granting access, we're basically describing the five W's that need to be addressed in order to make a decision about letting a person have access to information. A perfect access and identity system would answer all the questions below before letting someone in to the system - and may even use some of the answers to put a limit on what they can access at any given moment.

Who

  • Is the user logging in actually who they say that they are?
  • How confident are we in that?
  • Have they been educated and informed about security and privacy policies?
  • Is their ability to act responsibly expected?

What

  • What is being accessed; is it email, documents, some other data?
  • Is accessed content subject to regulation such as HIPAA, SOX, or GLBA?

Where

  • What network are they connected from?
  • Do we have any geo-location data?

When

  • Is it the normal workday or after-hours?
  • How does "now" jive with past or expected work patterns?

How

  • Is it a known PC, mobile device, or something new/different?
  • Are they using a browser (that can run JavaScript or CAPTCHA test), or could this be a bot?

Why

  • What's the business purpose behind needing the information?
  • Is it reasonable to expect responsible behavior?
  • If the behavior is unusual, is it known in advance or has it been vetted?

Okay, so now that we've been over what sorts of things go into granting access, let's get specific. The answers to "who" and "what" are already largely covered by conventional authentication and authorization systems. The topic in question - the one we're hearing about from our customers - specifically addresses the "when", "where", and maybe "how" above.

So, without further ado, here's my list of 12 things that can be done to control access to Office 365 and other resources in the cloud. Some are cheap. Some are definitely not. None are perfect for everyone. That's just life, I guess. If you'd like help finding a solution that will work for you, please talk to us about it, because that's what we do at Liquid Mercury Solutions.

Option 1: Just don't share the password with the user

It sounds stupidly easy, but if you don't want somebody to login from home, don't give them their own password. You can handle this in a couple of different ways. Either set up the Office 365 account on their work PC and save the credentials to it without telling them the password, or go ahead and give them their own account but have another account that is only used for access to important or sensitive information, and then keep that one under lock and key.

Plus side:

  • 100% effective once stored on the local PC.
  • Cheapest option available.
  • Can work even with Cloud Only users. No AD domain controller required.

Down side:

  • Creates a feeling of oppression and lack of ownership.
  • Ties access to a single person; people can't get access when people who know the password aren't there.
  • Tendency to use the same password on multiple logins is a bad idea.
  • Tendency to use the same login for multiple people is a worse idea.
  • These factors together mean that this approach may be abused in ways that are worse than the problem that its trying to prevent.

Option 2: Trust but verify

You know, I really think we spend too much time thinking about all the ways that people are going to steal from us. When you consider it, it's amazing how rarely someone actually does.

Today, our reporting tools are much better than our access controls, so it's much easier for us to build a solution that will help create accountability than it is to enforce compliance by making it impossible to violate policy. Instead of spending lots of money on IT, trying to fit a square peg in a round hole by making cloud services act like old-school computers, why not focus that same energy on making sure employees know their responsibilities to protect data.

If employees know that they are not supposed to access HIPAA sensitive documents from home - and that you can tell when they have done so and will fire them for it - chances are very good that nobody will ever cross that line. The hard part is making sure there is a system in place that makes you aware if there is a problem, and that your employees know they're accountable too.

Plus side:

  • Simply modifying employee policy to allow remote access can be cheaper than any technical solution.
  • Having HR policies in place should probably be done anyway to make sure users understand their responsibilities.
  • While not the cheapest option, no or very little IT cost required compared to other options.
  • Provides maximum flexibility in unusual or unplanned situations.

Down side:

  • There are a few options for decent reporting, but not as many as we'd like.
  • Taking the time to audit usage can be just as taxing as blocking it.
  • By itself, this does nothing to prevent an account from being used improperly.

Option 3: Use ADFS

If you absolutely need to make sure that nobody can login to Office 365 from home, there's one absolutely foolproof way to go about it and that's to federate with an ADFS server located in your office. Then, all you need to do is not expose the ADFS server to the internet and your users will never be able to get to anything in Office 365 - period.

This is actually a "broken" version of a typical ADFS configuration, since usually most folks want to be able to allow access from home. We know it works, because when the power or internet goes down at the main office where ADFS is running, people working from home can't login.

Of course, if you absolutely need remote access, or some users need cloud access, you can configure a second DNS domain for them and not enable it for SSO. Without ADFS, this second domain and its users would use the regular login process for Office 365, and thus be able to get in from anywhere.

Unless you are such a small company that you can't afford to maintain a domain controller in your office, this may very well be the best solution for you. I'd be hesitant to recommend it to companies of less than 25 employees unless they have a very compelling reason, like HIPAA for example. It does take an experienced IT person to get it set up and correctly configured.

Plus side:

  • Well established solution; well documented.
  • ADFS comes free with Windows Server.
  • Absolutely effective as preventing outside access; if you don't want users outside your network, simply don't expose ADFS to the internet.
  • Flexible enough to work in a variety of scenarios.

Down side:

  • ADFS has a high technical debt.
  • Requires a Windows AD domain controller; many small companies would rather eliminate on-premises servers.
  • Adds to technical complexity, especially if you also have some cases where access to Office 365 from outside the network is allowed.
  • Doesn't readily distinguish between access to e-mail and documents, so you may need multiple accounts if you want to access some systems remotely but not others.

Option 4: Lock account based on login times using a script

It's possible to enable and disable logins using PowerShell. It's also possible to run PowerShell as a scheduled task in Windows. Both of these can be done from a workstation computer and do not need a server or other fancy hardware. Using PowerShell, you could "open the cloud store" in the morning and close it in the evening. In this case, nobody but you would be able to sign in unless you logged into the web site and overrode the settings.

This is a sort of weird scenario, really. I don't know very many people willing to go to these lengths to keep people out of Office 365 when they aren't at work. Also, then they wouldn't be able to check email either. It might have an application against a special-access account that only gets used during the day, like the one I talk about above in Option 1.

I probably should mention that if you go with Option 3 and use ADFS, it will automatically follow login times configured in Active Directory, making this totally unnecessary. So, unless your company is very small, I'd probably recommend doing that instead.

Plus side:

  • This can be run from any Windows machine, even a workstation
  • Can work even with Cloud Only users. No AD domain controller required.
  • Easy to automate based on a schedule.

Down side:

  • Prone to problems if the script fails to fully "open or close the store".
  • Not a good fit for people who need around the clock access, but only from certain locations - or other scenarios that are not strictly time based.
  • Will take extra time and effort to manage and support.
  • Doesn't distinguish at all between access to e-mail and documents.

Option 5: Tie Office 365 Multi-Factor Authentication to a device that's only available in the office.

This is a lot like not sharing the password for an account, except that really what you'd be doing is withholding the second layer of authentication. Since the second factor authentication may not come up all the time, this would be more transparent and thus less destructive to employee autonomy than not giving them their own password.

Here's how it works: create an account in Office 365 and configure a password to use while you set it up. Then, configure multi-factor authentication and enroll the user against a device that's only in the office, like a desk phone or their supervisor's cell phone. Once they are enrolled, reset their password to a temporary one and share that with the user so they can pick one of their own. Now, you've effectively prevented them from logging in at home, since it will be an unfamiliar device and network, which would trigger the MFA.

Before you count on this method, you might want to test it for yourself. There are different flavors of MFA in Office 365, and some of them only come with E3 / E5 / EMS plans. The enforcement options, triggers, behavior, and configurability may all be different if you're using the vanilla MFA that comes with a Business Premium plan, for example.

Plus side:

  • Allows people to know their own password.
  • Adapts well to contingencies such as having to arrive early / work late.
  • MFA settings can be configured per user and overridden as needed.

Down side:

  • Requires a cell phone or voice phone to be present in the office; Most people have a voice line though.
  • You can't let users self-enroll in this scenario.
  • Takes about 1 to 2 minutes longer to login to the system each time.
  • May require multiple accounts/licenses per user if some information needs to be controlled but other information does not. For example, if you need MFA for access to HIPAA sensitive documents, but not to e-mail.

Option 6: Customize SharePoint to Increase Security

Most folks who want to protect documents from their own employees are not actually interested in preventing them from accessing emails. But, most security solutions for Office 365 are applied against all the Office 365 services. If the documents you need to protect are in SharePoint, there may be better ways to go about this that wouldn’t impact other aspects of your service.

Of course, the ultimate solution would be to deploy CipherPoint Eclipse. You can think of that as the very best form of SharePoint customization there is, because it will let you encrypt documents and then use a variety of different policies to determine whether they can be decrypted. It's an expensive option, comparatively, but also a good one that offers true security (rather than security through obscurity). And now that I'm done plugging for our partner, I'll tell you about a slightly cheaper one.

Microsoft won't actually let us run server side code in SharePoint Online as they once did. So, our options are limited from being able to control the users access and experience on the site. Even so, it's not too difficult to do some rudimentary access control using JavaScript in the browser. For example, you can hide the page contents and display apocalyptic warnings instead. In some cases, you can also end a user's login session.

However, it is important to understand that code that works this way can be circumvented by those with a moderate amount of computer savvy. If you're going to rely on sleight of hand tricks to protect your information, you'd better also back it up with a clear employee policy, firm contractual agreement, audit logs, and regular reviews for bad behavior.

Plus side:

  • Significantly easier (and cheaper) than implementing security at the login prompt
  • Relatively easy to track both IP address and login time when using SharePoint.
  • Transparent to non-SharePoint Office 365 services, so if you're just trying to protect HIPAA documents, but still allow email access, this may be the way to go.

Down side:

  • To be fully effective, use of OneDrive sync and the SharePoint API will need to be blocked in sites that have sensitive documents, and this can limit how you customize SharePoint.
  • Requires documents to be stored in SharePoint. Other Office 365 services can't be protected this way.
  • Can be defeated by a determined intruder; many would say this does not offer true security but is more "security theatre".

Option 7: Encrypt It!

Most people don't need to protect literally everything they store in Office 365. Further, not everyone needs to protect what they store in SharePoint lists too, or implement complex policies to determine which employees should have access to what documents. Thus, solutions like CipherPoint that I mention above would probably be a bit heavy handed for most small businesses. (If you fit the above description, we'd still love to hear from you, because there's a lot more we can do in these cases.)

If your need to protect sensitive information is moderate and limited to a particular site, document library, or classification of content, then Microsoft's solution that comes with the E3 plan is probably good enough for you. I'm talking about Azure Rights Management, and while it won't keep an employee from viewing a document on their home computer, it can keep them from downloading it to their phone, printing it, or copying its contents to an email. Also, should the unfortunate need arise to fire their ass, it can also let you take access to that information away after the fact - no matter how many times or places they've copied that file.

While I wasn’t a huge fan or early versions of ARM, it has matured a lot. It's easier to set up now than it used it be, which is good if you don't have a huge budget for IT. Since it can be purchased a la carte, you can let Business Premium users access ARM protected documents when necessary, without having to upgrade them to the E3. (Unless you want to. I'm totally cool with upgrading if you want to. Have you met the E5?)

Plus Side:

  • Encrypted documents are useless, even if copied off the network
  • Even your IT admin (or Office 365 support partner, like us) can't read the encrypted document.
  • Easily restrict who can read or edit a document - as well as some other things they can do with it (e.g. print, copy/paste)
  • Access can be revoked after-the fact.
  • A good solution if you only have a sub-set of documents you need to protect.

Down Side:

  • While you can control a lot of access, that does not necessarily include when or where users are allowed to read or edit a document.
  • Doesn't protect SharePoint data stored in lists or web pages, OneNote Notebooks etc.
  • Azure Rights Management is only included in E3 plan and above.
  • Third party solutions such as CipherPoint can be costly.

Azure AD Premium

Before I go on, here's a few notes about Options 8, 9, and 10 below regarding leveraging Enterprise Mobility + Security, Azure AD Premium, and Azure Advanced Security. These were things I think apply in general to the entire suite that go beyond the specific applications I mention in my list.

  • There will be additional monthly costs for service, and you may need on-premises hardware too.
  • Some solutions are simple while others can be quite technically complex.
  • While there may be features we're not aware of yet, there really doesn't seem to be the kind of access control our customers have been looking for, particularly for end-users. (See Identity Protection and Privileged Account Management below.)
  • Many scenarios, especially AD Premium, don't have a large user base yet outside a few big orgs and aren't well proven especially for in smaller companies.

Option 8: Registered Devices and Workplace Join

This is Microsoft's solution for adding PCs and mobile devices into Azure AD. And it's not a bad fit if you're interested in Windows as a Service, Intune, and the like. Joining devices to Azure AD basically makes it possible to login to your "domain" even when you're out of the office. It can also, conversely be used to require users to login only from approved hardware.

Plus side:

  • Prevents users from working on unapproved hardware, such as personal computers.
  • Controls access by physical device; if you want to control access by location, don't let the physical device leave the desired location (e.g. use desktop computers not tablets)

Down sides:

  • This is a fairly complex deployment, possibly requiring help from experienced experts, and may not be suitable for small businesses.
  • Requires modern PCs (the Windows 8.1/10 scenario is better than Windows 7/8)
  • Requires a modern (2012 R2) Windows Active Directory domain controller
  • Requires configuration of ADFS server, which need to be accessible form the internet
  • Requires a license to Azure AD Premium
  • Relies on AD Connect / Sync so it can take quite a while for hardware info to be fully synchronized.
  • This solution can't really distinguish between user access to e-mail and user access to documents, so if you need mobile access to mail but not sensitive documents, this isn't your best option.

Option 9: Azure AD Premium w/ Identity Protection

I actually like this option a lot, because of its simplicity. It's not easy to take something as complex as access security and make it as easy to set up and manage as Identity Protection is - especially if you're Microsoft who seems to thrive on complexity and options. It's a really good system, and they've done a good job of providing a solution to help users deal with the identity theft threats that are becoming increasing common nowadays.

But - and I'll cook my hat and eat it if I ever say these words again - Microsoft may have gone a bit too far into easy-to-configure territory, because there are a lot of options missing from Identity Protection that I would've thought would be obvious.

For example, where's my option to say "my employees only work in the United States, and for that matter they're only in Maryland for the most part." Or, how about, "We really don't work later than 8pm EST, so could any midnight logins please be labelled 'high risk'?" Why not let the admin get a notification in addition to blocking access or triggering MFA? All these things were missing, and I was really surprised by that.

Otherwise, it's pretty good and you should totally buy it. Maybe they'll improve it later. If not, please see Option 11.

Learn More about Identity Protection from Microsoft's Blog

Side note: We had a case recently where a client has an employee who was being targeted by a cybercriminal who had taken their credit card data and was trying very hard to target their email account in Office 365 too. Fortunately, Microsoft was diligent in locking the account after many successive failed attempts. However, it is important to understand that information which may have helped to lead to an arrest in this case was not being captured until we activated Azure AD Premium and Identity Protection for the customer. If you're locked out of your Office 365 account and you have good reason to think it was because of a hacking attempt, I strongly suggest that you do not wait, but go ahead and start the free trial for AD Premium and turn on all Identity Protection's logging features. From there, if you simply want to protect yourself, you can set up MFA - or consider setting up a honey pot if you want to try and catch the would-be thief.

Plus side:

  • No local server required.
  • Can work even with Cloud Only users. No AD domain controller required.
  • Microsoft Add-on for Windows Azure AD Tenants
  • Remediate risk by requiring multi-factor authentication, force password updates, and/or blocking access entirely
  • Uses threat analytics which includes data from other Azure users, not just your own company
  • Protects from: sign in from infected devices, new/unfamiliar locations, impossible travel distances, anonymous IP addresses
  • Tracks leaked credentials
  • Doesn't seem to add much burden in the way of administrative overhead or management
  • Most of the MFA enrollment is intuitive (at least for an IT person) and can be self-service.

Down side:

  • We thought that MFA enrollment left too many steps and choices to the end users and should be something admins could lock down or simplify.
  • Conditional access risks are managed by Microsoft and divided into low/medium/high; there does not seem to be a way to define things such as normal working hours or normal location.
  • Has a tendency to throw false alarms in some networks; for example whenever we visit the Microsoft office in Washington DC, it tells us we're trying to login from Redmond, WA.
  • Although you can resolve an event or mark it as a false alarm, there didn't appear to be anyplace for an admin to leave notes explaining why the login occurred, like the situation we describe above.
  • Despite some marketing materials that seemed to indicate this would be available in EMS E3 plan, it still required applicable users to have Azure Active Directory Premium Plan 2, which is part of the EMS E5 plan.
  • None of these Azure security and logging features are enabled until you activate this service.
  • We had to actually sign up for Azure AD Premium trial offer in order to get the system to recognize our existing AD Premium licenses from Office 365.

Option 10: Azure AD Premium w/ Privileged Identity Management

Okay, I'm going to sum this up nicely. If you're a Microsoft Partner, like us, supporting Office 365 customers, or if you have more than 2 Global Administrators on your Office 365 account - for whatever reason - this solution is for you. Everybody else will probably find this to be either too expensive or much too cumbersome to justify. It really only protects your admin accounts, so in most cases you'd probably do just as well to just configure MFA on them and be done with it.

Learn More about Privileged Identity Management from Microsoft's Web Site

Plus side:

  • No local server required.
  • Can work even with Cloud Only users. No AD domain controller required.
  • Microsoft Add-on for Windows Azure AD Tenants
  • Allows Just-in-Time Access to high level (e.g. global admin) accounts
  • Monitor how privileged access is being used
  • Notify other system admins in real-time when privileged accounts are used
  • Uses threat analytics which includes data from other Azure users, not just your own company.
  • Seems to have some really cool reporting capabilities, but they take time to populate.
  • Really the only way that I am aware you can give someone global admin access to Azure or Office 365 and still keep an eye on and require them to justify their use.

Down side:

  • Adds extra login steps and technical debt for admins.
  • There is significant complexity involved for those who will need to manage and support PIM.
  • Doesn't seem to provide an option for who should receive alerts about usage.
  • Does not provide JIT access or monitoring for regular user accounts.
  • The ticket number formats are a bit restrictive.
  • Required applicable users to have Azure Active Directory Premium Plan 2, which is part of the EMS E5 plan.
  • We had to actually sign up for Azure AD Premium trial offer in order to get the system to recognize our existing AD Premium licenses from Office 365

Option 11: Beowulf Identity Server

I’ve talked plenty elsewhere about how awesome Beowulf is, how it shuts the front door on SharePoint, and how it protects your public facing web sites and applications from unwanted access. You don't need to hear even more of that from me here, so I'll stick to what we haven't said before. (Aw, c'mon. You didn't think I was going to spend all this time and energy writing a two part blog about security without promoting my own product, did you?)

We’re working on a version of Beowulf that works with SharePoint Online and the rest of Office 365, which shouldn't be terribly difficult since we already fully integrate with ADFS which is what Microsoft is using for access control in the cloud.

Since others seems to have dropped the ball on some of the options and features we've talked about here, we're doing our best to include them in the new version targeted for release in early 2017. Well, that's the big problem isn't it. Unless you want to be part of our early adopter program - and get a big discount for helping us test these new features - you're out of luck.

Lean More about Beowulf Identity Server on Liquid Mercury Solutions' Web Site

Plus side:

  • Low cost cloud based solution
  • Transparent access layer between users and Office 365
  • Can work even with Cloud Only users. No AD domain controller is required.
  • Can block access or alert you (but not block access) when a user logs in from unexpected locations or at unusual times.
  • Configurable in a lot of ways that Microsoft's solution is not.
  • Has many of the same MFA capabilities as Azure AD Premium.
  • Integrates well with Azure AD, ADFS, and other MS solutions.

Down side:

  • There is an additional cost outside of the Office 365 subscription
  • Like many advanced security products, set up is relatively complex.
  • Though many of these features are available today, our full feature set for the next release will not be available until early 2017.

Option 12: Application Layer Security Enabled Next Gen Web Proxy/Firewall

You all knew I'd bring it up eventually. Why don't you just go out and buy an F-5 Big IP with the Access Policy Manager module on it? Then you can come back to us and hire us to configure it for you, and we can totally freak out because people hardly ever want to do that. Even so, this is a nice way to go if you have a lot of money lying around, and burning it would be inconvenient.

For large enterprises with hybrid cloud/on-premises deployments, I do recommend products from vendors like F-5, Kemp, or Cisco. This goes triply so if you run a large corporation with name recognition, store a lot of confidential customer data that hackers may want to steal, or your everyday business is something that might lead people wearing Guy Fawkes masks to try to ruin your holiday weekend. They offer security features that Microsoft doesn't even come close to having in Azure yet, but you can absolutely deploy them as Azure VMs in your environment or on-premises as real metal or VM.

But then, if you're going to go that far, why not also make sure you do all the other things I talked about too?

Plus side:

  • Really, really, configurable and powerful; can probably do anything you'd want in terms of limiting and responding to access requests and use.
  • Deployable in traditional on-premises and cloud-based scenarios.

Down side:

  • Really, really, complex to configure and expensive to implement.
  • Even cloud based subscription versions are going to cost a pretty penny.
  • It will require dedicated staff and constant upkeep, so probably only suited to large enterprises.

As you can see, Microsoft offers many choices - but none of them is the perfect solution for everyone. Better solutions I think will emerge in the coming months. I hope I've done a little here to shed some light on what is sadly a very complex answer to what seems like it should be a simple question. The most important thing to consider I think is that there are some low-cost things that you can do if you want to control how people use cloud services, starting with making sure that your employees know the rules.

Technology is always changing, and it often forces us to consider scenarios that previously were just impossible. If you're considering Office 365 as a solution, you may have concerns about people having access from home (or anywhere in the world really).

Office 365 provides a lot of security advantages compared to storing sensitive data on your laptop computer, a portable hard drive, or that server in the closet. Keep in mind that this is just one potential risk in a sea of others that we've all faced for a long time; the benefits should outweigh the risks if you approach the transition to cloud services with a little bit of thought and planning. We're here to make sure you don't have to go it alone.

Did I leave something out of my list that you'd like to add? Leave a message in the comments and I’ll reply.

Thomas is an acknowledged expert on information security, the creator of Beowulf Identity Server, and spoke on the SharePoint Security panel November 8th at the First Annual FITSI.org Federal IT Security Conference. You can follow him on Twitter and LinkedIn - but if you really want to connect, your best bet is probably to call us at 410-633-5959.

Something New is Springing Up at Liquid Mercury

Today’s technology marketplace is constantly changing. Larger IT departments are working with smaller budgets, and in-the-cloud capabilities are bringing abilities to smaller businesses that they’ve never had before. Disruptive technologies have everyone feeling a little bit irritable, and somebody keeps moving their cheese. The overall result is strong down-market pressure on the entire market.

Many companies tell us that they’re now working actively to reduce their recurring monthly costs for cloud based solutions such as PaaS, SaaS, and hosting. As such, they’re seeking to return to the earth with solutions that – while they may represent a larger investment in the short term – allow them to control the terms under which they incur costs for initiatives such as upgrades, maintenance, and support.

In short, our clients are reporting that cloud based solutions simply provide too much functionality for their money; they want to do less with less.

Liquid Mercury Solutions is constantly striving to stay ahead of these emerging business trends. In response to overwhelming requests from you the customer (usually made in the form of late or non-payment) we’ve made an important decision to diversify our offerings.

Announcing Liquid Mercury Farms, a new venture devoted to getting our head out of the cloud.

“I thought you were talking about server farms.”
– LMS founder and CEO Thomas Carpe seen with Attila (left) and Seth (right)
As an alternative to moving to the cloud, Liquid Mercury Farms offers a broad array of ground-based solutions for “subsistence IT”. For example, our premier line of Liquid Mercury Eggs* is Grade A extra-large. They’re an excellent source of protein, delicious with toast, and the perfect add-on once you’ve uploaded bacon into SharePoint.

Our farm is also highly secured, with all equipment stored behind electrified barbed-wire fencing. Production servers are kept in locked cages, and all gates require two-tractor authentication.

The farm is fault tolerant, offering five nines of capacity – that’s almost 4 dozen eggs a day. Farm infrastructure has been fully optimized for production layers. Also, our cluck-through ratio is off the chart. 

Best of all, our support staff work for chicken feed.

To make licensing Liquid Mercury Farms’ products as pain-free as possible, we’re now accepting sacks of potatoes and fresh dairy in addition to our usual methods of payment. So, at the risk of beating a dead horse, why not give us a call and save a few bucks?

*Please note that Liquid Mercury Eggs contain no actual mercury. Happy April 1st!

“I thought you were talking about server farms.”
– LMS founder and CEO Thomas Carpe seen with Attila (left) and Seth (right)

AgilePoint Anounces Office 365 and Forms Capabilities at SPC14

Well, it's that time of year again where all the SharePoint product companies trot out to Las Vegas to strut their stuff.

Today, we have a big anouncement from the SPC 2014 Keynote Sponsor, AgilePoint.

AgilePoint - SharePoint Conference New Product Highlights

In this release, there are two things I noticed right away that we've been eagerly awaiting for a long time. 1) AgilePoint support for Office 365 not just as something that can be manipulated by workflow, but in a fully integrated fashion similar to Nintex workflow. 2) An alternative to InfoPath forms that emphasizes responsive web design.

As readers of our blog will know, we're quite fond of AgilePoint's product. One of the difficulties we face in working with it, however is that it didn't really play well with customers working in Office 365. We're happy to see now that is a possibility, and we'll be putting together some demonstrations in the next few weeks, as we definitely want to be able to take this out for a test drive and see what's possible.

CloudPrep 2014 Development Update

I wanted to take a few minutes today to talk about what we've been doing since late January in regards to CloupPrep and the PowerShell commands for file migration and management of SharePoint Online.

First thing I can say is that one of our most difficult choices was in choosing an e-commerce platform and licensing API to use for our product. Even though we plan to keep our licensing fairly simply, we wanted to have options for future products and well as many of the items we also sell through our partners.

This turned out to be more challenging than I imagined, but we have settled down on using Fast Spring and LogicNP Crypto License. Perhaps in some future post I will talk about those more from a software developer's perspective. What I can say today is that it will be at least a couple weeks before we can get a working prototype of the licensing server and the store online, and so we have had to push back release closer to the end of March or early April, mostly for that reason.

Meanwhile, we have been developing features for the different editions of CloudPrep 2014. Progress on that front continues at a rapid pace and I am pretty satisfied with the way our tools are maturing.

When we decided to produce this software, we planned to release the lite and standard editions first and follow up with premium and professional features later this spring. I was a bit surprised to see that where we are putting our development efforts, probably all four editions of CloudPrep will be available at one time.

Now for the geeky stuff. Here's some of what's been happening as we've been building.

Features we've essentially completed:

  • Upload an entire folder or specific set files to document library
    We've tested that these commands will work against network drives and UNC paths. Take that, OneDrive!
  • Preserve metadata about the local file system that the document was uploaded from
  • Create and Modified dates on files are preserved, though we did find that with larger files there are limits to what we can accomplish here
  • You can specify the content type for uploaded files, root folders, and sub-folders - including Document Set and its child content types
  • A bunch of other random stuff including commands for manipulating SharePoint lists and reports to make sure that file uploads won't exceed SharePoint limits


We noticed that Office 365 throws us a lot of connectivity errors that we don't normally see in on-premises SharePoint environments. If you've been trying to copy files using their standard UI or using OneDrive, some of these errors might be hidden from you. However, they're readily apparent if you're using Web Folders (WebDAV) or Client Side Object Model to connect. We see unexpected dropped connections quite often, and certain upload methods will time out on files that are too big and required some fun workarounds. There are different methods needed for files under 2MB, under 35MB, and larger.

Our path was also complicated by the fact that on certain Office 365 sites, our rights come from delegated admin privileges. This is the preferred way that consultants get their rights to help clients manage SharePoint Online, so we figure a lot of folks who are interested in CloudPrep are seeing this phenomenon as well. When you log in with delegated admin to a client's Office 365 site using the credentials from your own Office 365 account, you sometimes see the access denied page; login again a few seconds later, everything is fine. Our code had to expect and handle this contingency.

Another thing that we did not expect is that we're seeing some reasonable evidence that Office 365 uploads are being throttled. Most of the time, file transfers seem to be limited to about 300KB/sec; there are days when the transfer speed is even slower than that, sometimes by half. As such, it is difficult for us to estimate file upload times, and we're having to improve our algorithms to take these fluctuations and sea changes into account.

As for the cause, we can't say if this is something Microsoft is doing, or if it comes from the erosion of net neutrality. We do wonder if Comcast or other providers may be limiting traffic to Office 365 in order to give their own offerings a competitive advantage or just to control their own costs. I expect we'll be doing some tests in the near future, and we've been kicking around some ways to circumvent these bandwidth caps - at least partially. One test we did in January showed that if we took half our files to a different physical location, we were able to upload them to SharePoint Online in about half the time it would have taken if we'd uploaded them all from one server.

One thing that became clear during early development was that the disconnected nature of cloud storage was going to introduce multiple random problems along the way. As a result, in any large set of documents to be moved to the cloud, there would be some which for one reason or another may not be successfully copied. We started by trying to get this failure rate as low as possible, down to less that 0.25% of files in most cases. We did a lot of work in early February to improve the code and reach this threshold.

Even so, we needed to be able to easily run multiple passes on any file copy operation and track the results. Our first prototypes had to crawl the Document Library in SharePoint one folder and file at a time. This proved to be incredibly slow, and it quickly became apparent that we needed to be able to gather status information for thousands of files at a time if we wanted to hone in on only those which required an update from the local copy. This is something we added to the code base about a week ago, and we're now in the process of replacing some of our early code to use the new file comparison analytics logic.

As a side note, a bevy of SharePoint management features found their way into our PowerShell library simply because we had customers who needed them in short order. For example, we now have the ability to take a View from any SharePoint List and make a copy of it in the same List or a different one even on another SharePoint Site. Of course, one must be very careful with this kind of power, since creating Views with field references that don't exist in the List will certainly break the View if not the entire List itself. When we've added sufficient safety checks, we'll open the capability up as part of the CloudPrep product.

This week, we introduced the concept of using a hash algorithm to test whether files in SharePoint match those on our local drive. Use of a hash in addition to checking the file size and date stamps of a document ensures that the document has been uploaded into SharePoint and that it has not been corrupted in the process. We developed this ability in order to add credibility to Office 365 migrations where we may be moving hundreds of thousands or even millions of files, and we need to establish that the migration process has been completed satisfactorily. This capability can also be used to perform duplicate file detection, and we may develop a follow on product or feature to do just that later on.

Next week, we're planning to work on some important features that we feel are a must for getting this product to where we want it to be.

The first is the make sure that we can translate between Active Directory permissions on the local file system and users in SharePoint. The primary purpose here is to preserve meaningful data for Created By and Modified By fields in SharePoint; this is something we can't do yet. As part of this process, we'll be introducing PowerShell commands to add new users into SharePoint sites and manage groups. For most customers, this is probably of limited use. However, those with several hundred users or groups to manage will find it much easier to deal with these via PowerShell instead of using the SharePoint admin web pages. For consultants, it will make migrations faster by speeding up the time it takes to implement the security configuration. Our goal here is to lower the cost of our migration services.

The next things we do after that will be:

  • Download documents from SharePoint to the local drive
  • Assign metadata from CSV file as you upload documents
  • Flatten a folder structure as you upload it.


These are harder to do than you might think. I'll post more on this in coming weeks, including our challenges and progress updates.

Anouncing CloudPrep 2014 Migration Toolkit for SharePoint Online

We do a lot of Office 365 migrations. Most of these are for businesses with fewer than 50 employees. This should surprise nobody except maybe Microsoft, who seemed to be slow to realize that their cloud platform would have the most appeal to companies with limited budgets – or that most jobs in the US are provided by small businesses. Go figure.

Over the years, I’ve written several times about the challenges of moving from a conventional file store to Office 365. Fact is, it’s just not simple to do. It really makes sense to have an experienced IT professional help you make the move. I like helping customers make the switch, but doing so has presented interesting challenges for my business that I’m sure other SharePoint consultants share too.

Firstly, there are great third party tools out there for migrating files. We often use ShareGate and Content Matrix from MetaLogix. MetaVis is another great company that has great tools with lots of features. Fact is that even though these tools are great, they are also quite expensive. They’re feature rich, so really knowing the tool is a skillset of its own – and it makes good IT people hard to find when I need them to do a job. We also run up against serious limitations when trying to use these tools; sometimes we cannot find a way to use the tools to migrate the files in exactly the way we want to.

Second, some of my client already have a part-time IT person or managed services company that helps them service their PCs and on premises servers. Traditionally, we’re a SharePoint consultancy and we never set out to try and replace other IT folks; they need work too. They have the relationship with my customer, and the local presence needed for that on-site work. Over the years, I’ve seen that customers prefer to have their own local IT provider for most small requests. We needed to find a way to coexist with these other businesses in a way that would benefit us both.

Back in 2012, at the behest of a marketing consultant (who gave me lots of advice that was either bad or I couldn’t follow it at the time) I created a small tool called CloudPrep. This tool wasn’t much; I never had much confidence in it and so I never really promoted it. But, it did the work of renaming files that SharePoint didn’t like, and combined with WebDAV it was enough to make getting 20 to 50 GB of customer files into the cloud in a few days’ time. I released it into the wild, and CloudPrep has been getting downloaded a few times a week – mostly by other Office 365 consultants to my chagrin. Lesson learned and another checkmark for finding a way to compete with other IT providers; there are more of you than there are of me!

One problem I’ve noticed is that Office 365 migration budgets are small – I mean really tiny! That’s weird when you consider that for a 25 person company the ROI could be hundreds of thousands of bucks. But, we have been in an economic slump for something like 5 years now. I guess that takes its toll; even if you knew it would make you a thousand dollars next month, you can’t spend $100 today unless you have it to spare. Some companies are reluctant to spend even a few thousand to plan and execute.

There are a few tools that are in the “beer money” range. I tried FilesToGo once – and only once. It lacked some features that seems obvious to me, but made my client extremely angry. It didn’t have a lot of options either, one size fits all. I won’t discourage anyone from using it if it meets your needs, but I’m not going to risk my relationship with my clients on it. I am honestly surprised that after all this time, there’s nothing else in its price range.

I guess you could say that I’ve gotten fed up with this situation. Yet another migration we had to do where the current tools on the market couldn’t meet our needs for the client’s budget. That story gets old.

So, the boys in the lab and I finally built our own!

Announcing CloudPrep 2014! Forget everything you ever knew about that crappy tool we made back in 2012, because this is completely something at a whole new level.

CloudPrep 2014 is not one of those big expensive tools with a fancy GUI. It’s a set of PowerShell command-lets that work with SharePoint Online and your local file system. These commands and the sample scripts provided with them are designed to empower IT people and make migrating files to and from SharePoint Online a piece of cake.

These tools don’t replace an IT person or their experience. You’ll still need an experienced consultant to tell you how to organize your files, use metadata, overcome or avoid SharePoint Online limitations, and of course actually use the tools. You needed all that before anyway. The difference is that now much of this can be provided by your own experienced IT staff; or if you’re an IT consultant yourself, you can use our tool and make your small-business and small-budget migrations a breeze instead of a quagmire.

Our commands fall into basic categories: planning, preparation, file migration, and SharePoint management. We’re still putting the finishing touches on the product now. We’re hoping to have the Lite and Standard editions released to market sometime in February, with the Premium and Professional versions available as soon as March or April.

In the meantime, please take a look at our feature matrix and proposed pricing structure. There’s still time to collect some feedback. So, if you have a feature you’d like to see that isn’t here, then leave us a comment and let us know. Even if you don’t add a feature by the launch date, we’re planning to add even more features later. We’ll entertain any reasonable suggestion – except charging more for the product.

Like what you see and can’t wait to try it out? Contact us and I’ll give you a 15% discount if you purchase during the early access period.

Edition->Feature        Lite   standard Premium Professional
Release Date   Feb  Feb  March  April
Proposed Price Free $285 $576

$1,092

+$300 Per Tenant>2

Number of Office 365 Tenants Unlimited Unlimited unlimited Unlimited
Numbre of Site collections Unlimited Unlimited Unlimited Unlimited
Requires powershell 2.0 or higher Yes Yes Yes Yes
Requires Sharepoint client connectivity Yes Yes Yes

Yes

1 year support and Updates

(renewable Annually)

  Yes Yes Yes
Supported OS: Windows server 2008 or 2008 R2 N/A Yes Yes

Yes

Supported OS: Windows XP N/A   ?? ??
Supported OS: Windows Server 2003 N/A   ?? ??
Planning and Reporting        
Sizes and Numbers of items by folder, extention, ect. Yes Yes Yes

Yes

 

Check for Potentially Illegal file types   Yes Yes

 Yes

Folder and File Path Length Checking   Yes Yes

Yes

Permissions Checking for Local Files     Yes

Yes

Target URL Length Check Report     Yes

Yes

Upload Time Estimates      

Yes

File Preparation        
File Renaming for Illegal charaters Yes Yes Yes Yes

File Renaming for Illegal Paths

(_files,_forms)

Yes Yes Yes

Yes

Preserve Author and Editor for uploaded Files

  Yes Yes

Yes

Check for and Automatically ZIP files with illegal extentions (EXEs, Ect.)

    Yes

Yes

Check for and Automatically ZIP "_files" Folders

  Yes Yes

Yes

Migrate and Manage Files

       

Supports Network Mapped Drives

yes yes yes yes

Supports Network UNC Paths

yes yes yes

yes

 

Upload Entire Folder to Document Library

Yes Yes Yes

Yes

Upload Specific File to Document library

  yes Yes

Yes

Download Document Library to Folder

  Yes Yes

Yes

Download Specific File

  Yes Yes

Yes

Warns if Source Exceeds 5,000 items

  yes Yes

Yes

Warns if Target URL length Too Long

  yes Yes

Yes

Specify Content Type for Uploaded Documents

  Yes Yes

Yes

Specify Content Type for Top Level Folder

    Yes

Yes

Specify Content Type for Sub-Folders

    yes

Yes

Support for Documents Sets

     

Yes

Flatten Folder Structure with duplicate filename handing

    Yes

Yes

Flatten Folder Structure at 1 or more levels deep

     

Yes

Convert Folder Names to Metadata Fields

    Yes

Yes

Create Source URL Field for Uploaded Files

    Yes

Yes

Create MD5 Hash Field for Uploaded Files

     

Yes

Export Metadata to CSV File when Downloading Files

     

Yes

Synchronize of Local and Cloud files using File Modified Time

    Yes

Yes

Synchronize of Local and Cloud Files using File Modified Time+ MD5 Hash

     

Yes

Automation Features

       

Powershell command-lets

Yes Yes Yes Yes

Unattended Execution

  Yes Yes Yes

Sharepoint Management &

Development      

Create and Edit SharePoint Users

  Yes Yes Yes

Set Common Properties for Lists and Document Librarys

  Yes Yes Yes

Create and Edit Columns in Lists and Document Libraries

  Yes Yes Yes

Create and Edit Views Lists and Document Libraries

    Yes Yes

Copy a view to same or Different Document Library or list and site

    Yes Yes

Import and Export Site Columns

    Yes Yes

Import and Export Content Types

    Yes Yes

Import and export views

    Yes Yes

Add, Remove users and Groups, Permission Sets

    Yes Yes

 

CloudPrep Lite
This edition is a good fit for small file migration needs and try-before-you-buy. You can use it to do basic reporting on the structure of your files, rename files that are known to cause problems during migration, and upload folder structures to your SharePoint Online document libraries. In most cases it has a 99.7% or better success rate, and it produces a handy report so that your remaining files can be uploaded manually.

CloudPrep Standard
This edition includes a standard set of features designed to help you move files into Office 365 with a minimum amount of difficulty. You can upload and download large file collections without having to stand by the computer, perform multiple upload/download passes, and specify a default content type for files. Run it from anywhere, including various versions of Windows Server. We also include some additional pre-migration reporting tools that help to identify problems before you migrate your files.

CloudPrep Premium
For the seasoned SharePoint admin or IT professional, this edition includes features that will help you get the most out of Office 365 in the cloud. We include even more reports to give you a 360 degree view into any potential file migration issues. The file upload tool includes a variety of features for setting metadata and flattening folder structures.

CloudPrep Professional
This edition enables the true Office 365 IT professional to handle migrations for multiple clients. All the features of the Premium Edition plus advanced content type features including support for Document Sets. It also includes the ability to create MD5 Hash file uploaded files, which helps in detecting duplicate files and in determining that if two files are not the same even when their date stamps match.

Save Money for Your Small or Midsize Business by Moving to the Cloud

There are many small companies out there with a rack of servers in a closet. Years ago, this was the expected way that companies supported their internal operations. My company has one too. Many companies depend heavily on this equipment to perform vital functions for the business operation. E-mail and files typically live here - lots of files!

In recent years, there's been a shift to a new IT strategy called "the cloud". For small companies that may not have a lot of cash to make big changes, a move to the cloud can seem to involve a lot of risks and requires spending precious resources.

Today, I want to take a few minutes to explain some of the most compelling reasons that you might want to find a will and a way to turn that closet full of equipment off - because losing that ball and chain could help to set your business free.

Cloud Savings from Electrical Utility Costs
For starters, all that stuff running in your closet uses a lot of electricity. It's hard to tell how much exactly, because that depends on how old the equipment is and things like how many CPUs, drives, extra power supply it might have installed. Air conditioning costs energy too, and many people fail to take cooling costs into account when they try to estimate how much energy their computers use.

You can make some educated guesses based on the size of the circuit breaker on your equipment rack. For example, if you run everything on a single 20 amp circuit and it isn't blowing out like a Christmas tree in a hundred-year-old house circa 1974, then you are probably consistently pulling less than 18 amps and it's probably more like 15. Converted to watts, that's 1800 to a max. of 2400 watts. That's more than enough to run 5 servers with 500 watt power supplies - assuming you don't power them all up at one time. If you have fewer servers than that running, you either have older equipment that consumes more power or you aren't really using the circuit to its capacity.

1200 w at 120 v = 10A

500 w / 120 v = 4.15A

Another rule of thumb would be to assume about 550 watts per server, unless there's something fancy going on like it has a redundant power supply.

So let's use my own equipment as an example and I'll see if I can guess how much it costs me every month.

Here's my inventory:

  • Firewall
  • Domain Controller
  • 2 Virtual Servers
  • Database Server
  • Other Small Load Equipment: Wi-Fi Router, Network Switches, Battery Back UPS 


5 x 550 w = 2,750 w

2,750 w / 120 v = 22.9167 A

Maybe it's a little more than that if you include all the low end equipment.

This runs on a 20 amp circuit, so if I were really pushing 22A or more then I'd be blowing the circuit all the time, but I do know that if we add anything like a mini-fridge to the mix then we will trip the breaker, so I'm probably not far off. I could use this figure and call the overhead the cost of air conditioning.

Fortunately, I have another way to tell. I have these two APC 1500 VA back-up batteries and each is nice enough to tell me their load. Right now each is sitting at about 50% load. So, that's about the same as saying that we're running is 15 amps. This figure makes more sense, because you have to figure that the servers need a little extra capacity for starting up and such.

I could've come to the same conclusion by guessing that my equipment uses about 70% of its max. capacity. All these methods brings me to about the same figure.

My system uses 15A * 120v = 1,800 watts. I'll round it up to 2,000 watts to make the math easier and account for cooling costs and spikes in use that occur once in a while.

So, how much is that in money? The power company charges me per kilo-watt-hour. That's a fancy term for saying that if I use 1,000 watts for 1 hour, that's one unit on my electrical meter - for which they charge me $0.12.

24 hours in a day times an average 30.4 days in a given month equals 729.6 hours per month. Remember that this equipment runs 24 x 7 x 365, in case some employee wants to VPN in at an odd hour and get a little extra work done. So we have 2 kilo-watts times 729.6 hours times 12 cents. That's about $175.10 a month or $2,101.25 per year. Over time that really adds up.

What if I could cut that power consumption in half, by removing some of that equipment? If I had a thousand bucks, I could do a lot of things with that money instead. Here are some examples: 

  • Office 365 E3 plans for 4 employees
  • A small virtual server in the cloud with a VPN connection to my local network
  • Business-grade broadband internet service
  • A fancy office lunch for all the employees once per quarter
  • An extra grand for me to take home as a bonus

In fact, over five years this alone could pay for about 25 to 50% of the budget for moving to the cloud.

We do some really fancy stuff with our servers, but most companies are doing pretty ordinary things with their equipment. Here's some examples: 

  • Domain Controller
  • File server
  • Backup server
  • E-mail server
  • Anti-spam appliance
  • Company Intranet site
  • Remote Login / VPN / Terminal Server
  • Accounting Software
  • Other Customer Application Servers


If you replace that old equipment with cloud services and virtual servers in the cloud, you can eliminate a lot of these. In fact, only the domain controller and those last two items are particularly challenging to phase out completely. Depending on how your systems are configured, that could be as many as 3 servers (maybe more) that are just sitting there chewing up power that you could save.

Cloud Savings by Avoiding Upgrades to Hardware and Software


All of that hardware may be aging; the recession hit a lot of businesses that haven't had spare funds to update their servers since before 2008. That was 5 years ago, when Windows Server 2003 was still considered reasonably current. A lot of it isn't upgradable, because it's 32 bit architecture and won't support the newer operating systems, which means you have to figure hardware into your upgrade costs as well.

Even if your hardware is state of the art with the latest operating system, chances are good that you'll probably want to upgrade it sometime in the next 3 to 5 years. Depending on what the hardware does and what software runs on it will say a lot about how much you could save by freeing yourself from that burden.

Likewise, at some point you're probably going to want to upgrade Microsoft Office. Many companies say they're perfectly happy using Office XP or 2007; often, they just don't know about certain features that could be of really high value to them. Because they can't afford to upgrade, they never get the chance to discover the benefits on their own. Office 365 solves that problem because your Office desktop client software is included with the service.

For example, modern versions of Office have improved abilities to collaborate on documents when they're saved on a SharePoint server. Two people can edit the same Word document or Excel spreadsheet at the same time from two different computers. Most folks also don't realize that Excel has some very promising business intelligence features now that can let you crunch your business data in ways that could give your company the competitive edge.

One customer told us that because they were switching to a cloud architecture, they would be able to stop buying the more expensive laptops they'd been providing to their employees, in favor of units that we about half the price. If you have less than 10 employees that may not seem like a big deal, but if you're buying computers for a larger team, the multiplying effect can lead to formidable savings.

Here are some examples of some hardware and software costs you can save by switching to the cloud:

  • Typical mid-grade business server: $3,000 to $6,000 per server
  • Cheaper desktops or laptops: $250 to $1000 per user
  • Windows Server operating system: $1,000 per server
  • Exchange Server software: $1,000 + $120 per user
  • Microsoft Office client software: $400 to $700 per user depending on edition 


There are other miscellaneous software expenses too, like Remote Desktop Server (terminal server) clients, VPN devices, anti-spam appliances like the Barracuda, or backup solutions like Veritas. Having some of these in your company typically comes with annual support contracts that must be renewed - that's kind of like paying for cloud services without getting the cloud. You may not be able to discontinue all of these services, but especially for those which charge per user, scaling back the number of seats can save you a lot.

Cloud Savings by Reallocating IT Service Costs
Of course, computers don't take care of themselves. Some companies have an IT staff of their own, others hire managed services companies or freelance IT tradespeople to help maintain their computer systems.

These services come at a cost. A full-time IT person can cost $80,000 a year to keep on staff. Part time workers will usually charge consulting rates of around $50-100 an hour or more. Such a consultant might cost you $25,000 a year even if you bargain shop and only give him 10 hours a week. 

Such services are necessary. Backups need to be run. Users need help with malfunctioning software or broken equipment. Server drives will get full, fail, or both. Learning all those systems and which levers to pull in order to keep them running is a distraction from your business operation. Most business consultants agree that smaller companies should outsource their IT needs.

It might be tempting to think that you'll be able to cut the budget for IT support if you move to the cloud; after all your IT staff or MSP will have less equipment to maintain. The truth is that this will probably be a wash, because it's common to see both IT departments and managed service providers starved to the bone for resources. Likely, some of your resources will shift to supporting the new cloud solutions instead of the old infrastructure. Also, there are probably projects that have needed attention for a long time where you could redirect those funds or hours instead of cutting back.

So, look for changes in where you get your IT support, how it is delivered, and what platforms it will support - but don't expect to unearth a gold mine of savings by cutting back on IT work when you switch to the cloud. Fortunately, there are so many other places to find savings that it probably won't matter.

Cloud Savings from Stupid Accounting Tricks
Another thing to consider is that in some cases there are significant differences between CAPEX and OPEX, meaning that capital expenditures - those which result in obtaining assets - require different accounting treatment then ongoing expenses like your phone bill. Because cloud services are operating expenses, you may be saving money on stuff like business property taxes and depreciation if you go into the cloud.

Another thing to point out is that cloud services do not have to be paid all at once. For example, buying Office 2013 for 25 employees could mean coming up with over 17 grand up front, tapping into a line of credit, or having to phase the purchase in slowly. Getting that kind of money for big expenditures can also involve jumping through flaming hoops. Such obstacles might delay purchases you need to make, and they'll certainly drain your productivity.

Cloud services also scale much better than conventional server infrastructure. For example, you might provision an Exchange server that is reasonable for 15 employees. Over time, as employees are added to the company and old e-mail accumulates that server would be overburdened, thus accelerating the pace at which you'd have to spend more to upgrade it. Or, alternatively, you could plan ahead and buy a server that could support up to 30 employees, but then all that added expense is an opportunity cost and wasted resource for every year that you don't use the server to its full capacity.

Cloud services typically come with an annual agreement, just like your cell phone plan, which means there are some limits on how fast you could scale back if you have to, but you can increase capacity at any time. So, there's no excess supply except in cases where you shrink the company a bit - and your maximum liability is something you can plan for. With the traditional server all you could do is wait for users to drop to zero and then turn it off - just before hitting the light switch on your way out of the office.

Cloud Savings from Productivity Gains
This is the fun part that I always like to talk about, because people really overlook it when they're trying to find ways to save money - and this is where the real money is.

Suppose your small company grows, and you need to hire another office employee to handle the work. That probably costs you anywhere from $50,000 to $80,000 per year - maybe more depending on their qualifications, experience, and the value they bring to your company.

Suppose your company does less well than you'd like and you want to cut your staff. Everyone else would feel the pinch as their work is transferred to the rest of the team. The added workload affects morale, and productivity could drop - increasing the chances that you'll continue to slide downhill.

My point here is that whether your company is struggling or growing, both of these come with a cost. What if you could mitigate that cost by cutting out wasteful activities that aren't really productive but have just sort of become habits because you've always worked that way before?

If your business is like a lot of other companies, you probably have some pretty typical work patterns in your office. Here are some examples:

  • You have a network file share that you've been using for years; maybe you have everything going back to the early days of the company; there's an elaborate folder structure to keep everything organized, which has changed over time; finding things involves digging around in different folders until it turns up or asking the office admin if they know where it is.
  • Once in a while, somebody deletes a file off the network file share; either you don't ever find out about it, or when you need it you have to go to a backup since there's no recycle bin for the file share.
  • Since there's no official document retention policy - or way to automate it - old documents just pile up and lay around making everything else harder to find.
  • You have tons of documents living in e-mail; when you need a document you have to search Outlook to find it; sometimes you're not sure if it's the latest version or not.
  • You archive your old emails to gigantic PST files which you can only access on your work computer, because the file has to live on the network share in order to get backed up.
  • When you're on the road or working from home, you have to remote into a terminal server so that you can get access to all of your files at the office; you can't use your tablet or smartphone to do it; it's extremely slow compared to working on your home computer.
  • Your version of Office at home is different than the one you have at work, and so some of the stuff that you can do in the office can't be taken home with you
  • If there's an internet connection or electrical issue at the office, you can't really work from home, because VPN is down; business just shuts down for the day until the crisis is past.
  • If you do most of these things, chances are you could gain a lot of productivity by moving to the cloud. And, if you do any of these, chances are pretty good that everyone else in your office has the same bad habits and coping skills.

Logging into VPN, working with slow connections, foraging for documents, lugging portable drives back and forth, trying to find the correct version among duplicates, waiting for e-mail and file searches to finish running, being at specific computers to in order to complete certain tasks, and having to ask other people where to find that important file are all wasteful unproductive activities. Up to a certain point in time, they were considered necessary, just like people still consider driving to and from the office to be necessary - at least some of the time.

According to one McKinsey study, workers spend about 30% of their time reading and answering emails, 20% of the day looking for things, and 15% communicating and collaborating with their fellow workers. That's a whole workday every week spent looking for information, much of which may already exist inside your own company.

And yet, if each employee in a 25 person company could save just 2 hours a week by cutting down on how long it takes to find things, that'd add up to 50 hours a week in reclaimed productivity. In other words, you can add an entire virtual employee to the rolls without paying a penny - whether you simply avoid hiring another warm body or have to make due with less staff, either way you're looking at an effective savings of $50-80k.

The reality is that you can probably save a lot more than just 2 hours per week; that's just 24 minutes a day. If you think of it more like a worst case scenario, it's a pretty darn compelling argument to go ahead and make the change even if it costs you a little in the short run.

Think your company could benefit from a move to cloud architecture including Office 365? Reach out to us and we'll develop a custom migration plan, cost breakdown, and ROI.

How to Smoothly Migrate Files to Office 365 SharePoint Online

Many clients ask me what they can do to make their Office 365 migration into SharePoint Online go as smoothly as possible. Often, they're sincerely looking for a way to reduce the costs for everyone involved.

In part 1 of this series, I'll focus mainly on the technical complexities of moving files from traditional shared folders into SharePoint Online. In future episodes (that's what my shrink calls them) we'll take a look at things from the architectural and strategic perspectives.

First, let me say that Office 365 customers come from all sorts and sizes of business. Some have IT staff supporting hundreds of users. Others are just a handful of folks with one or two office administrators - who often work double-duty as the IT help desk. Whether you're big or small, IT savvy or technophobic, on the bleeding edge or upgrading from Windows 3.1, planning to use commercially available migration tools or making your unpaid intern Carl copy all the files by hand, there's something here for you in the lessons we've learned.

Communication Plans
They say that communication is key. Like any other project, this is absolutely true with a major migration like moving into SharePoint Online. There are lots of things that probably should go into every communication plan. In a future part of this series, we'll tackle the fuzzy stuff. For now, let's focus on some of the hard technical details that many project managers forget to include when they coordinate an Office 365 migration.

Locked Files
It is important for users to understand that files which are open are likely to interfere with the migration procedure. This can even include folks who have a network share open in Windows Explorer on their desktop. While it's possible to work around these issues, during critical times, it might be a good idea to encourage users to close unused folders and documents in Office.

Note this is especially true for files open in MS Office, as it not only locks the file but also creates an auto-recovery file starting with ~$ that can't and shouldn't be migrated into SharePoint Online.

Backup Window
Often users will want access to their documents right up until the last minute. Whether you're using ZIP files, Acronis, or some other strategy to do a last minute backup - really any backup method, it is important to let the users know after what point their files will no longer have a backup.

File Move Timing and Locations
If you're doing everything under cover of darkness, then one day folks will come into the office to find all their files have been moved. If you're migrating during business hours (my personal preference is to not stay away all night babysitting a file move) then some users will still be working up until the last minute. Either way, you need to communicate to the users when the files will move and where they are going. This is especially true if you're doing some reorganization - whether elective or forced by Office 365 limitations.

Remember to Include in Your Communication Plans:
When will the move be taking place, including regular updates
Instructions about closing windows or office apps - or logging out completely
Advice on when files will no longer be backed up
Information about who to reach out to if there seem to be problems after the move


Technical Gotchas

Permissions


File permissions are one of those things that we've found can create a lot of problems when migrating to Office 365. Even though you may be using an administrator account with high level access, there may still be the occasional file buried deep in a folder structure that maybe was copied from a user's desktop and dragged very limited permissions along with it. Or, there may be entire folders that somebody locked down - sometimes with good reasons long forgotten about.

Obviously, you can't do much with these files if you can't get to them. It can be even more frustrating to find this kind of issue late in the game, when they'll blow up a move operation or a ZIP archive that's only halfway done.

Illegal Characters in File Names
One of those things that can be really vexing about moving to Office 365 is the limitations on the characters you can use in file names. The restrictions on Windows file names are not as stringent as they are on the web. So many users will have files with ampersands, hash-signs, and other "illegal" characters. But, I've also seen really exotic characters such as the registered trademark and copyright symbols, long-dash, and other unusual stuff.

Here's something to consider. When you get into Unicode and upper ASCII, the list of invalid characters can be quite long. If you're screening against the list of known illegal characters, you might let one slip through. Consider checking against known legal characters rather than known illegal ones.

Why is it that Office 365 has these restrictions?
Well, a lot of it has to do with the way the web actually works. For example, characters like “#”, “&”, and “?” All have special meanings when they are used in URLs. When you upload a file to SharePoint using the web site, these characters get converted to special codes that are safe in URLs; this process is called URL encoding. However, the encoded filenames look like unreadable gibberish when they're viewed through Explorer View (WebDAV) and other places, so we try to avoid using them at all.

Still other special characters may be using something called Unicode that allows the display of things like Chinese and Japanese characters alongside English. Unfortunately, web addresses use older standards called ANSI and ASCII which only allows English characters (and a few extended characters that are implemented so inconsistently across all the different computer systems that we really can't rely on them anyway).
Oh yeah, and these restrictions count for folders too! And while we're at it, did I also mention that there are certain names you're not allowed to give to a file in SharePoint, and certain words you cannot end the file name with? 

Renaming these files by hand is extremely tedious; it's so tedious that I actually made my son do this as a rite of passage when he first joined us here as an intern at Liquid Mercury Solutions. Needless to say he was very excited to have software that would do this instead.

Fortunately, you can use PowerShell, download a tool, or buy any of several products that will take care of this step for you.  You'll want to have some input into how files ought to be renamed, and the configurability will vary from solution to solution.

If you hire IT folks to do your migration for you, then they should have at least one of these methods.  Here at LMS, we generally use a combination of PowerShell and products from MetaLogix and ShareGate to complete Office 365 migrations. Other good choices come from MetaVis, Dell Software (formerly Quest Software), AvePoint and others.

File Name and Folder Structure
But, file names alone are not the only problem. There are additional limits.

For example, ZIP files have a limit in how long any given file name can be, so good luck backing your files up if you go past this threshold. (Some of this can be avoided by using WinRAR or newer versions of WinZip instead of the built in Windows ZIP file utility.)

Windows also has a slightly longer limit of no more than 248 characters in a file name and no more than 260 characters for the full file path including folders. This can be problematic if the file was copied to a network location using a mapped drive, but you need to access it using the longer UNC file share name.

SharePoint itself has a limit that the URL pointing to a file, including the DNS name of the server and path to the document library, can't be longer than 255 characters. So woe to you with lots and lots of sub-sites - or long domain names like liquidmercurysolutions.com.

Blocked File Types
There are a number of file types that are blocked in SharePoint Online. This is done for security reasons, including DLLs and executable files (EXEs).

It's worth noting that Microsoft has sole discretion on what file types are blocked. I was recently very surprised during a demo I tried to upload an MSI file to show that it would get blocked - and surprisingly it worked! So check to see what the latest list of block files is before you start; this can be done from the Office 365 management portal.

Miscellaneous Poor Fit for SharePoint
In the course of evaluating content, you're likely to encounter files or even whole folders that just aren't a good fit for the cloud. For example, thumbs.db and desktop.ini files are great for making Windows UI more useful, but they serve no real purpose in SharePoint - and they can lock the file system which can complicate moves. Likewise, Office recovery files (those that start with ~$) or their analog in WordPerfect (*.TMP) can be a nuisance since they're marked as hidden and system files.

Large files like AVIs, ISOs, and ZIPs may be a poor fit for SharePoint. While technically you can put up to a 2GB file into SharePoint Online, some folks just don't have the bandwidth to put the files into the cloud and also have them be useful. Even in circumstances where you're just archiving the file, you should consider that you might spend a very long time uploading it only to find that the operation will time out and the effort will be wasted. You best bet is to test your bandwidth, then decide where to draw the line.

A folder full of executable programs - and probably all the non-executable stuff in that folder too - is a poor fit for SharePoint. Use your own judgment, and don't be afraid to exclude certain parts of the folder structure altogether if they seem to support running software.

  • Before You Start: File Structure Check
  • Permissions
    • Take ownership of all the files you need to archive and/or move.
    • Make sure you have permissions to all the files by overwriting the current permissions.
  • Illegal Characters
    • Avoid &, ?, #, and others - get the complete list here
    • What tools are you using to rename files?
    • Check folders as well as files
    • Legal characters or illegal characters
  • File Types / Filename Extensions
    • All files must have an extension
  • Check for blocked file types - get the complete list here
    Length of Individual File Names
    • Beware any single file with a name longer than 60 characters; here there be dragons!
  • Best bet: try to keep them 50 characters or less
    Length of Folder Paths
    • Can your backup solution handle long file paths?
    • Check the windows path of all files to ensure it doesn't go over 248 characters
    • Check the intended destination URL for each file to ensure it isn't longer than 255 characters.
    • Best bet: try to keep them under 200 characters
  • Misc. "poor fit" for SharePoint
    • Delete desktop.ini, and thumbs.db files
    • Move or otherwise deal with any ~$*.* or *.TMP recovery files
    • Test your bandwidth to see if it supports very large files
    • Consider what to do with very large files such as AVI, ISO, ZIP, etc.
  • Optional, if using ZIP with legacy support:
    • Make sure your archives will not exceed the 4GB limit.
    • Make sure your archives will not have more than 65,535 files/folders total.

Office 365 and SharePoint Limitations

As if all this were not enough to consider, you're going to have a number of limits and options on top of this when moving into the SharePoint Online platform.

A lot of these impact choices about site and information architecture, which is where Liquid Mercury Solutions comes in to help our customers. Even if you're using custom tools, it's often best to have a SharePoint specialist who can help you choose the right way to organize your sites and files.

Here are a few of the not-so-fuzzy limitations that we often have to work around.

Begin Your Start: Check for SharePoint Limitations

  • User quotas - if you're not a site collection admin
  • No single file can be > 2 GB
  • No more than 5000 items in a single folder
  • No more than 100GB in a single site collection ("supported limit" is 200GB)
  • No more than 30,000,000 items total in a single document library, including all documents and folders
  • SkyDrive Pro has its own weird limits which are different


Other Best Practices


There are lots of other best practices to take into account when moving to SharePoint Online - so many that we'll have to save them for another episode in this series. So, tune in next time!

Meanwhile, here's some more information for you. We recently saw a very thorough presentation on this topic given to the Baltimore SharePoint User's Group by my friend Mark McGovern from MetaLogix. And, our colleagues over at ShareGate have written a good article that starts to scratch the surface on some of the complexities involved.

If you'd like to purchase tools to help with your Office 365 migration, or if you want a professional consultant's advice please contact us and we'll be happy to help you. If you find these tips helpful, and want to give migrating to SharePoint Online a try on your own, please consider joining our free support network. Whatever path you decide to take, we wish you the best of luck!

--------------------------------------------------------------------------------------------------------------------------------------------