PnP Provisioning PowerShell, Site Scripts or CSOM scripts – which one to use and when?

There are various approaches to plan and automate the process for Site creation and management of SharePoint Online Sites. In this blog we will look at these options and how to use with a best possible approach.

Pnp Provisioning PowerShell is a great way to automate creation of SharePoint assets through an xml or pnp template file using PowerShell. Similarly, Site scripts and site design allows us to create Site using JSON templates and also allows call to any Provisioning automation scripts or use a Template for custom implementation. SharePoint CSOM is also a way to apply extensive custom changes onto a site. If you would like to know more about each of these individual approaches, check the blog below

1. PnP PowerShell and SPO PowerShell

2. Site Scripts and Site Design

3. SharePoint CSOM scripts

So the most important question is now when to implement what and when. Actually there are so many scenarios that play into it that there is no correct answer to it. Hence in this blog, I will try to put provide some pros and cons for each of them and provide a plan to consider the best options when considering an approach.

PnP PowerShell & SPO PowerShell

One of the powerful admin tool is PowerShell. It has a greater extensibility with regards to modules and also supports C# method implementation with proper dll references.

PnP PowerShell adds to that and allows Admins (both Tenant and Site Collection Admins) to manage and implement automation workloads from scripts

Pros:

1. Easy installation and cmdlet usage

2. No hosting requirements besides a system where PowerShell (>5.0) is installed

3. One Line cmdlets for most complex SharePoint implementation scenarios

3. Error handling and tracing support

Cons:

1. Parallel tasking for more data intensive operations is challenging

2. Not much options for extensibility besides the the ones provided

3. Not all Site Operations such as taxonomy field updates (unless do XML schema update) are supported yet

Site Scripts and Site Designs

This is the latest way of creating and applying changes onto modern sites using a custom template called Site Designs. Site designs extend the OOB templates allowing to add more site assets than provided.

Pros:

1. Simple scripting process (use JSON) to create the script file for site assets and apply features

2. Allows associating a custom theme to the template.

3. Allows extensibility to more complex process by supporting calls to Flow

Cons:

1. No support yet for complex structures such as taxonomy fields, web part implementation, content type support

2. No hooks or support yet to wait for custom provisioning process to finish

SharePoint CSOM scripts

SharePoint CSOM allow us to programmatically script elements using the SharePoint object model. This allows us to do script elements that can pertain to particular elements on the site and have more extensibility into component handling.

Pros:

1. The most extensive model available for SharePoint Online that allows us to add/change elements

2. Work on components that need complex provisioning requirements such as dynamic Navigation components, Library and Site settings, default folders etc. More info at here.

3. Provides extensibility for the above two options through explicit referencing when needed

Cons:

1. Need additional hosting outside of Office 365 via Azure Function, App or equivalent

2. Need maintenance for any breaking changes

The Verdict:

Considering the three options above, each of them have their strengths and weaknesses. The best approach would be plan and use them according to the requirements. Below are few recommendations to combine them for each type of elements in SharePoint Sites.

Note: The options listed below are valid as of Nov 2018 and might change as they are updated from MS
Scope\Tools PnP and SPO PowerShell SharePoint Online CSOM Site Scripts
Site Collections Yes Yes No
Site / Web Yes Yes No
Office 365 groups Yes Yes No
List/Library Yes Yes Yes
Theme Yes Yes Yes
List Item Yes Yes No
Content type/Fields/Views Yes Yes Yes
Client Side Pages Yes Yes No
Folders Yes Yes No
Hub settings Yes Yes No
App Deployments Yes Yes Limited
List Configuration Limited Yes No
Site Settings Yes Yes Limited
Taxonomy Store No Yes No
Taxonomy Fields No Yes No
Navigation Limited Yes Limited
Site Search Settings Yes Yes No
Enterprise Search Yes Yes No
Security Yes Yes Yes

Conclusion

In this blog we compared the various options for Site Provisioning and how those could be used to create and prepare site components.

 

 

 

Walkthrough of Site Provisioning process using PnP PowerShell

In the previous blogs here, we have looked at the Provisioning process for a complex Team site. Much of complexity was easily handled by the PnP Provisioning process.

In this blog, we will look at the similar Provisioning process but from an Admin point of view and use PnPPowerShell for create and provision the site.

Steps:
The steps are actually quite simple and could be done quickly.

  1. Build a Template Site to be used for creating the Provisioning Template
  2. Manually apply changes to the Template site and extract the Provisioning template from the Template Site using the Get-PnPProvisioningTemplate
  3. Create the SharePoint Team site to apply the template
  4. Apply the Provisioning template to the above site using the Template obtained in Step 2
  5. Finalize the creation with any remaining changes.

Before we start looking at the above steps in detail, below are few key items to keep in mind during the provisioning process. Some of the these will become clear as we go through the detailed process steps

1. During the Get Template process, one of the key switches is to include/exclude any dependancy items which are not related to the new site. More details with Get-PnPProvisioningTemplate section

2. During the apply template process, only the differential changes are applied. Hence it is safe to apply the template many times or repeatedly in case of an error occurs during the process

3. Since the apply template process only adds differential items, any explicit changes for eg. left navigation default links removal needs to be done after the apply template process is complete.

4. The apply template works by matching the Provisioning Template schema with latest PnP code schema. Hence, if you are working with a old template then use the Schema switch as specified below.

Get Template Process

The first step would be to install the PnP PowerShell module. Check this blog here to see how to get started with PnP PowerShell.

Next create a template site that will be used to create the new sites.

After the site is created, we could create a template from it using the Get-PnPProvisioningTemplate cmdlet. Some of the key switches while creating the template are below.

Helpful Switches for Get-PnPProvisioningTemplate cmdlet

-ExcludeHandlers – Custom handlers to exclude settings or elements from the site sucha as ApplicationLifecycleManagement, Site Security, WebApiPermissions etc.
-ExcludeContentTypesFromSyndication – Exclude the Content Types pushed from the Hub. This is generally helpful if content types no longer persist and could cause conflict issues.
-ExtensibilityHandlers – Allows to specify ExtensbilityHandlers to execute while extracting a template for doing custom actions such as exporting pages schema into PnP export
-Handlers – Custom handlers to explicitly include settings or elements from the site such as Audit settings, Features, Fields, Search Settings, Term Groups etc.
-Schema – PnP Schema version of the exported template
-IncludeSearchConfiguration – Include the Search Configuration of the site
-PersistMultiLanguageResources – Persist Multilingual files

Apply Provisioning Template Process

After the template is extracted, we could apply this template to any newly created site. Before application of the template, we will create a new modern site using the PnP PowerShell command below

Next is to apply the template which takes about 20-25 minutes for a complex template with about 20 libraries, 30 site columns, 15 content types etc. Since the process takes so much time, it would be good to trace log any issues and error handle the application process.

The Apply Provisioning Template cmdlet details are below. It is important to note some of the switches for the cmdlet as it does apply some custom settings for the template.

Helpful Switches for Apply-PnPProvisioningTemplate cmdlet

-ClearNavigation – Clear the Navigation of the site before template is applied
-IgnoreDuplicateDataRowErrors – Ignores to stop the script because of duplicate daa errors
-ExcludeHandlers – Exclude the handlers when applying template
-ExtensibilityHandlers – Apply the extensibility handlers for the custom scripts

Conclusion

In this blog we saw how we could use PnP PowerShell and PnP Provisioning Template to apply custom template to a newly created site in order to automate the creation process.

Provisioning complex Modern Sites with Azure Functions and Flow – Part 3 – Post Provisioning Site Configuration

In the previous two blogs part 1 and part 2, we looked at steps to create a Modern team site and apply a custom provisioning template to it. In this blog, we will have a look at the steps for the post provisioning process to implement site specific requirements. Some of them could be:

1. Apply default values to list fields

2. Create a bunch of default folders

3. Manage Security groups (SP level) and permission level.

4. Navigation level changes

5. Add/Enable web parts or custom actions (SPFx extensions)

Most of the above steps are part of SharePoint Provisioning processes for a long time, just are less complex now with Provisioning templates doing a lot of heavy lifting. I will be writing very soon about PnP Templates, do and don’ts.

Pre-requisties

One key point to add is that, with Modern Team Sites and Office 365 Groups we cannot add AD security groups into a Office 365 Unified Group. For more information, please see this link.

The apply template process (link) takes about 45-90 min (long running process) for complex templates so wouldn’t be possible to start the post process on a flow wait state. Hence, we could trigger the post provisioning process on update of an inventory item or poll an endpoint for the status. In our case, we triggered the process when updating the inventory list with a status that apply template process is complete.

Post Provisioning Process

1. The first step of the Post Provisioning proces is to make sure that noscript is enabled on the team site (link) and all dependencies are ready such as Term stores, Navigation items, Site Pages, Content types, site columns etc. For a complex site template, this step will check for failure conditions to make sure that all artefacts are in place.

Note: The below sequence of steps could vary based on the solution and site structure in place 
but this was the faster way to isolate issues and ensure dependencies.

After the failure checks are done, we will start with the Site structure changes and navigation changes. For implementing navigation changes, check the blogs here and here.

2. Next, we will update any site specific site columns, site content types and permission level changes. A upcoming blog will have more details to this.

3. After that, we will update the changes for list structure, we will move to list specific updates such as default value setting, modifying list properties etc.

4. Next let’s move on to Apps and Site Pages updates. These changes take time because of SharePoint ALM lifecycle any possible duplications. So error handling is the key. Please check the blog here and for steps to deploy app and web parts here.

5. Before we finalize the site, let’s provision folders and metadata. This is not a simple process if you have to set metadata values for large number of folders like in our case 800 recursive folders (child in parent). So we will use the metadata file override. All the values in the defaults file have to be hardcoded before override.

Note: The metadata file override is not generally a good approach 
because of possible corruption that renders the library unusable
so do error handling for all cases. For the CSOM approach check here.

6. Finally, we will set the site Property bag values for Site to set some of the tags making the site searchable. Here is the blog for the same.

Conclusion

The above we saw the final process of Site Provisioning process with setting up site properties and attributes for preparing the site before handing it off to business for use.

 

Provisioning complex Modern Sites with Azure Functions and Flow – Part 2 – Create and Apply Template

In the previous blog here, we got an overview of the high level Architecture of a Complex Modern team site provisioning process. In this blog, we will look at the step 1 of the process – Create and Apply template process, in detail.

Before that, below are few links to earlier blogs, as a refresher, to prerequisties for the blog.

  1. Set up a Graph App to call Graph Service using App ID and Secret – link
  2. Sequencing HTTP Trigger Azure Functions for simultaneous calls – link
  3. Adding and Updating owners using Microsoft Graph Async calls – link

Overview

The Create and Apply Template process aims at the following

  1. Create a blank modern team site using Groups Template (Group#0 Site template)
  2. Apply the provisioning template on the created site.

Step 1 : Create a blank Modern team site

For creating a modern team site using CSOM we will use the TeamSiteCollectionCreationInformation class of OfficeDevPnP.  Before we create the site, we will make sure the site doesn’t already exist.

Note: There is an issue with the Site Assets library not getting intialized 
when the site is created using the below code. 
Hence, calling the EnsureSiteAssets library is necessary.

Step 2:  Apply the Provisioning Template

Note: The Apply template process is a long running process and takes from 60-90 min to complete 
for a complex provisioning template with many site columns, content types and libraries. 
In order to prevent the Azure function from timing out, it is required to host the Azure Function 
using a App Service Plan instead of a Consumption plan so the Azure function 
is not affected by the 10 min time out. 

For the Apply Provisioning Template process, use the below steps.

1. Reading the Template

It is important to note that the XMLPnPSchemaFormatter version (in the code below) must match the PnP version used to generate the PnP template. If the version is older, then set the XMLPnPSchemaFormatter to read from the older version. In order to find the version of the PnP Template, open the xml and look at the start of the file

PnPTemplateVersion

2. Apply the Template

For applying the template, we will use the ProvisioningTemplateApplyingInformation class of the OfficeDevPnP module. The ProvisioningTemplateApplyingInformation also has a property called HandlerToProcess which could be used the invoke the particular handler in the provisioning template process. Below is the code for the same.

After the apply template process is complete, since the flow will have timed out, we will invoke another flow to do the post process by updating a list item in the SharePoint list.

Conclusion

In this blog, we saw how we could create a modern team site and apply the template on it. The next blog we will finalize the process by doing site specfic changes after applying the template.

Provisioning complex Modern Sites with Azure Functions and Microsoft Flow – Part 1 – Architecture

In one of my previous blog here,  I have discussed about creating Office 365 groups using Azure Function and Flow. The same process could be used also to provision Modern Team sites in SharePoint Online because Modern Team Sites are Office 365 groups too. However, if you are creating a Complex Modern Team Site with lots of Libraries, Content types, Termstore associated columns etc. it will challenging to do it with a single Azure Function.

Thus, in this blog (part 1), we will look at the Architecture of a Solution to provision a complex Modern Team Site using multiple Azure Function and Flows. This is an approach that went through four months of validation and testing. There might be other options but this one worked for the complex team site which takes around 45-90 mins to provision.

Solution Design

To start with lets’ look at the solution design. The solution consists of two major components

  1. Template Creation – Create a SharePoint Modern Team site to be used as a template and generate a Provisioning template from it
  2. Provisioning Process – Create a SharePoint Inventory List to run the Flow and Azure Function. There will be three Azure Functions that will run three separate parts of the provisioning lifecycle. More details about the Azure Functions will in upcoming blog.

Get the Provisioning Template

The first step in the process is to  create a clean site that will be used as a reference template site for the Provisioning template. In this site, create all the lists, libraries, site columns, content type and set other necessary site settings.

In order to make sure that the generated template doesn’t have any elements which are not needed for provisioning, use the following PnP PowerShell cmdlet. The below cmdlet removes any content type hub association, ALM api handles and site security for provisioning requirements.

Get-PnPProvisioningTemplate -Out "<Folder location in drive>" -ExcludeHandlers ApplicationLifecycleManagement, SiteSecurity -ExcludeContentTypesFromSyndication

The output of the above cmdlet is ProvisioningTemplate.xml file which could be applied to new sites for setting up the same SharePoint elements. To know more about the provisioning template file, schema and allowed tags, check the link here.

ModernSitesProvisioningFlow_GetTemplate

Team Site Provsioning Process

The second step in the process would be to create and apply the template to a Modern SharePoint Team site using Flow and Azure Function. The detail steps would be as follows:

1. Create an Inventory list to capture all the requirements for Site Creation

2. Create two flows

a) Create and Apply Template flow, and

b) Post Provisioning Flow

3. Create three Azure Functions –

a) Create a blank Modern Team Site

b) Apply Provisioning Template on the above site. This is a long running process and can take about 45-90 min for applying a complex template with about 20 libraries, 20-30 site columns and 10-15 content types

Note: Azure Functions on Consumption plan have a timeout of 10 min. Host the Azure function on an App Service Plan for the above to work without issues

c) Post Provisioning to apply changes that are not supported by Provisioning Template such as Creating default folders etc.

Below is the process flow for the provisioning process. It has steps from 1 – 11 which goes from creating the site to applying it. The brief list of the steps are as follows

  1. Call the Create Site flow to start the Provisioning Process
  2. Call the Create Site Azure Function
  3. Create the Modern Team Site in Azure Function and set any dependencies required for the Apply template such as Navigation items, pages etc, and then return to flow
  4. Call the Apply Template Azure Function.
  5. Get the previously generated ProvisioningTemplate.xml file from a shared location
  6. Apply the Template onto the newly created Modern site. Note: The flow call times out because it cannot wait for such a long running process
  7. Update the status column in the Site Directory for the post provisioning flow to start
  8. Call the Post provisioning flow to run the Post provisioning azure function
  9. The Post provisioning azure function will complete the remaining SharePoint changes which were not completed by the apply template such as, set field default values, create folders in any libraries, associate default values to taxonomy fields etc.

ModernSitesProvisioningFlow_ProvisioningProcess

Conclusion:

Hence in the above blog, we saw how to create a provisioning process to handle complex modern team site creation at a high architectural level. Next, we will deep dive into the Azure functions to create, apply template and post process in the next upcoming blogs.

Happy Coding!!!