Tag Archives: Import

This article covers how to import publishing pages to SharePoint.

This can help you if you need to bulk create pages in SharePoint and you have the data stored in a database or spreadsheet or if you need to migrate pages from another content management system such as WordPress.

yourpages

Scenario

We are going to assume that you have your page data in a database.  For this example we are going to use MySQL and the WordPress schema.  As you will see in the coming steps this could just as easily be any OleDB or ODBC data source and any schema.

Import Tool

We will use the free Import for SharePoint toolset to import the pages.

You download it from here.

When you download and install the import tool you will have full documentation and additional example import configuration files which will help further.

The Source

Column

The source data must contain a column with the HTML mark-up in it.

In our example content this is called ‘PageContent’.  Inside it looks a bit like this.

Select Statement

Now from the database source we need to ‘Select’ the data that will create our pages.  Since we are working with WordPress in this example the data is in wp_posts as we see below.

This select statement is also, cleverly, giving us the destination page name and setting up the page to be automatically published.

Import Configuration File

This is the file that tells Import for SharePoint how to create the pages in SharePoint.

The schema is fully explained in the documentation but the important bits for this exercise are explained here;

DestinationItemType

So we want to create publishing pages (We could alternatively create wiki pages, modern SharePoint pages, site pages or blog posts) but lets stick to the most common (publishing pages) for now.

<DestinationItemType>PublishingPage</DestinationItemType>

PageLayoutASPXName

So if your source select statement does not have a column of this name then “ArticleLeft.aspx” will be used.  If you want to use another page layout then ensure you select statement returns a column of this name containing the name of your desired page layout.

<PageLayoutASPXName>PageLayoutASPXName</PageLayoutASPXName>

ImportMapping

This bit maps your HTML data (in the column PageContent) to the SharePoint page content (field Page Content).

<ImportMapping xsi:type=”ImportMapping_String”>
<DestinationField>Page Content</DestinationField>
<SourceColumn>PageContent</SourceColumn>
</ImportMapping>

Execution

Ok so rather than re-invent the wheel we’ll let you read the documentation installed with Import for SharePoint on this one.

Result

Ok so originally in WordPress the page looked like this.

source

And now in out of the box SharePoint it looks like this.

result

Great, but seems a bit simplistic

Ok so we have shown how to import publishing pages into SharePoint.

Realistically a project is always going to be more complicated than that.

So lets talk about real life….

Targeting Branded SharePoint

Page Layout

So the destination is likely to be branded?  That’s no problem we’ve already talked about PageLayoutASPXName and custom branded SharePoint really just means using a different page layout.

Content Type Fields

But the destination page has extra fields, like managed meta data “Tags”, a Byline, an Article Date?  Again no problem you just need more of these ImportMappings to map data from your source into those additional SharePoint fields.

<ImportMapping xsi:type=”ImportMapping_String”>
<DestinationField>Title</DestinationField>
<SourceColumn>post_title</SourceColumn>
</ImportMapping>

Data Manipulation

So what if the source data is not in the exact format that SharePoint needs?

No problem this manipulation can be done in SQL as shown below.

WordPress was never going to contain a column giving us a file name like “MyPage.Aspx” so we create one on the fly using concat here.

If (when?) your manipulations get too complex for inclusion in the SQL statement (on the fly) you can directly manipulate the source table, just make sure you take precautions if the source data is used by anything else (like working from a copy).

So what does this get used for?

We have seen this approach used for the following;

  • Legacy Content Management System (CMS) migration.
  • Bulk creation of pages from Excel
  • Scan to Mark-Up / Republishing – Loading data that has been scanned and OCR’d into pages.
  • WordPress to SharePoint Migration
  • Drupal to SharePoint Migration
  • Joomla to SharePoint Migration
  • Custom Intranet to SharePoint Migration

Great, Makes more sense now but I’m still an bit unsure

No problem just get in touch.

Share

This article covers how to import documents to SharePoint.  Creating a folder structure, meta data, importing files with content types and why it is important to do this with some consideration.

yourstuff

Scenario

We are going to assume that you have your documents on a file share / disk and that you have gathered your meta data in Excel, though your meta could quite easily be in a SQL or an Access database.  We will further assume that the documents being uploaded are Staff Employment Records

Import Tool

We will use the free version of Import for SharePoint toolset to import the files.

When you download the import tool you will have some Excel files, import configuration files, and screenshots of the content types which match this scenario.  Using these will make the next steps much easier.

The import tool has a lot of functionality which is covered in the documentation

Bulk Folder Creation

We need a folder for each employee in our scenario.

We assume that you have created a document library, attached a custom “Staff Folder” content type and to that some site columns.

Using the import tool we can create the folders from our spreadsheet.  The sheet is shown below.

excelfoldersource

From this the import tool will create a folder structure in your SharePoint library. Import meta data such as employee number is attached to each folder.

createdfolders

File Import

Now we can import our files into SharePoint.

We assume that you have created a document library, attached custom “Staff Document” and “Staff Disciplinary” content types and to those some site columns.

We can use the Excel spreadsheet as the import source.

excelfilesource

Once the import has processed this the files will have been imported into the correct locations and with the correct meta data set against each one.

uploadedfiles

Why did we do this?

Ok so now we have a good structure to support common requirements.

Retention

How so let us assume that HR want to delete Staff folders 20 years after staff have left the business.  We can add this retention policy onto the staff folder content type and for employees who have already left we have the date already set (See Bulk Folder Creation) .

retention

Ok, so usually it’s a bit more involved that this but you get the point.

Search

Adding meta-data gives us a better chance of an item showing up in search results and in the instance of managed meta data will give us access to refiners on the search results page.

5000 Item Per Folder Limit

Ok so we know it’s not a good idea to have more than 5000 items in a folder.  But doing our import as set out in this article you should be able to design a great structure that works inside this boundary even it your original folder structure did not.

So is this a packaged solution for Staff Records?

Well the reality is that the treatment of employment records will vary for each jurisdiction, can often complicated by different treatment for pension records,  and the SharePoint implementation will change dependent upon whether you have a HR system and how that works.

That said it’s a great demo scenario and hopefully demonstrated some techniques will can be applied in all areas of your work.

Share

This article covers how and why to do a considered file share import to SharePoint.  Creating a folder structure, meta data, importing files with content types and why it is important to do this.

Before you import your files you have hopefully prepared them and maybe have some Excel spreadsheets to import from.

No?  Check out our article on preparing for import.

Scenario

So we are going to stick with the staff folder scenario prescribed in the above article.  Your needs are likely to be different but you should be able to transfer the thinking and techniques demonstrated here.

Import Tool

We will use the Import for SharePoint toolset to import the files.

When you download the import tool you will have some Excel files, import configuration files, and screenshots of the content types which match this scenario.  Using these will make the next steps much easier.

 

Bulk Folder Creation

We need a folder for each employee in our scenario.

We assume that you have created a document library, attached a custom “Staff Folder” content type and to that some site columns.

Using the import tool we can create the folders from our spreadsheet.  The sheet is shown below.

excelfoldersource

From this the import tool will create a folder structure in your SharePoint library. Import meta data such as employee number is attached to each folder.

createdfolders

File Import

Now we can import our files into SharePoint.

We assume that you have created a document library, attached custom “Staff Document” and “Staff Disciplinary” content types and to those some site columns.

We can use the Excel spreadsheet as the import source.

excelfilesource

Once the import has processed this the files will have been imported into the correct locations and with the correct meta data set against each one.

uploadedfiles

Why did we do this?

Ok so now we have a good structure to support common requirements.

Retention

How so let us assume that HR want to delete Staff folders 20 years after staff have left the business.  We can add this retention policy onto the staff folder content type and for employees who have already left we have the date already set (See Bulk Folder Creation) .

retention

Ok, so usually it’s a bit more involved that this but you get the point.

Search

Adding meta-data gives us a better chance of an item showing up in search results and in the instance of managed meta data will give us access to refiners on the search results page.

5000 Item Per Folder Limit

Ok so we know it’s not a good idea to have more than 5000 items in a folder.  But doing our import as set out in this article you should be able to design a great structure that works inside this boundary even it your original folder structure did not.

So is this a packaged solution for Staff Records?

Well the reality is that the treatment of employment records will vary for each jurisdiction, can often complicated by different treatment for pension records,  and the SharePoint implementation will change dependent upon whether you have a HR system and how that works.

That said it’s a great demo scenario and hopefully demonstrated some techniques will can be applied in all areas of your work.

Share

Migration of file shares into SharePoint is neither a new nor unusual requirement.

That said it is often done without appropriate preparation resulting in a poor end result that does not meet the overall objectives of the organisation.

It is a common misconception that you can get the full value out of SharePoint simply by dragging and dropping your existing file structure.

In this article we will run through a simple preparation example to see how easily an improved outcome can be achieved.  The article is based around staff (employee) records but the principals apply equally to most profiles of data.

The article covers;

  • Scenario – Be clear on the profile of data that you are going migrate.
  • Why Migrate? – Make sure there is a case for migration.
  • Design and Implement the Destination – Migrated data needs somewhere to live.
  • Catalogue the File Share – Catalogue your file shares, get a deeper understand of what is there.
  • Prepare the Excel Spread Sheets – Use Excel to cleanse the data and supplement meta data.
  • Get User Input – Use Excel to garner use input, particular where the users might need to plug in additional meta data missing from the current source.
  • Preparation Complete – You should have a source suitable as input to a migration process.
  • Importing the files – How to get the files into SharePoint

Scenario

Our theoretical scenario mirrors what is so often encountered when dealing with file shares.

We have a basic file structure.  At the outer nodes the files are stored.  The “meta data” is inferred by the location of the file in this structure as shown below.

Scenario_HR

The file share is the only data source.  There is no Staff Database.  If there was then this might be handled differently with that database providing some of the data.

The example has you have now seen is for some Staff Records.  Such records need to be stored in a compliant manner and retention scheduling is key to ensure that we retain only the correct records for each staff member.  The requirement in this post is vastly simplified in comparison to most Staff Record scenarios but it serves to illustrate the concepts very well.

Why Migrate?

Be clear on why you are migrating the data into SharePoint before you start the migration process.  You may need to design the migration process to ensure that the desired benefits are achieved.

In our scenario the key drivers are;

  • Compliance – Specifically data retention scheduling.
  • Efficiency – Consolidation into SharePoint.
  • Efficiency – Ease of use.
  • Efficiency – Process automation.

Design and Implement the Destination

Before you execute migration you need to have a destination to migrate into.

This is key for two reasons;

  • Migrating a live file share which is being updated is harder to manage.
  • Until you have designed and implemented the destination you won’t necessarily know how to define the import sources, in this example the two Excel spreadsheets.

For our scenario we have implemented;

  • A record centre – /sites/Staff/Records.
  • A record library – Records
  • A top level folder “Staff”
  • A content type for each staff folder “Staff Folder” which has a date of leaving field and retention action set from that date.
  • A content type for each staff record “Staff Record” which has an employee number field.
  • A content type for each disciplinary record “Staff Disciplinary Record” which has an Disciplinary Date and retention action set from that date because these records are kept for a shorter period of time.

It is sometimes useful to catalogue the file shares as part of this design process.  This will give you an insight into the scenarios that your destination will need to cope with.  The file shares can then be re-catalogued for migration at a later date.

Catalogue the File Share

To catalogue the file share we can use a PowerShell script.

You can run the script either from a PC as a user with access to the file share OR from the server hosting the file share.  The script below will audit both the files in the file share and the directories, each to separate CSV files.

The CSV files will contain the basic information that is available from the file system.

Files

Directories

Tip: Try and use UNC paths instead of mapped drive letters when cataloguing file shares.

If you are unsure how to execute PowerShell scripts then pop “how to execute a powershell script” into your favourite search engine.

Prepare the Excel Spread Sheets

The CSV files can be imported to Excel and turned into a spreadsheet.

This will enable us to automate the population of meta data that is so important to the success of migration projects

Folders

In Excel we can easily populate some extra columns (shown here in green).

Here we are going to create an import source which will create a folder of content type “Staff Folder” for each staff member.  This is going to have the Date of Leaving (which will drive retention schedules) and the staff name as meta data.  The employee name and the employee number will be extracted from the folder name from the file system using an Excel formula.

Excel_HR_Folders

Files

In Excel we can easily populate some extra columns shown in green.

Here we are going to create an import source which will import all of the files for the Staff.  The employee number is extracted from the folder name which will tell us destination folder and the set some meta data against each document.  This is extracted from the folder path.

Excel_HR_Files

Get User Input

One of the strengths of Excel is that most users will have skills in using it.

What this enables us to do is use it to capture from the user base any additional data and permit the users to generally cleanse the data.

Tip: Give the users some guidance notes on how the Excel spreadsheets should be completed.

Folders

Here the user has completed the date of leaving field which in turn will allow SharePoint to manage the retention schedule accordingly.

Excel_HR_DateOffLeaving

Files

Here the user has spotted that one file is a disciplinary document.  They have therefore changed the disciplinary date and content type accordingly.

Excel_HR_DiscDate

Preparation Complete

Once preparation is complete you should have a set of Excel spreadsheets that you can use to drive your folder creation and file migration and which will work with most good SharePoint migration tools including our own.

This should be double checked and quality controlled before you commence the migration process but the core work is done.

Preparation NOT Complete!

Struggling?  The likelihood is that the problem is either in the quality of your source data or the design of your destination and these areas will need to be revisited.

The good news is that that you have uncovered the issues at the right time – BEFORE you migration large quantities of data.

Importing the Files

There is a follow-up article here.

 

Share