1. Passwords
   1. These are auto-generated on demand. While doing this we can select either 1 year or 2 years or Never expires (as shown in screenshot below).
2. Public Keys
   1. These are the manual keys created using certificates. The screenshot shows the thumbprints for the certificate that was used as a key.

This post will demonstrate how you can iterate through a large list (or library) circumventing the list view threshold restrictions and update the items.

Solution is simple: Get list items in chunks (of say 512 items) and iterate over each item and call into a child Flow (nested flow concept, remember?). This child flow will take care of updating the list item. The Flow that we will create will introduce you to following concepts:

1. Nested flows
2. Using Microsoft Cognitive API – Language detection
3. Parsing JSON
4. Data operations/manipulations

Which column are we going to update?

If you remember from my blog post part 1, we created a large list from tweets from Twitter. I have screenshot below, you can see that there is a column called LangDetected which is empty. We are going to pass the Tweet text (Title column) to a Cognitive API to detect the language and then update this column with the detected language.

Detailed Steps:

1. We will begin with the child flow that we created (steps 7.1 to 7.4) in Part 2 of this blog series. For your reference here is the screenshot of this child flow in its current state:

   

2. Add Apply to each step next and pass the output from Get items step to it as input.
3. Action: Search for Detect Language, Select 'Text Analytics – Detect Language'. I configured the Cognitive API Text Analytics already in Part 1. Configure it by selecting the Title column as its input.

   

4. Action: Search for "Parse JSON", Select Data operation – Parse JSON. Pass the output from Detect Language step to this step. In the schema, past the following:

   {

   "type": "array",

   "items": {

   "type": "object",

   "properties": {

   "name": {

   "type": "string"

   },

   "iso6391Name": {

   "type": "string"

   },

   "score": {

   "type": "number"

   }

   },

   "required": [

   "name",

   "iso6391Name",

   "score"

   ]

   }

   }

   Tip: To create schema, you can pick sample output from DetectLanguage step and use it in jsonschema.net website to generate the schema.

5. Action: Search for Compose > Select Data Operations – Compose. We will use expression first(body('Parse_JSON'))?['name'] to get the first language detected from the Parse JSON step.

   

6. We will create a separate flow that will be called within this flow. This separate flow will take the language detected and item index as input parameters and update the corresponding list item.
7. Create a Child Flow by starting from blank Flow template.
   1. Trigger: Search for Request, Select 'Request/Response – Request' trigger. This will create a URL for this flow that we can call into from the parent flow. This is similar to HTTP Triggered Functions in Azure. We want to send the item index and language detected to this flow as JSON data. Configure this trigger to accept ID (item id) and Lang (Detected language) using below JSON schema.

      {

      "type": "object",

      "properties": {

      "ID": {

      "type": "integer"

      },

      "Lang": {

      "type": "string"

      }

      }

      }

       

   1. Action: Search for Update Item > Select SharePoint - Update item action. Configure it by passing the ID and Lang as follows:

      

   2. Here is the full screenshot of this child flow

      

       

   1. Expand the trigget 'Request' action and copy the HTTP POST URL

      

    

8. Continuing editing parent flow from step 6. We got the Language detected in step 6. We need to now call the child flow and pass the item index and language detected as JSON parameter.
   1. Action: Search for HTTP, Select HTTP – HTTP action. Configure as follows
      1. Method: POST
      2. Uri: Paste the URL copied from stpe 7.d
      3. Body: as shown in this image

      

9. That completes the parent flow. Here is the full screenshot of this flow.

   

10. After running these flows (from part 2 and 3) I was able to update the Language column for every tweet for all the 5500+ items. Here is the screenshot of the large list. Notice the LangDetected column (last column)

    

1. Using the Content Search in Security & Compliance center
2. Using the eDiscovery feature again in Security & Compliance center

1. If you want to use Global Admin account or Security & Compliance admin account then you can start with step 3. If you want to assign someone permissions to allow them to run this then follow step 2.
2. Office 365 Global Admin must create or assign the right permissions for a user to be able to export crawl logs.
   1. The minimal permissions I could figure out after playing with permissions is as follows:
      1. Export
      2. Preview
      3. Compliance Search
   2. Logged in as Global admin or Security & Compliance center admin, Navigate to Security & Compliance center > Permissions page.
   3. Click the + button. In the pop up window, give it an appropriate name (I named mine as 'ContentSearchUser').
   4. Within the popup window, click the + button under Roles. This opens another popup window.
   5. In this window, select the 3 roles: Export, Preview and Compliance Search. Click the 'Add' button and then the 'Save' button.
      
   6. Click the + button under Members. Search for the user(s) or security group, select it, click Add button, then Save button.
      
   7. These members added in above step can access the Security & Compliance center by navigating to this URL: https://protection.office.com and follow from step 3 onwards.
3. Navigate to Security & Compliance center > Search & Investigation > Content Search page
4. Click the + button to create a new Content Search request.
   1. Give it an appropriate name and add the SharePoint Online Site URL using the + button in the bottom section. In my case I added one SP site URL. You can add one or more.
      
   2. Click Next button to see next screen.
      
   3. Click 'Search' button.
5. After some wait you will see the options to Preview search results, Update search results and 'Start export' link. Click that link.
   
6. Start export link opens a pop window. Select 2^nd or 3^rd radio button and click 'Start Export' button.
   
7. Once export is finished, you will see the 'Download exported results' link activated.
   
8. Click 'Download exported results' link.
   1. If this is the first time this functionality being used on your computer, you will see prompt like the one below to download 'Microsoft Office 365 eDiscovery Export Tool'.

   

   1. One this is installed, if it doesn't prompt you automatically, click the download result link again. The prompt will let you download results and copy an export key. Shown below.

1. When you click on the 'Download results' link, you will see another prompt to enter the Export key and a folder on your PC to save the files:
   
2. In the downloaded results, you will see a bunch of files. Look for this file: Unindexed Items.csv. This file contains the list of items that failed to index.
3. Below is sample data that I identified in my test site:

• Error Code
• Error Type
• Error Message
• Last Crawl Attempt (Date)
• Source (URL)
• Source Name

This is the part 2 of the 3 part series related to handling large lists with Microsoft Flow – a no-code solution.

This post will demonstrate how you can iterate through a large list (or library) circumventing the list view threshold restrictions.

Once you have the list iterator in Flow, you can use and reuse for man use cases, such as updating a List column data for all the items or copying list items to a different list or identifying items based on a certain criterion, so on so forth.

Solution is moderately complex: Get list items in chunks (of say 512 items) and iterate over each item to read the items. The Flow that we will create will introduce you to following concepts:

• Using Variables
• Using Workflow definition language (WDL)
• Using Nested Flows
• Using Arrays
• Using Data Operations (like Compose)

What's the need for this post? The main reason I even thought about blogging this was because the 'SharePoint – Get Items' action used to support only 512 items i.e. it could fetch only 512 items at a time. But things have changed since then, it now supports large items (>1000) too. I tested with a large list to fetch 5000+ items and sure enough it failed with below error:

{ "status": 502, "message": "The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.\r\nclientRequestId: bac7333a-f73b-4338-a431-ca88c70fd8a0", "source": "https://mytenant.sharepoint.com/teams/corpcomm//_api/SP.APIHubConnector.GetListItems(listName='c23b10c5-c190-45f2-bc9b-aabe72f87d08',queryOptions=@q)?@q='%2524top%3d5100'", "errors": [ "-2147024860", "Microsoft.SharePoint.SPQueryThrottledException" ] }

This means, this blog post is still relevant, because the only way to get around this is to get the large list items in smaller chunks.

Detailed Implementation Steps:

1. Trigger: We can either use a Schedule or a HTTP trigger as the first step. I chose Schedule and configured it to once per day.
2. Action: Search for 'Variable' > select 'Variable – Initialize Variable' step. Configure it as follows. This variable will be used to identify how many times we need to iterate to get all the list items in chunks of say 512 items.

   

3. Action: 'SharePoint – Get Items'. Get the total item count using the ID desc , top count 1 settings.

   

4. Add 'Apply for each' and pass output from step 3 as input.
   1. Action: Within Apply for each > Search for 'Compose' > Select 'Data Operations – Compose'. Compose is a powerful action; it can be used to evaluate expressions (Excel type formulas from Workflow Definition language), create variables etc. We will use compose to calculate the number of times we require to iterate in chunks of say 512 items to process all the items. We simply divide (using Div() expression) the total item count by 512. By default, the Div() expression always returns the floor value for the result. For example, Div(925, 512) will give you 1 (floor value), not 2 (ceiling value). This is important to note and has a downside that's fixed in step 4.c. Here is the expression that you need put in the compose: "@div(item()?['ID'],512)" if you are going to type it. Flow designer now supports Expression in the pop up shown in the below image.

   

   1. Add a Condition after 'Compose' action. The condition checks if the output from step 5 is greater than 1.
   2. IF YES,
      1. Add another 'Compose' action with this formula: add(outputs('Compose'),1). This basically increments the value that we got in step 4.a by one. This ensures that we cover a scenario where there is are residual items that's not covered by the iterations because step 4.a gives floor value but not ceiling value. E.g lets say total items count = 925. Step 4.a gives results as 1. But if we iterate just once, we get back the first 512 items only. To get around this we iterate one more time to get the residual items.
      2. Add action > search for variable > select Set Variable action. Set the IterationCount variable to the output from the above step.
   3. IF NO, do nothing

   

5. Action: Add 'Data Operations – Compose'. We use the expression range() to create an array. The expression to be used is: "@range(1,variables('IterationCount'))". We will use this array as input to a for-each loop and iterate that many times. The array is like this: (1,2,3,4,5) if iterationcount = 5.

   

6. Add Apply to each step (for-each loop). Pass the output (array) from step 5 as its input. This will loop n times where n is the iteration count that we calculated already.
   1. Action: Add 'Data Operations – Compose'. This will apply an expression to identify the starting index (ID) of the items to fetch the next 512 items. So, for 1st iteration this value should begin with 0, 2^nd iteration it should be 512, 3^rd iteration should be 1024 so on. I used the expression: "@mul(sub(item(),1),512)". This uses subtract and multiply functions. Here item() signifies the current loop index (from the array) that we supplied as input to the for-each step. For 1^st iteration, mul(sub(1,1), 512) returns 0.

   

   1. Now introducing the concept of Nested Flows. We will create a separate flow that will be called within this flow. This separate flow will take the item index as parameter and fetch the next 512 items starting from that index. This is very similar to writing code, that has methods/functions/subroutines. This child flow will be a method that parent flow will call into. Now let's create the child flow.

       

7. Create Child Flow by starting from blank Flow template.
   1. Trigger: Search for Request, Select 'Request/Response – Request' trigger. This will create a URL for this flow that we can call into from the parent flow. This is similar to HTTP Triggered Functions in Azure. We want to send the item index to this flow as JSON data. I added the JSON schema for an ItemIndex variable of type integer. See below:

      

      {

      "type": "object",

      "properties": {

      "ItemIndex": {

      "type": "integer"

      }

      }

      }

   2. Action: Add SharePoint – Get Items action. Now configure to fetch the next 512 items starting with the index number that's passed as a JSON parameter to this flow. As shown below, we use the Filter Query to filter by ID greater than or equal to ItemIndex.

      

   3. Here is the full screenshot of this flow:

      

   4. Expand the 'Request' action and copy the HTTP POST URL

      

8. Continuing editing parent flow from step 6. We got the itemindex figured out in step 6. We need to now call the child flow and pass the itemindex as JSON parameter.
   1. Action: Search for HTTP, Select HTTP – HTTP action. Configure as follows
      1. Method: POST
      2. Uri: Paste the URL copied from stpe 7.d
      3. Body:

         {

         "ItemIndex": "@{outputs('Compose_3')}"

         }

      

   2. The for-each loop created in Step 6, will not call the child flow n number of times, where n is the iterationcount that we calculated.
   3. That's the end of the parent flow. Here is the full screenshot with minimized actions/steps

      

In Part 3, I will show you how to update the list items from within the child flow, creating grandchild flow and also show you how to use Cognitive API to detect tweet text language.

UPDATE (8/15/17)

Just a week before I published this article there was an announcement that allows "pagination" using the Flow actions like SharePoint – Get Items. Essentially that solves the problem that I went through documenting in this article. I set out to test this new feature and created a Flow to get more than 5000 items from my list using pagination. I found some limitations which makes this blog post still a valid one. Yay! More details below.

1. Started from a blank flow. Added Schedule as my trigger. Configured it to run once per day. This doesn't matter. This the quickest way to create a flow with least amount of work/settings.
2. Action: Added SharePoint – Get Items. Provided my SPO site url, large list and set Top Count to 10. See screen below.

   

3. Now, click on the ellipses i.e the three dots (…) in the top right corner for the Get items action and select settings.

   

4. In the screen opened you will see a way to turn on pagination and set max items to fetch.

   

5. I turned on the Pagination, set the Limit to 50. So, the Get Items action will fetch 10 items (since we set Top count to 10) each up to 50. Essentially it iterated 5 times getting 10 items each.
6. I ran the flow and voila within a few seconds, it's done and it got back 50 items as expected.
7. Now, I changed the max Limit from 50 to 5100, to test the limits. Also, changed the top count to 1000, so that it paginates 5 times fetching 1000 items each. Saved/updated the flow. That's when I got this error message in the top:

   The pagination policy of workflow run action 'Get_items' of type 'ApiConnection' at line '1' and column '751' is not valid. The value specified for property 'minimumItemsCount' exceeds the maximum allowed. Actual: '5100'. Maximum: '5000'.

   

8. That's it, it still has the 5000 items limit. Pagination is a great feature, but we still have to go through a manual steps using flow to iterate more than 5000 items as shown in this blog post.

This post will demonstrate how you can create large lists with test data.

There could be many reasons to create a large list. In my case, I had to demonstrate how PowerApps can deal with large lists and I needed a large list (with >5000 items). I could create a Microsoft Flow to do exactly that. Now my Flow can be saved as a template and reused by others too. This makes Flow even more powerful.

Solution is very simple: Monitor twitter feed and get the tweet data into a SharePoint List.

I added something extra to the mix. I also used Microsoft Cognitive API (specifically Text Analytics API) to detect the sentiment (positive or negative tweet) of the twitter feed and extract some good metadata out of it which will be saved into List columns.

I also added the logic to insert items only if the total items count is < 5500 items. This effectively stops populating the list forever. This will save your Flow execution cycles and also your Azure spend on the Text analytics api.

Below are screens of the list after populating it with >5000 items.

Screen 1:

Screen 2:

Screen 1: Shows the items sort ascending on the ID column.

Screen 2: Shows the items sort descending on the ID Column. You can see that my list has >5000 items.

Detailed Steps:

1. Create the SharePoint list (I named it MyCorp Tweets). Add columns:
   1. TwitterHandle (Text)
   2. RetweetCount (Number)
   3. FollowersCount (Number)
   4. SentimentScore (Number, two decimal with %)
   5. LangDetected (Text)
2. Begin by selecting the Tigger "when a new tweet is posted". You could start from existing template or from blank.

   

3. I configure it to search for certain keywords that's popular: SharePoint OR Azure

   

4. Add an action as the next step and select SharePoint – Get Items and configure it to retrieve only a single item with the max ID. I used Order By and Top count parameter to achieve this.

   

5. Next Add an Apply to Each action. Pass the output from the SharePoint – Get items to this step as input.
6. Now Add a Condition action. The condition checks if the ID (gotten from SharePoint – Get items) is <= 5500. Below screen shows it in action:

   

7. Let's shift gears and go to Azure Portal to create Cognitive Services – Text Analytics Service. In Azure Portal, clicke +New button, search for "Cognitive Services". Select it and click Create button. Fill up the details as follows to create a new Text Analytics service.

   

8. Once the Text Analytics service is created, select it and then select 'Keys' that is under the "Resource Management" within the settings blade. Copy one of the keys to clipboard.

   

9. Now, lets get back to Flow designer. If the above condition (step 7) is satisfied (i.e IF YES branch), lets pass the tweet text to the Text Analytics – Detect Sentiment action. To do this, add an action in the IF YES branch and search for "Text Analtyics". Select Text Analytics – Detect Sentiment. Give it a unique connection name and paste the key into the Account key text field.

   

10. Next, configure it by passing the Tweet text as its input.

    

11. Now add new action SharePoint – Create Item. Configure it as follows:

    

12. Note that I am using the 'Score' value which is the output from the Text Analytics – Detect Sentiment step.
13. Overall, your flow should look something like below.

    

14. I have not added any action to the 'IF NO' branch for the IF condition. I will leave that as an exercise for you. You can add an Email action to alert you that the list has enough items.

In the Part 2 of this series I will show you how to Iterate and get all items from a Large List/Library using Microsoft Flow (no-code solution). When I worked on this demo, Flow could get a max 512 items only. So the solution to iterate a large list is quite interesting.

Preview

General Availability

How does image renditions work in combination with the Office 365 CDN:

• Image rendition support for images from the CDN can be enabled with height and width query parameters in the URL
  • Note. both height and width parameters are required for the renditions to work
  • You can also control the cropMode for the images with cropMode query parameter with following options: fit, fill, top and bottom (fit is default value).
• You can use renditions with query parameters for the image URL

1. Finalize the metadata (columns) for the documents. Plan beyond the already identified metadata in your CSV files. This post is based on assumption that you already have the metadata identified and its in a CSV file. Try to think ahead and add additional metadata columns, even if those are empty/placeholder column types.
2. In SharePoint site, create all the site columns corresponding to the metadata columns identified in Step 1.
3. Create Content Type. Add all the site columns created in Step 2 to the content type.
4. Create a single or multiple SP document libraries depending on the approach that’s decided. Associate the Content Type to the Document Library(ies).
5. Go the library settings, and add indexes for all of the Site Columns, that will be used to filter or sort the data by users. These columns will be used in Step 6 for creating views. Note that you can have max up to 20 indices in a list/library.
6. Plan and create as many Views as possible for the library. Make sure that the Filter or Sort conditions used for Views are set as Indexed columns. This is to avoid the 5000 item threshold limit. Also, limit the number of items shown in the View using the Item Limit property. If you are using nested folders, then you can leverage the ‘flat view’ property Show all items without folders while creating the views.
7. Another great and useful feature that you can leverage is the Metadata Navigation feature. You must first enable it by navigating to Site Settings – Manage Site Features and activate ‘Metadata Navigation and Filtering’ feature. Once enabled, it will add Key Filters panel to the left navigation whenever a user navigates to this library. It automatically manages indexes too. Note: Metadata navigation feature is not supported in Modern library experience. Once this feature is enabled, the library will automatically revert to Classic UI.
8. You might need additional prep work if you decide to use Search to let users find documents. See the last section in this email.

1. Although drag and drop works great, it’s not feasible for a large number of documents.
2. Use OneDrive for Business (ODB) Next Gen sync client to sync the library to a local machine and then copy all the files to the local machine ODB folder. Then ODB sync client will take care of syncing/uploading those files to the SP library.
3. Use Content Organizer feature – that will create a drop-off library, upload docs to this library and the documents will be routed to folders automatically. These folders are created automatically for every 5000 items.
4. Use PowerShell/CSOM to upload the documents, read the metadata from CSV and add metadata to the columns.
5. Steps 1 to 3, will require additional step to read the CSV and add the metadata to the columns

1. Let users leverage the Views that’s created.
2. Let users use the Metadata navigation that can be helpful to filter and find data
3. Let search index the documents and use search center or provide custom page that is powered by search API. If planning to depend on search, you may need additional work:
   1. Ensure all the crawled properties are mapped to managed properties. If new managed properties are created, then you must re-index the library.
   2. It’s recommended to setup these mappings before uploading bulk of these documents.

Install-Module SharePointPnPPowerShellOnline

A command with name 'Get-SPOSite' is already available on this system. To override this cmdlet use -AllowClobber.

import-module : The specified module ‘OfficeDevPnP.PowerShell.Commands’ was not loaded because no valid module file was found in any module

 Import-Module C:\Users\<UserId>\AppData\Local\Apps\SharePointPnPPowerShellOnline\Modules\SharePo
 intPnPPowerShellOnline

Get-Help Microsoft.Online.SharePoint.PowerShell\Get-SPOSite

Get-Help SharePointPnPPowerShellOnline\Get-SPOSite

Microsoft.Online.SharePoint.PowerShell\Get-SPOSite

SharePointPnPPowerShellOnline\Get-SPOSite

Import-Module SharePointPnPPowerShellOnline -Prefix "pnp"

Get-PnpSPOSite

Get-PnpSPOWeb and so on.