Deeper Insights https://docs.microsoft.com/archive/blogs/deeperinsights/feed.xml Deep Insights across Microsoft Platforms Tue, 16 Apr 2019 05:01:30 GMT en-US hourly 1 New Day. New Role. New Possibilities https://docs.microsoft.com/archive/blogs/deeperinsights/new-day-new-role-new-possibilities Mon, 11 Jun 2018 16:47:06 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=495 New Day Things change. That is a constant in life. Sometimes they change for the better, other times we are not so lucky. The best representation of this change in nature, in my opinion, is the path of the sun on a daily basis. It rises and it falls. In between, it is in a state of constant movement through the sky. That comparison is one of the reasons I love Florida. You can watch the sunrise at a Gulf Coast beach, and watch it set at an Atlantic Coast beach, live, and in the same day.     Some days are better than others. While some start well, they might not end as nicely, and vice versa. Our careers can be the same way. While I've lived through my share of stormy days, for the most part, I've enjoyed a very "comfortable climate" in my career. As much as we may enjoy a certain day, the only constant in life is that things will change, constantly, like the sun through the sky. Recently, I've experienced a new sunrise in my career.

Old Role

I've been at Microsoft since October 2016. I had the amazing opportunity to join the the Premier Support for Developers(PremDev) consulting team. If you're not familiar, the PremDev team is the team that Bradley Ball works on as well. It's a great team of some of the most knowledgeable professionals in the Cloud, Development, and Data spaces. They work with customers to help with proactive projects and advisement on using new and emerging technologies. The last almost two years has been a great learning experience for me. Being a part of the team helped to open my eyes to the "development" side of our world. I've had the opportunity to work with some of our most interesting customers and help them take on their most interesting and perplexing work loads. But just as my analogy shows, the sun must set on every day, and each opportunity.

New Role

As the sun rises, I am now a member of the FastTrack for Azure team at Microsoft as a Senior FastTrack Engineer. We are a part of the Customer Experience team under the Azure Engineering group. We help customers move their workloads to Azure, by offering customers education and advisement. For me, it has always been a dream to work with the engineering teams (aka the Product Group) to help make our products better. Now I have that opportunity.  

New Possibilities

With a new sunrise, comes a wealth of new possibilities for the new day. In my new role, I get the opportunity to help bring customers in to the cloud, while taking the feedback from our customers and passing it back to the engineering team. Also, this role gives me the opportunity to get better at not being the "give me the keyboard" guy. We want to teach our customers "how to fish" when it comes to deploying in Azure, not "bring them the fish". Our goal is to help customers become self sustaining with Azure. My hope is that with this position I can get back on the conference trail and get back to speaking at conferences. As far as "sunrises" go, this is one of the more bright and enjoyable sunrises of my career, and I'm excited for the opportunities that the new day brings.  ]]> Twitter Sentiment Analysis Part III: What We Learned From Justice League & Star Wars https://docs.microsoft.com/archive/blogs/deeperinsights/twitter-sentiment-analysis-part-iii-what-we-learned-from-justice-league-star-wars Wed, 10 Jan 2018 23:12:32 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=475 JUSTICE LEAGUE   In the cases we have examined thus far, we noticed that sentiment taken before a movie premiers is higher than that of when it premiers.  The sentiment before the premier was 57%.  Where we went wrong was assuming this correlated with what we should expect opening weekend.  Once the movie opened it continued to drop. Thursday's positive sentiment was only 47%. Friday was 45%. Saturday was 44%. Sunday we stopped the collection half way through the day and it was at 51%. The high for the entire weekend was 51%, this was 6 points off the sentiment taken on Monday and for the weekend was only at 46%.  This was 11 percentage points off the high sentiment.  In the case of Justice League this showed in a lower box office.  Justice League was the first movie this year we tracked that did not make back its budget in Domestic Box Office gross.   STAR WARS: THE WORST SENTIMENT  Then came Star Wars.  Star Wars has made back it's budget.  The real story here is how far off the previous Star Wars movie in the same trilogy it has fallen.  The box office is not yet closed, but as a year over year average Star Wars: The Last Jedi is behind Star Wars: The Force Awakens by a staggering $242.8 million in Domestic Box Office gross.  Star Wars: The Last Jedi has the lowest positive sentiment of any movie tracked so far.     As you can see from the visual even though Star Wars has made it's money back by over $300 million, its positive sentiment was 33%.  That is staggering.   This has lead us to look at additional sources.  You'll notice the IMDB and Metascore ratings are on each report.   SO WHAT NOW New questions are awesome.  What we've decided is we need more data, we need to branch out to different movie genres and see if patterns hold up.  To do this we need to be able to collect data from Twitter historically, we wanted to add additional scores like IMDB and Metascore.  We want to see what patterns emerged. Is sentiment always lower after a movie premiers vs. the sentiment before?  What does that drop mean?  If you have a strong brand, like Star Wars, with low sentiment can you still be profitable at the box office?  If you have low sentiment, an established brand, and the movie is not profitable is that indicated in social media sentiment?   To answer these questions we will use python to scrape and collect data.  Then we will review those patterns and see what we can find.  In our next blogs we will review how we scraped the data, ingested it to SQL DB, and used Azure ML for semantic scoring. As always Thank you for stopping by.   Brad]]> Twitter Sentiment Analysis Part II: Thor Ragnarok and Justice League https://docs.microsoft.com/archive/blogs/deeperinsights/twitter-sentiment-analysis-part-ii-thor-ragnarok-and-justice-league Thu, 16 Nov 2017 12:59:58 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=446 Spider-man: Homecoming.  We even blogged a step by step process for how to re-create this experiment, see How to Measure Twitter Sentiment with Azure Logic Apps, SQL Database, and Power BI.  The movie has progressed and the next movie in the Marvel line has arrived, Thor: Ragnarok.  We saw the movie opening night and before we went we asked ourselves a question.  What would the social media sentiment be for Thor and how would it compare to Spider-man?  Would sentiment indicate a trend for opening weekend gross, the amount of money the movie made on it's opening weekend? So we re-created our experiment.  Let's look at the results below:   Thor had a very successful opening weekend.  Looking at the matchup between the two movies reveals an interesting trend.
Thor beat Spider-man in sentiment analysis by 7.61%.  Thor also beat Spider-man in opening with an opening day Domestic gross of $122.74 million, winning by $5.17 million.  If we measure this by sentiment that would mean that for every extra point of positive sentiment Thor gained an additional $750,000.
 
In the session we reviewed how we stored, and scored the data.  As an experiment with the class we then sought to develop a hypothesis: Could we measure the Twitter sentiment of a movie before it opens that is in the same type of demographic/genera and use that to estimate the Domestic gross for the box office?
 
Well the long awaited Justice League movie comes out today, and we took this opportunity to use Logic Apps to stream data into our database.  We did this for the duration of Monday.  The Justice League sentiment, for that specific window, was 56.9% positive.  Not as much as Thor.  If we use Spider-man as a baseline and estimate $750,000 * 2(the percentage difference) then that means Justice League would have an opening weekend of $118.5 million dollars.
 
This also raises the question, is the sentiment before a movie equal to the sentiment of the opening weekend?  We didn't track that with Thor or Spider-man.
 
This was some of the fun that we had in our pre-con to show how you can utilize modern data sources like social media data to try to spot and anticipate trends.  Alright Dear Reader, now go see the movie and we'll find out how close we are!  As always Thank you for reading!
 
Thanks,
 
Brad
]]> When is the Best Day to See Your Favorite Baseball Team? https://docs.microsoft.com/archive/blogs/deeperinsights/when-is-the-best-day-to-see-your-favorite-baseball-team Thu, 13 Jul 2017 13:46:07 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=25 Hello Dear Reader!  Continuing on our focus of real data, last year Josh Luedeman (@JoshLuedeman) were trying to figure out this exact problem.  How could we find real, free, and very cool data?  Enter RetroSheet.org. RetroSheet is a team of volunteers that have managed to salvage the play by play data for every game of baseball from 1922 - 2016*.  Josh and I looked at this data set as a great way to use some new tools and learn more about the teams and game we love.  We then used our data skills to ETL out all of the box scores, cleanse the data, and review the data for accuracy.  Then we decided to review a subset of data based on the modern era of baseball.  For this we selected all games played between 1990 and 2016. Now we had the data, what we needed was a question.  Eventually we settled on what was the best day of the week to see your team play.  My favorite team is the Chicago Cubs, has been my whole life.  Last year was a good year to be a Cubs fan, they broke a 108 year drought and won the World Series.  This picture is from my trip up to Wrigley Field to see Game 5 of the World Series last year.   Enough exposition let's look at the data!  This is an interactive embedded Power BI report, please pick your favorite team and look at the results!  I'll keep typing below. My predicament is that I have a family of 5 that I would want to take to a game.  Attending an in person baseball game can get expensive.  Winning and losing in sports can have a lot of different criteria.  I wanted to keep this simple.  Could I find the best day of the week to see my team play? Historic Data Analysis based on:
  • Day of the week a game is played on
  • If it is a home or away game
  • Does my team have a winning or losing record
  • What opponent are they facing
I felt this would give me a good basis to determine if it was worth it to make an expensive trip to take the family to a game.  I started out my experiment in R.  I found very quickly that a trend emerged when I looked at only the days of the week and wins and losses.  This is a plot I created in R studio. In this we can see that the day of the week the Cubs win the most on is within a grouping of Saturday, Friday, and Sunday.  Monday looks low on the list.  When  I look at the day they lose the least, that happens to be Monday.
Day of the Week Wins Losses TotalGames
Sunday 329 360 689
Monday 260 222 482
Tuesday 302 340 642
Wednesday 314 343 657
Thursday 221 238 459
Friday 336 349 685
Saturday 338 353 69
After looking at the data it is clear the most likely day for me to go see the Cubs and have them win is on Monday!  At home on Monday when the Cubs have a winning record, I interpreted that to mean a record of .500 or greater, I have a 60% chance of seeing them win.  If I'm going to take the family all the way up to Chicago to see a game, I should do it on a Monday.   There are a lot of cool things waiting to be discovered in this data.  For instance did you know that if you are a New York Yankees fan and the Yankees have a loosing record, they have a better chance of winning on a Thursday than they do if they had a winning record!  We'll talk about how we built this in a future blog post.  We used Azure Storage, Polybase, SQL Server with advanced analytics, SQL Azure DB, R, R Studio, and Power BI to develop this post.  Thank you for reading! How about you Dear Reader?  Do you have a favorite team?  Did you notice any interesting trends?  Sound off below!   Thanks,   Brad   *There are some games where data may be missing, and there may be some errors in the data set, for many Retrosheets.org is the go to place for baseball data.  This data and anything we put forward may have a +/- confidence value of 1 standard deviation.  This Sample Code is provided for the purpose of illustration only and is not intended to be used in a production environment.   Please don't use this information for anything that could cost you money.  This data is for fun, and should not be used to interpret future events or outcomes.]]> How to Measure Twitter Sentiment with Azure Logic Apps, SQL Database, and Power BI https://docs.microsoft.com/archive/blogs/deeperinsights/how-to-measure-twitter-sentiment-with-azure-logic-apps-sql-database-and-power-bi Wed, 12 Jul 2017 09:58:42 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=195 Spider-Man Homecoming, with my son Zachary.  I had been looking at Azure Logic Service and playing around with it.  I watched a great video by Jeff Hollan (@Twitter | Channel 9 Video) PM of Azure Logic Apps on the subject.  I also found a really great post about Creating a function that integrates with Azure Logic Apps. So I did just that.  I tracked, measured, stored, and visualized Spider-Man Homecoming's Twitter Sentiment.   I used several services along the way.  Initially I started out tracking everything in Azure Logic Apps, but I shifted a bit from there when I produced my final reports.  This blog will walk you through what I did. First we will create a real time data source for Power BI using Azure Logic Apps and add a parallel task to store Tweet data using in SQL Database.  Finally we will build Power BI reports off of this data.    Create a real time data source for Power BI using Azure Logic Apps First we create a Cognitive Services API.  You do this in the Azure Portal, and after creation you want to get your API key. To do this you can click on the Show Access Key's link after creating the service. Copy Key 1, you'll need that later. Here's a quick picture of what we will build.  Don't build this yet!  We will come back to it.  There are a few things we need to build first, but I like to know the big picture first. We will stat off with a When a new tweet is posted trigger.  This will require you to have a Twitter account and authenticate with the service.  You will then create a new action.  Select Text Analytics, this will require the key that we created earlier to score the sentiment.   The next two actions we will create require  our additional pre-work.  You will need to create a live data set in Power BI and create a SQL Database and a table that data will be stored in.  So before we put this together let's create the live data set and SQL table.   If you got excited and started creating a Logic app before creating these two you can save just your tweet and cognitive service API.  MAKE SURE YOU DISABLE THE LOGIC APP BEFORE LEAVING IT.  We are not ready for it to run yet.  Otherwise it will just start recording Tweets and sentiment without anywhere to store it. In Power BI to create our live data source, log into PowerBI.com and enter your workspace.  Here you will see I'm logged into my Power BI account and I'm in my Spider-Man Homecoming workspace.   Select the Streaming dataset option under Create in the upper right hand corner. The default option of API should be selected.  If not click API and click next. Now we need to create the data format that we want to store our data in.  You can select anything you want to consume, for me I just wanted to get the tweet text, location, when the tweet was created, users number of followers, and our sentiment score.  Adding these fields and setting them to the appropriate data type will allow us to create a live streaming visual on a dashboard.  It will also dynamically build the JSON required for this.  Make sure you click Historical data analysis   Click Done when you are finished.  Now Create your SQL Database and name it, then create a table.  This is the T-SQL Script I used.  If you do not know how to create a SQL Database click here for a quick tutorial. drop table if exists dbo.tweetinfo_scored go create table tweetinfo_scored(         myid int identity(1,1) primary key         ,userName nvarchar(500) null         ,TweetText nvarchar(max) null         ,description nvarchar(500) null         ,location nvarchar(500) null         ,retweetCount int null         ,TweetedBy varchar(500) null         ,SentimentScore nvarchar(20) null         ,followers int null         ,createdAt nvarchar(500) null ) Now we create our Azure Logic App. Then we will select a Blank Logic App. Now we start to build everything.  Remember you will need four things:
  1. Twitter credentials to authenticate to the service
  2. The API key for Sentiment Analysis
  3. A Power BI account
    1. *authentication to Power BI works best with the the Chrome browser, I'd recommend using that
  4. Database level user name and password for your SQL Database instance and make sure you've created your table.
First create the Twitter event.  If you want to see the wide selection of different events to collect data feel free to scroll down.   There are quite a few selections e.g.  Dynamics, Facebook, Twitter, Oracle, Salesforce, and more. We'll select the Twitter event.  Then we will need to authenticate to Twitter.  Now that we are authenticated to Twitter a token has been sent that will allow us to access the Twitter stream from this account.  Initially I set my frequency to 1 minute.  I changed that to 15 seconds later.  All of this depends on how quickly you want to collect data and process it.   Now we can click our next step and add our Detect Sentiment action.  Add the Tweet Text to determine sentiment based on the text and create our connection to the Cognitive API, remember your key from earlier this is where you will need it. Click Next step.  We'll add our Power BI action next.  We'll select the Power BI - Add rows to a dataset action. Once we select that we will need to authenticate to our  Power BI account.  After you authenticate select your workspace, mine is spiderman homecoming.  Select the name of the streaming data set we created, mine is Twit2, and finally the Table name by default this should be RealTimeData. Now we will populate the fields we had created earlier. From here I could keep all of my actions inline.  However, I would like to add a parallel branch to write to my Azure SQL Database.  On the arrow between Detect Sentiment and Add rows to a dataset click, select Add a parallel branch, select Add an action. Choose SQL Server.   Select SQL Server - Insert Row. Dear Reader you may be asking.  "Why SQL Server?" I could have used Azure Storage, Hadoop, Spark, Azure Data Lake, why SQL Server?  I would have needed to create a CSV file and write the contents to the file and deposit it in the big data source.  Later I'm going to use this data in Power BI, and I have the option of using Direct Query or loading the data using Power Query.  I can also use Azure ML to Import data from my SQL Database.   Granted I could also do this doing each, but I chose SQL Database for simplicity and how it just works with everything else I wanted to do.  Your choice of data source may change based on what you are collecting.  You have options, and that's good. Now we will need to authenticate to our SQL Server.  Once we do that we can select our table and begin entering our data.   At this point our Logic app should look like this.   Now save and run your app.  Back in the screen for the Logic App you will see if everything runs successfully.   At this point you can go into Power BI and start dragging data onto your dashboard, or go into SMS and do query to validate data is streaming into the database.   In this case I can see that I'm getting data back in SSMS and when I added a Map visualization and the sentiment score I could see where tweets were coming in, from all over the world.  Pretty cool!  Alright Dear Reader, I'm off for the next cup of coffee.  I hope you enjoyed this and as always, Thank you for stopping by.   Thanks,   Brad  ]]> Spider-Man Homecoming Twitter Sentiment https://docs.microsoft.com/archive/blogs/deeperinsights/spider-man-homecoming-twitter-sentiment Mon, 10 Jul 2017 14:47:24 GMT https://blogs.msdn.microsoft.com/deeperinsights/?p=96 built all of this?  Click here to read How to Measure Twitter Sentiment with Azure Logic Apps, SQL Database, and Power BI.     The majority of the twitter sentiment was positive!  Over 55% of the response based on the Twitter Semantic Model I used in Azure ML.
Response Count Percentage
negative 4777 17%
neutral 7668 28%
positive 15150 55%
Reviewing the text of the tweets I felt that was low, but that just means the model has room to be trained.  Looking at our Power BI Report I can see the influential Twitter users, those with the most followers, had the most re-tweets.  This has more than just social relevance, this translates to dollars quite easily.   Since the launch of the Marvel Cinematic Universe, MCU, back in 2008 the largest opening weekend for a Marvel character in their origin movie belonged to Iron Man.  This is fitting.  Iron Man and Robert Downey Jr. kick started the MCU.  They started in style with a $99 million dollar opening weekend, going on to gross $318 million dollars domestically and over half a billion world wide. The next closest movie was the Guardians of the Galaxy.  While it was not a solo-hero movie, it was still the premier for the group.  They garnered $94 million on their opening weekend.  By the end of their run they had netted over $333 million domestically and over $770 million world wide. Looking at the overall Marvel Cinematic Universe, Spider-Man Homecoming is in the mid-range for openings overall.  Spider-Man Homecoming has already earned the distinction of unseating Iron Man to capture $117 million in it's opening weekend.  If the trend continues, our friendly neighborhood Spider-Man may just end up with over $300 million. Based on the sentiment and the first weekend, Spider-Man is just getting started.  So is this a big gain on previous Spider-Man movies?  The last two Spider-Man movies averaged $232 million, the latter earning $202 million.  The average MCU movie thus far has averaged $296 million.  Spider-Man is off to a good start, we'll see where it goes from here! Tune in for later this week when we explore how to collect, store, and create these visuals easily and quickly using the cloud! Thanks for stopping by.   Thanks, Brad]]>