Awareness – Part 3: Learning & Optimizing from Experience

In my last 2 posts on Information Security Awareness, I provided a little overview of the program and then discussed our framework around socializing security. I’d like to now discuss some of the things we’ve learned from driving awareness over the years and how we’re looking to optimize our awareness programs.

There are 5 key learning areas that we’ve looked to optimize in Awareness:

1. Simple, Precise & Actionable Messaging

This is one of those things that I feel is true not only for security but for any point that you’re trying to make in any context. There are always a lot of things going on in any large enterprise like Microsoft and to ensure people pick up on your message, we have to keep the message simple, precise and actionable. Doing campaigns around large nebulous concepts like “secure your application” isn’t always as effective as driving the message of “Use AntiXSS to protect against Cross-Site Scripting attacks”. The former is a large domain with lots of controls that can be thrown in and the latter is a very precise and actionable message. Granted that you may have to create a lot of simple and precise messages to cover the complete concept of secure applications, but in the long run, it turned out a lot more effective for us.

2. Awareness Activities: Marketing, Evangelizing or Educating?

One of the things we realized over the course of driving awareness programs and campaigns is that there are really 3 different types of activities that we were undertaking. We were either trying to:

1. Market something
2. Evangelize something
3. Educate people on a specific topic (which was more often the case)

The need for these 3 different types of approaches was born out of our understanding of driving simple, precise and actionable messages. When we started reducing our messages down, we saw that we either needed to market, evangelize or educate.

There are times when you just want to drive some mini-marketing around a shiny new tool you’ve released like CAT.NET. And then there are times when you want to evangelize a new policy, an upcoming tool like TAM 3.0 (which is now released by the way) or the importance of design reviews. And then of course there is the most common case where you are trying to provide prescriptive education to the end user, like defending against specific SQL injection attacks.

In each of the 3 cases, you are driving awareness of something. It turns out that it was helpful for us to understand the type of activity we wanted to drive in order to ensure we are producing the most optimal resource type.

3. Persistence & Innovation

Security awareness really does come down to changing behavior and changing behavior requires a lot of persistence. We would drive campaigns to promote certain security activity and you would see a spike in that activity but over time it would phase out. We learned the need to not only persist our messaging but also to keep it fresh and innovative. If we just recycle old messages and campaigns, it doesn’t land as effectively as freshening up the delivery a bit with perhaps new incentives or new mediums of delivery.

4. Integrating into the Environment

Clearly we’re in an incredibly fast paced environment and it’s getting increasingly difficult to get someone’s attention. So rather than trying to capture people’s attention so we can tell them about a new security practice or tool or some piece of education, it helped us to identify outlets where individuals are gathering (in person or virtually). We would then seed our message into these outlets. This approach of taking the message to the people rather than forcing the individuals to come to our message turned out very effective in reaching a larger audience with lesser effort (not necessarily a new concept as it’s very much marketing 101 but nonetheless a good learning for us early on in Awareness). Some examples include:

• Getting important security messages into team all-hands meetings
• Publishing best practices or links to security tools into various team or discipline focused publications/newsletters
• Having booths at corporate events

One important lesson for us in seeding content was to ensure that we are contextualizing the message. For example, it would do little good to seed the importance of secure development practices into an all-hands meeting for call center teams that deal with taking customer’s credit cards and have virtually no developers.

5. Measuring Success

This is perhaps one of the most important learning that we continue to experience: just how exactly do you measure the precise success of an awareness initiative (or information security initiatives in general!). If awareness is about behaviors changes and most people agree that behavior changes can take long time to achieve, how do you show incremental results from your campaign? This is where we stuck to basics and saw that it’s important to measure our engagement level and our effectiveness.

The engagement level came down to understanding how many individuals we are able to engage with our programs and campaigns and how much it cost us. The engagement type could be passive or active. For instance, passive engagement could entail someone just visiting a web page that talks about certain security controls or best practices. An active engagement, on the other hand, could be when someone decides to participate in a training or survey on that web page. On top of engagement type, we measure cost per engagement which gives us a nice benchmark to understand the ROI from a budgetary perspective. If it cost us $1,000 to reach 100 individuals, how can we improve that further. This exercise, more than anything else, was vital in understanding the high-investment-low-return activities in our campaigns and potentially cutting them out. An example is print. Typically the most costly element of a traditional awareness campaign is posters and other print material which costs quite a bit to produce only to return minimal results and end up at the bottom of the recycle bin.

The effectiveness was important for us to understand whether what we are trying to message is relevant to the target audience. This was a bit trickier to measure but we again stuck to a simple approach. We would drive things like “spot-the-bug” contests to understand if certain classes of bugs are still not understood well by a certain developer population or seed surveys with questions to the likes of “did you know about X prior to this campaign? ”.

As simple as these approaches may sound, not only do they help us reaffirm whether the message we’re driving is relevant but they also give us data points that we can then track over months or quarters or years to trend how effective is our specific program or campaign while simultaneously optimizing them for impact.

Much like our overall information security program, there is definitely room for improvement and we continue to learn from our experiences. Yet I’m very proud of the progress we’re making on different fronts like Awareness as just one of our overall effort in driving a holistic people, process & technology solutions for information security.

As always, would love to continue to hear your thoughts.

-Todd