Online Video Analytics - KPIs

posted by Dennis R. Mortensen
Friday, February 15, 2008
Bookmark: Online Video Analytics - KPIs

We (and that means every online marketer) have seen an extreme interest in Online Video (streaming video) because it simplifies the process of getting video to the consumer and with streaming video, unlike downloaded video, the content (the video and any interaction with it) can be tracked. AND if any business, that be B2C or B2B invest in online video – they are obligated to measure the return of that investment!

First I would like to apologize for the aggressive title - as this is more about essential Online Video Metrics than KPIs – I think it is fair to say that these online video metrics affect your KPIs, but are not KPIs by themselves, unless of course if you are in the business of delivering video as a revenue model. :-)


9 Essential Online Video Metrics
  • Online video started
  • Online video Pre-roll advertisement started*
  • Online video core content started
  • Online video Post-roll advertisement started*

  • Online video positive consumption action
  • Online video negative consumption action

  • Online video ended
  • Online video played, percentage of total
  • Online video played, seconds
*pre- and post-roll metrics are of course only of importance if and when a online video monetization model is in place. For some of us these metrics does not exist.


Essential metrics suggest that there is indeed more exotic metrics -- and you are right -- but I stoutly suggest getting the essential metrics up and running, as in being able to collect, report and analyze on them before working towards any of those more exotic ones.

Furthermore, surrounding metrics such as “send to a friend”, “rate this video” and other Online Social Networking activities are separate metrics that are important, but not distinctive online video metrics.

Consumption actions can be pooled together (dependant on your users) in positives and negatives; such as e.g. fast forward, slide forward and stop as negatives and rewind, slide backward, pause and play and positives. You have to be very careful about using these as it can be difficult to derive valid insight; AND you might want to split them into different actions, however, I am still personally a fan of pooling them together.

All that said; we have to differ between (from a technical tracking point of view) between two ways of serving a video online.


Simplified Online Video serving methodologies
  • Server Streaming (e.g. played through windows media player)
  • HTTP Streaming (e.g. played through your own Flash player)

Both of them (and their sisters) require completely different technical tracking methodologies and I actually posted a Web Analytics HACK on how to track server streamed online video not that long ago.

And before you fall for any bullshit, patent pending, we are the only online video analytics provider - marketing spiel, I would like to conclude that you can collect and analyze most of the above mentioned Essential Online Video Metrics in all of the enterprise analytics packages out there. :-)


Online Video Optimization comments

The above metrics are to be collected on a per video basis and you can aggregate per visit (session), per visitor or any other specific grouping (segment) you want to analyze. It is of course a given that when collecting any of the Essential Online Video Metrics, such as Online video Started, you MUST collect information about which unique piece of content (online video title or other identifiable content ID) we are dealing with. That said, here is a couple of suggestions on how we can work with the metrics:
  • Conversion rate segmented by Online video started COMPARED to the site average
  • Conversion rate segmented by Online video ended COMPARED to the site average
Where the above two suggestions immediately shows the success of using video for conversion purpose. Imaging a positive scenario where you have 1% site average conversion, 2% on those who started a video and 3% on those who finished it (expecting that the more a visitor consume the more engaged and interested he is).
  • Compare the “Conversion rate segmented by Online video started” between visitors who had a positive consumption action and those who had a negative consumption action.
Refining a positive attitude towards online video as an element for conversion increase – hereby getting further insight on the importance on questions like; does it matter if they watch the video to the end (or do they “just” have to engage positively with it).
  • Trend the metric “Online video played, seconds” grouped by videos
Creating insight on WHERE people fall of; in the first 5 sec. Or in the first 30 sec? – essentially getting to know how long you can captivate your specific audience.
  • Create traffic source goals and alerts for Pre-roll advertisement started and Online video Post-roll advertisement started
This of course creating instant insight into which traffic sources send you online video engaged visitors. And not only as a comment to the point above; whenever you look at “Video view abandonment” (did NOT watch the video to the end), it is critical to distinguish between whether we talk about complete site abandonment or just that specific video.


But in conclusion; this post is more about creating awareness about the Essential Online Video Metrics and then making sure that you have a Web Analytics system available where you can utilize these metrics in retrospective – becoming able to answer marketing questions in regards to Online Video.

Cheers
Dennis

Labels: , , ,

The difference between a KPI and a Metric

posted by Dennis R. Mortensen
Tuesday, February 5, 2008
Bookmark: The difference between a KPI and a Metric

Summary:
How does one decide if a Metric qualifies as a Key Performance Indicator? and if so, what are the characteristics of an excellent online marketing KPI?

I am actually an extraordinarily positive advocate of creating and using KPIs and I truly believe that one has to understand that using KPI and Metric interchangeable is not only wrong, but it can significantly damage your optimization initiatives!

Remember: a KPI is a Metric but a Metric is not necessarily a KPI!

Without real KPIs an organisation will not perform to its maximum and this is in particularly true for businesses where online performance is a vehicle for success. And I think it is noteworthy to remind us all that It is rarely about financial metrics - They represent the results of driving a good KPI.

Essentially concluding that KPIs are a communication and steering vehicle for management.

I will present the following 7 KPI characteristics at the RIMC conference in Reykjavik later this week.


7 KPI characteristics

1.
a KPI echoes organizational goals

2.
a KPI is decided by management

3.
a KPI provides context

4.
a KPI creates meaning on all organizational levels

5.
a KPI is based on legitimate data

6.
a KPI is easy to understand

7.
a KPI leads to action!


You are more than welcome to challenge me on any of the above, add to the list or send me free diet cokes as a praise of their brilliance :-)
Find a presentation on the above KPI characteristics below:



Other great comments to the KPI debate from my fellow Analytics bloggers:

Cheers.. Off to Iceland! :-)

Labels: , , , , ,

What and how to measure Online Finance websites

posted by Dennis R. Mortensen
Saturday, December 15, 2007
Bookmark: What and how to measure Online Finance websites

In today’s competitive marketplace, the Finance and Banking industry is bound to establish a strong online presence and to conduct marketing activities that will enhance customer experience and support lead generation and cross-selling activities.

But as always - there MUST be a set of measurable KPI’s that are strongly aligned to ones business objectives, like any other organisation that operates with the web being a channel of theirs. Hereby concluding that KPI’s are meant to be acted on and not just reported on. Therefore I suggest optimizing towards the following two business objectives.


Online Finance business objectives:
  • Customer Acquisition
  • Reduce Interactions Cost

Success in customer acquisition can be calculated by considering the number of accounts applied for by new customers. In addition, existing registered customers may add new account types to their profiles – perhaps as the result of a specific campaign. The number of loan / credit card / account application forms downloaded (if this is the means of registration as opposed to a fully online application) will also indicate customer interest. Loan / credit card pre-approvals can also be a means to evaluate the success of a campaign or a site content update. Finally, interest in products and services can be gauged by the number of visitors searching for agent information or bank branch locations

For tactical decision-making, you need to break these strategic indicators down into manageable segments and analyze their performance by - for example - geography or property, in order to understand how well each is contributing to the overall result; or, assess them by traffic sources and marketing channels to see where you should direct your marketing budget.

But, let me outline a set of leading KPIs which will enable us to gauge the effectiveness of online initiatives and help us steer a Online Finance website in the right direction:


Online Finance Customer Acquisition KPI’s:
  • Account sign-up
  • Addition of new accounts
  • Application downloads
  • Pre-approvals
  • Locate an agent

Online Finance Interaction Cost KPI’s:
  • Average cost per interaction
  • Self-service visits
  • Response (Emails, Calls etc.)
  • Web % of customer interactions

Making the above objectives the core of your Web Analytics strategy and paying attention to these KPIs will give you an advanced level of insight and trigger the most effective changes to your web property.

You can find a complete and in-depth PDF version of our inputs for Best Practices for Online Finance industry here:




DOWNLOAD GUIDE
(4 page commercial version 617K PDF)




N.B.
This is the fourth post in a series on “What and how to..” measure a specific industry. You might find the previous posts interesting:

- What and how to measure Online Travel and Hospitality websites
- What and how to measure Social Networking websites
- What and how to measure Media and Content websites

Labels: , , ,

Customer Lifetime Value – KPI

posted by Dennis R. Mortensen
Thursday, October 25, 2007
Bookmark: Customer Lifetime Value – KPI

This is going to be the first - of a couple of posts - about this metric. Lately, we have seen that Enterprise Web Analytics vendors have put some thought into a Visitor Lifetime Value metric (KPI) – I would like to add the following comments:

First and foremost, I think we should use a stronger description (skipping the traditional Analytics Vendor description of Visitor) and call it: Customer Lifetime Value (which is also the more general and common marketing term) – Secondly, I think one have to accept that a Customer Lifetime Value metric (without even debating how we are supposed to calculate it) directly referenced to e.g. a campaign is far from telling anything about either ROI (we must expect multiple campaigns attribute to a sale/s) or even whether we reached a high value customer segment (as the metric does not recognize whether a customer is frankly lost, have not bought anything in years or otherwise not represented rightfully).

That said; Introducing a Customer Lifetime Value metric, I suggest that the following accompanying metrics be introduced as well:


Customer Lifetime Metrics (KPI's)
  • Customer Lifetime Value (definition: Aggregated revenue SUM)
  • Customer Lifetime Acquisition Cost (definition: aggregated campaign costs SUM
  • Customer Recency (definition: Time since last sale)

Furthermore, given the strong filter and segmentation opportunities in Web Analytics solutions (that compared to e.g. more traditional CRM systems) - defining “Lifetime” (usually a very difficult task) as ALL tracked data so far, seems quite appropriate to me. We can then use this as a proxy for the absolute customer lifetime value (and we probably do not have the time to wait it out OR even the history of enough customer life cycles to have an educated idea anyway).

Then again; coming back to the ever returning dialogue about “attributing” a given sale to e.g. the Original campaign, The Direct Campaign or perhaps something even more glamorous or Intelligent. Does this matter and is this really a concern? – should we not “just” present the aggregated SUM of all Customer Lifetime Values of those visitors who reacted to a campaign. As in, if 19000 visitors reacted to a campaign (as per a campaign summary) then a SUM of all the visitors Customer Lifetime Value is presented alongside the other campaign metrics. The period chosen (e.g. February) is used to present active campaigns and their visitors in the period, but will otherwise not have any implication on the SUM calculation of all the visitors Customer Lifetime Values.

Then more importantly; one can filter/ segment on Customer Recency (dependant on business-model) to create a better understanding of what kind of customers (and their value) we are looking at, thus more an indicator on whether you are talking to the intended audience (as in; is this your typical repeat buyer, the lost customer coming back etc.). AND if one want to generate a true (or probably just better) ROI calculation; a Customer Lifetime Acquisition Cost metric can be applied.

This is a good beginning – leading me into the second post about the Customer Lifetime subject. How one should consider using the described Customer Lifetime Value metric (KPI) as part of ones SEM optimization efforts.

Labels: , , , ,

What and how to measure Online Travel and Hospitality websites

posted by Dennis R. Mortensen
Saturday, October 13, 2007
Bookmark: What and how to measure Online Travel and Hospitality websites

In today’ multi-channel marketing world, the Web is quickly becoming not only the first “point of contact” with an overwhelming majority of travel and hospitality customers but also a key tool in achieving incremental growth and competitive advantage. What’s more, an effective Online Marketing strategy in travel and hospitality plays a crucial role in establishing interactive relationships with customers and utilizes all channels available on the Internet.

To gain advantage in this highly competitive industry and win customer loyalty, travel and hospitality website owners need insight into:



  • performance of their online business across customer segments, markets, properties, packages, routes and destinations
  • effectiveness of their online marketing initiatives by traffic source, online channel, advertising partner or individual campaign
  • success of their e-CRM efforts in terms of customer satisfaction, retention and loyalty
But as always - there MUST be a set of measurable KPI’s that are strongly aligned to ones business objectives, like any other organisation that operates with the web being a channel of theirs. Hereby concluding that KPI’s are meant to be acted on and not just reported on. Therefore I suggest optimizing towards the following two business objectives.


Online Travel and Hospitality business objectives:

  • Revenue
  • Profit margin growth

These are determined by a number of indicators, the most important being:

- Volume of bookings (total number of bookings)
- Bookings (sales) conversion rate (number of bookings/number of visitors to site)
- Average booking value (booking revenue/number of bookings)

For tactical decision-making, you need to break these strategic indicators down into manageable segments and analyze their performance by - for example - geography or property, in order to understand how well each is contributing to the overall result; or, assess them by traffic sources and marketing channels to see where you should direct your marketing budget.

But, let me outline a set of leading KPIs which will enable us to gauge the effectiveness of online initiatives and help us steer a Online Travel and Hospitality websites in the right direction:


Online Travel and Hospitality KPI’s:

  • Bookings conversion rate
  • Average booking value
  • Look-to-book ratio
  • Top Geography and Demography
  • Return on Ad Spend (ROAS)

 

Online Travel and Hospitality optimization opportunities:

  1. Visitor profiling and segmentation in Travel and Hospitality.

    Knowing your visitors is an extremely important consideration for the success of your Web site or online marketing strategy. Addressing your key audiences and providing them with relevant information is one of the key aspects of any travel and hospitality site. If your site does not speak directly to each of these audiences, you will lose most of them to the competition.

    Segmentation can help in the analysis and understanding of how important groups of people use the site by looking at the page-by-page activity of the group without being distracted by activity from other types of users. Some of the key criteria for segmentation would be:

    - Demographics
    - Traffic Sources and channels
    - Activity and Navigation patterns
    - Custom built segmentation criteria

    Opportunity: Target the right groups of visitors with the appropriate marketing message, adapt content to users’ preferences and improve usability to fit your visitor’s browsing behaviour.

 

You can find a complete and in-depth PDF version of our inputs for Best Practices for Online Travel and Hospitality here:


DOWNLOAD GUIDE
(4 page commercial version | 754K PDF)



 

Labels: , , ,

Online social network participation inequality and the impact on the User Engagement KPI

posted by Dennis R. Mortensen
Friday, August 10, 2007
Bookmark: Online social network participation inequality and the impact on the User Engagement KPI

In my recent post about what and how to measure Social Networking websites I suggested the following online social network user engagement KPI’s:




  • User Engagement
  • Anonymous visitors to members conversion rate
  • Active member length
  • Time since last login
  • Total time spent on site
With a comment added in regards to the User Engagement KPI saying that: “The User Engagement KPI is a custom session metric designed and calculated from a set of basic metrics such as for example; pages viewed, time spent on site, time since last login, comments posted or other content submitted, subscribed to a feed or alert and so on – the calculation of this KPI is highly dependent on the structure of the Social Network in question).

I think it is of the utmost importance that I add a note to that, as you cannot presently look at an average or median User Engagement KPI without segmenting it, due to the fact that there is an online social network participation inequality. The fact is that most users do not participate very much!

Your UGC-users (User Generated Content-users) are split into three groups:
  • Observers - those who do not contribute
  • Contributors - those who contribute now and then
  • Participators - those who account for most of your contributions
And as you probably guessed - the distribution of the contributions follows Zipf's law – should you plot the User Engagement KPI for each user (If you run a blog, you can for fun play with the idea that User Engagement equals “comments” – and as such plot the number of comments per unique user, assuming you have access to full visitor segmentation in your analytics tool).

I also suggested in the above mentioned online social networking measurement post that you had to create a KPI Index. Here you will have to extend the User Engagement KPI Index with an attitude on the distribution.


Online Social Network User Engagement KPI Index (Distribution attitude)
  • 90% - Observers
  • 9% - Contributors
  • 1% - Participators

The general attitude and rule-of-thumb is not surprisingly that you have about 0.1% participators on a blog (which differs from online social networks in general, by having a very low participation) - this is of course something you have to determine for your blog or online social network.


My conclusion is that:

When using a User Engagement KPI (as suggested by myself) it is highly unrepresentative using either average or median numbers (or voices for that matter, as in specific comments) to conclude on the overall attitude of your online social network, due to the fact that most online social networks have participation inequality! Segmentation must be applied.

Labels: , , , ,

What and how to measure Media and Content websites

posted by Dennis R. Mortensen
Thursday, July 26, 2007
Bookmark: What and how to measure Media and Content websites

By media and content websites, I am talking about pure online media properties like CNET as well as companies (e.g. traditional newspapers like New York Post) who must somehow, either be in the midst of an involuntary transitioning of readers, from offline content consumption (their paper) to online content consumption (their website) – or in the fortunate circumstance that they managed to set up a business model that compliments both channels at the same time. (I am of course talking about any other website where the product is content – thus your typical affiliate website!)

Studies by analysts in general and in particular Jupiter Research concludes that - we are seeing an increased hunger for personalization and thereby a disaggregation of the website - and at the same time the emergence of an online-only audience who at the same time demand participation (social media as we konow it today).

Which makes me conclude on a progressive path and journey most media and content website have to take:
  1. Media
  2. Online Media
  3. Online Social Media
This seems straightforward and as understandable as in taking the newspaper (Media) and reprocess the content on a website (Online Media) and then throw in a couple of “post a comment” widgets (Online Social Media). I do hope that we all agree that is takes a lot more than that! ...and that a lot more is at stake. As online social media continues its striking invasion into the overall media landscape and seize valuable mindshare, particularly from the younger demographic, the degree to which a traditional media company move towards the final goal of social integration of its audience is rapidly becoming a proxy for its ability to survive the future.

If we partly agree on the above I think we also agree that It is simply NOT enough to just randomly report on the effects of this transitioning – in vague hopes of moving towards this goal of social integration. One have to accept that however sexy Online Social Media or even just your Online Media (your basic content website) sound, there MUST be a set of measurable KPI’s across all three steps from above - and they should very much be aligned to your business objectives.


Content websites business objectives:
  • Increase Advertising Revenue
  • Increase Subscriptions
Having in mind that I am allowing for a broad interpretation of the term “subscription” – the above clearly defines the business objectives for most Media and Content websites. Assuming that you agree with me, we then have the opportunity to define a set of specific KPI’s and accompanying important metrics to drive those objectives forward:


Content websites advertising revenue KPI’s:

  • Advertising Revenue
  • Visits per week
  • Ad units per visit*
  • Ads served*
  • Ad CTR
* The “Ad units per visit” KPI is a replacement for the old "page views per visit" as a way of indicating the size of your Ad inventory and the “Ads served” KPI is the actual number of revenue generating banners or other media type served in those units.

Remember, that when talking about these 5 important KPI’s (and no business should really have more than 5 KPI’s on a management level) – we are not just talking about a set of basic reports on their performance in retrospective – we are talking about collecting data on a granular level so that we can both report AND more importantly do analysis on these – by segmenting, slicing and dicing them any way imaginable.


Content websites subscription KPI’s:
  • Subscribers
  • Anonymous visitors to subscriber conversion rate*
  • Cost per new subscriber (CPA)
  • Subscriber churn rate
  • Articles viewed per visit
* "Subscribers" is not necessarily the same as or equal to that of an off-line subscriber, this could be something as loose as a RSS subscriber to something as tangible as an online paying subscriber (who get’s access to premium content by login and password).


Content Websites KPI control elements:
  • KPI Targets
  • KPI Indexes
  • KPI Competitive Intelligence
I am sorry to add to the list of tasks, but one simply have to create a credible target for every single content websites KPI that we discussed above (credible as in utilizing competitive intelligence to set realistic goals) so that there is a clear driver within the organization on where it is going. At the same time while working towards those targets one need to create a sensible content website KPI index that can be used when doing analysis for the KPI in question (sensible as in taking into consideration how the KPI fluctuates due to season, campaigns and other factors). Finally utilizing competitive intelligence to spot market opportunities for optimization and general KPI improvement.

NOW - having clear business objectives, 10 well defined content website KPI’s and a set of KPI control elements in place - we get the opportunity to put all this to work, as in nobody should go to the extent of setting up a framework as the above without aggressively pursuing a performance improvement in ones business objectives. It's really not worth measuring something if you cannot or will-not take any action on it, and to be worth taking action on, it has to have some kind of measurable monetary value. There is no magic 5 points to optimize, but given the framework as described above one would have an unlimited number of opportunities to make more money! - find a couple of suggestion below:

But before starting to think about optimization (however basic) – you need to make sure that the Web Analytics tool that you have deployed can collect and not only report on the mentioned KPI’s – but more importantly: That you are given the technological opportunity to do analysis!


Content websites optimization opportunities:
(7 basic and to some extent obvious suggestions)

My first comment is that before trying to optimize anything for the better - some sort of potential monetary valuation should be put on the effort as in; are we looking to increase revenue EUR 10.000 per year or are we looking to increase revenue EUR 2.000.000 per year and with what certainty can this be determined.

As an example, let’s say that we have 15 Ad units served per visit and that we have an Ad unit CPM value of EUR 3 and that we have 1.000.000 visits per week (notice how all of these are KPI’s) – this leaves us with a weekly Advertising Revenue on = EUR 45.000 (1.000.000 visit * 15 Ad units per visit / 1000 (CPM) * 3 EUR). The optimization opportunity in question estimates that we can increase the number of Ad units served to 18 by adding a small additional unit AND that there will be a 0.30 EUR decrease in Ad unit CPM value due to this (notice that we utilize our KPI’s index here). Therefore a monetary opportunity on EUR 3600 per week or in perspective approximately an opportunity per year of an additional EUR 188.000 in revenue!

Here goes is my 7 basic and obvious suggestions (very specific and less strategic though) on what you should look into when optimizing a content website:

  1. Segment articles based on ad unit placement to create insight on what ad unit placement drive the biggest Ad CTR KPI without decreasing the Ad units per visit.
    Opportunity: Increase Advertising Revenue by optimized ad unit size and placement.

  2. Segmenting the Anonymous visitors to subscriber conversion rate KPI by campaign channels (in my world ALL incoming traffic should be part of a campaign, this including SEO activities) to find and focus on better performing channels and campaigns.
    Opportunity: To increase the Anonymous visitors to subscribers conversion and thus if visits are constant the number of new subscribers

  3. Reporting on the Subscriber churn rate KPI over a given period and drill through (on spikes) to articles (or specific HTML pages) - to determine and create insight of less compelling content. Although it is nearly impossible to determine why subscribers end their relation with you (as basic as stop coming to the Website) – it is still possible over time with repeatedly insight on the same fact to drive a conclusion on what type of content is badly chosen.
    Opportunity: Decrease the Subscriber churn rate KPI and thus increase the numbers of total subscribers.

  4. Setting up basic external campaign tracking on ALL channels from PPC to Email to affiliate networks and compare not only channels, but compare campaign to campaign across channels, sorting out high performers and low performers to Decrease the Cost per Subscriber (CPA) KPI.
    Opportunity: To decrease the average cost per new subscriber and thus an the opportunity to re-invest that into new external campaigns

  5. Utilizing the articles viewed per visit KPI together with the Visual Overlay tools from your Web Analytics package to define Homepage content blocks -- not unique articles -- (this works for content category front pages as well). These content blocks typically on e.g. a newspaper website being; “Main article”, 2 sets of “Sub articles” and a list of “Most popular articles” and so forth. Then determining a CTR (Click trough rate) baseline for each of these content blocks – Now having an Index on the expected click through on e.g. the main article on the homepage on, say, 14% - any underperforming articles (as compared to the index and thus below 14%) can be replaced.
    Opportunity: Increase the Articles viewed per visit by pulling underperforming articles away from prominent positions.

  6. Tracking ALL your Internal Promotions (essential campaign tracking for banners or links on your site promoting other sections on your site) and then segmenting your visitors by these Internal Promotions and looking at the Ad units per visit KPI – you will to begin with, find out which internal promotions actually drive more Ad units (potential revenue) and more importantly which internal promotion does not work and at worst drive DOWN Ad units per visit according to your index.
    Opportunity: Increased Ad inventory by better internal promotions.

  7. Try to increase the “quality” of your Ad units per visit KPI by using External Data Sources and relate your CRM user database to the Member-ID tracking done by your Web Analytics package (this will give a plethora of opportunities), but one obvious is generating a simple report by content group and thereby get insight into the demographics and sociographics for every specific content group, whether that be sports, politics or tech.
    Opportunity: Increase the Ad unit CPM value by demanding higher pricing for better targeting and at the same time expect higher Ad CTR
Even though these 7 basic suggestions (and both you and I could come up with 50 others) somehow seem simple – then I am confident that anybody running a content site would be far better off looking at and concentrating on the KPI’s framework described.

My overall conclusion in this post about media and content websites is that; one simply cannot be successful -- in the long run – as a media company moving towards embracing Online Social Media, if not one or more activities around enterprise optimization analysis is deployed. I hope that the above somehow indicated and inspired which direction to go in.

N.B.
I recently wrote a post about what and how to measure Online Social Media websites, this including 7 online social media optimization opportunities (which I suggest you take a look at – should you be on the path towards becoming an Online Social Media). You will see that the KPI’s in that post to some extent are a progression of the KPI’s for Media and Content websites. Which is of course only natural and expected.

Labels: , , , , , , ,

What and how to measure Social Networking websites

posted by Dennis R. Mortensen
Sunday, July 15, 2007
Bookmark: What and how to measure Social Networking websites

By social networking websites, I am talking about the likes of Facebook, Myspace, Linkedin, Xing and StudiVZ (and of course any other web property who are engaged in Social Media activities – like any decent blog you know).

Social Networking is the prime headline for Web 2.0 and highly successful so far. This fact only leading towards an even greater responsibility in trying to measure ROI on ones social networking activities, especially as it is somehow agreed upon that only a limited number of players can win (having Metcalfe's law in mind). Remember how quickly Friendster somehow lost momentum, I bet you that they did not deploy detailed analysis, at the level required in this game, within their organisation! – if so, they would have spotted the decreased user engagement and its relating metrics in time.

First I would like to conclude that one have to accept that however sexy social networking is as a trade, there MUST be a set of measurable KPI’s that are strongly aligned to ones business objectives, like any other organisation that operates with the web being a channel of theirs. Hereby concluding that KPI’s are meant to be acted on and not just reported on. Most of the social networking sites that I have been working with somehow work to optimize the following two business objectives.


Social Networking business objectives:
  • Increase Advertising and/or premium member-ship Revenue
  • Increase User Engagement

Now that we have clear business objectives defined – and some that in my humble opinion are pretty much spot on in regards to the overall success of any social networking business. Assuming that you agree with me, we now have the opportunity to define a set of specific KPI’s and accompanying important metrics.


Social Networking advertising revenue KPI’s:
  • Advertising Revenue
  • Visits per week
  • Ad units per visit*
  • Ads served*
  • Ad CTR

* The “Ad units per visit” KPI is a replacement for the old "page views per visit" as a way of indicating the size of your Ad inventory and the “Ads served” KPI is the actual number of revenue generating banners or other media type served in those units.

Remember, that when talking about these 5 important KPI’s (and no business should really have more than 5 KPI’s on a management level) – we are not just talking about a set of basic reports on their performance in retrospective – we are talking about collecting data on a granular level so that we can both report AND more importantly do analysis on these – by segmenting, slicing and dicing them any way imaginable.

In the end for any Social Network to succeed and survive it’s users have to be engaged with the website - and that engagement actually turns into positive revenue as they spend more time, take more action and convert on a higher rate than non-engaged users. Concluding that we are looking at a win-win-win situation for the user, the social network and finally it’s advertiser. We of course have to agree on what “user engagement” is to begin with and inspired by Eric Peterson I would say:

Social Network User Engagement is an estimate of the degree and depth of visitor interaction on the website against a clearly defined set of measurable goals.” – and with that in mind we can define our KPI’s.


Social Networking user engagement KPI’s:
  • User Engagement*
  • Anonymous visitors to members conversion rate*
  • Active member length
  • Time since last login
  • Total time spent on site

* The “User Engagement” KPI is a custom session metric designed and calculated from a set of basic metrics such as for example; pages viewed, time spent on site, time since last login, comments or other content submitted, subscribed to a feed or alert and so on – the calculation of this KPI is highly dependent on the structure of the Social Network in question). The “Anonymous visitors to members conversion rate” is very much a growth KPI that is used until the Social Network reaches its final plateau.

I think it is of the utmost importance (as in any other media company) to understand what benefits the advertiser and how he get the best possible ROI. Social Networks have the opportunity to create value for the advertiser far beyond the click -- if the Ad creative and Ad content showed allow this -- so it is also important to track the actions taken beyond the click (and in general for the website). Such accompanying social network user engagement action metrics like:

- Forwarded advertiser content
- Endorsed advertiser content
- Rated advertiser content
- ..and other obvious types of engagement with an advertisers content

As you can see the secondary audience is as important as the immediate audience and somehow one should be able to illustrate that value to ones advertisers - as in being able to increase revenue based on the fact that the advertiser gets proven value increase by word of mouth with the secondary audience derived from the immediate audience.

I also find it important that one set up specific custom fields for tracking such accompanying social network user engagement metrics like:

- User profile completion level
- Premium member level

Both the accompanying set of metrics for advertiser and user engagement are super specific - so the above two small lists are only a couple appetizers. More importantly how do we “work” the KPI’s


Social Networking KPI control elements:
  • KPI Targets
  • KPI Indexes
  • KPI Competitive Intelligence

I am sorry to add to the list of tasks, but one simply have to create a credible target for every single social networking KPI that we discussed above (credible as in utilizing competitive intelligence to set realistic goals) so that there is a clear driver within the organization on where it is going. At the same time while working towards those targets one need to create a sensible social networking KPI index that can be used when doing analysis for the KPI in question (sensible as in taking into consideration how the KPI fluctuates due to season, campaigns and other factors). Finally utilizing competitive intelligence to spot market opportunities for optimization and general KPI improvement.

NOW - having clear business objectives, 10 well defined social network KPI’s and a set of KPI control elements in place - we get the opportunity to put all this to work, as in nobody should go to the extent of setting up a framework as the above without aggressively pursuing a performance improvement in ones business objectives. It's really not worth measuring something if you cannot or will-not take any action on it, and to be worth taking action on, it has to have some kind of measurable monetary value. There is no magic 7 points to optimize, but given the framework as described above one would have an unlimited number of opportunities to make more money! - find a couple of suggestion below:


Social Networking optimization opportunities:
(7 basic and to some extent obvious suggestions)


My first comment is that before trying to optimize anything for the better - some sort of potential monetary valuation should be put on the effort as in; are we looking to increase revenue EUR 10.000 per year or are we looking to increase revenue EUR 2.000.000 per year and with what certainty can this be determined.

As an example, let’s say that we have 15 Ad units served per visit and that we have an Ad unit CPM value of EUR 3 and that we have 1.000.000 visits per week (notice how all of these are KPI’s) – this leaves us with a weekly Advertising Revenue on = EUR 45.000 (1.000.000 visit * 15 Ad units per visit / 1000 (CPM) * 3 EUR). The optimization opportunity in question estimates that we can increase the number of Ad units served to 18 by adding a small additional unit AND that there will be a 0.30 EUR decrease in Ad unit CPM value due to this (notice that we utilize out KPI’s index here). Therefore an monetary opportunity on EUR 3600 per week or in perspective approximately an opportunity per year of an additional EUR 188.000 in revenue!

Here goes is my 7 basic and obvious suggestions (very specific and less strategic though) on what you should look into when optimizing a social network:

  1. Segment users based on their community activity (this could be anything from viewing pictures or videos to commenting and participating in forums, this is essentially all the defined actions in your web analytics package) - and list this according to the Ad CTR KPI and comparing the different segments to your Ad-CTR index. This giving you an opportunity to do better internal promotions (as basic as a link in the right place) for better converting content (that above your index) on poorer performing content pages.
    Opportunity: To increase the Ad CTR and thus the Ad unit CPM value.

  2. Segmenting the Anonymous visitors to members conversion rate KPI by campaign channels (in my world ALL incoming traffic should be part of a campaign, this including SEO activities) to find and focus on better performing channels and campaigns.
    Opportunity: To increase the Anonymous visitors to members conversion and thus if visits are constant the number of new members

  3. Perform single-page FORM Analysis (tracking every single FORM field) on the sign-up page and cross-reference this to the Anonymous visitors to members conversion rate KPI - getting to know which fields make Anonymous visitors abandon the sign-up form.
    Opportunity: To increase the Anonymous visitors to members conversion and thus if visits are constant the number of new members

  4. Create a custom report with the metric Time since last login KPI and a grouping by entry pages. Filter out all user who have visited the site the last 5 weeks so that you end up with a list of somehow disengaged users who returned to the site. The most successful entry page for disengaged user should be the one heavily used in e.g. Email Marketing. (A common page on this list is the “Your Friend x added you as a friend”).
    Opportunity: To decrease the Time since last login and thus increase the numbers of Visits per week and thereby increase Ads served and finally increase Advertising Revenue as a result. At the same time increasing Active member length.

  5. Run an out of the box standard Action Participation report and add the User Engagement KPI – creating an easy an instant understanding of what content triggers a passive visitor to become a active contributing user and member of the community. As of now you probably do not really know whether the highly engaged users connect to the community by watching user submitted videos, read gossip, chat with friends or what have you.
    Opportunity: To increase the User Engagement compared to our index and thus we typical see an increase in Visits per week and thereby an increase in Ads served and finally increased Advertising Revenue as a result.

  6. The Total time spent on site KPI is not only important because Nielsen//NetRatings has officially replaced "page views" with the "time spent" metric for their official ranking -- whether you agree with Nielsen or not -- it is still very important to know which content groups that users spend their time on - the most valuable resource they have. Create a report that shows Total time spent on site divided per content group and conclude the individual standing on these content groups weighed against your competitors using competitive intelligence. Not as a relative result, but as in looking at the fact that if a user is willing to spend 15 minutes watching user generated videos in general, this is the actual number to target). Optimizing content groups that deliver below industry standard time consumption.
    Opportunity: Increase Total time spent on site and thus an increase in Ads served and finally increased Advertising Revenue as a result.

  7. Try to increase the “quality” of your Ad units per visit KPI by using External Data Sources and relate your internal user database to the Member-ID tracking done by your Web Analytics package (this will give a plethora of opportunities), but one obvious is generating a simple report by content area and thereby get insight into the demographics and sociographics for every specific content area (network of people).
    Opportunity: Increase the Ad unit CPM value by demanding higher pricing for better targeting and at the same time expect higher Ad CTR

..and a bonus optimization comment/suggestion inspired Jim Novo on why using the “User Engagement” KPI on a management level can save your business (or Job). Let say you have a Social Network for students and the number of members and total number of actions in general is increasing (up and to the right), but the “User Engagement” KPI and the related “Time since last login” is decreasing in value more rapidly – then at some point the disengaged users overpowers the increase in members and the website starts to spiral downward in volume. One can predict this downward spiral will happen by monitoring these KPI’s (even as a basic management Alert in your Web Analytics tool) so one can try to take action before it's too late. Most people look at volume as a measure of popularity and growth and by the time volume starts dropping, it is already to some extent too late to save the website. The audience has already disengaged. The user engagement drops off before the traffic volume does, and that is why user engagement is predictive and directly addresses future value.


Even though these 7 basic suggestions + bonus comment (and both you and I could come up with 50 others) somehow seem simple – I am confident that anybody running a social network would be far better off looking at and concentrating on the KPI’s framework described.

My overall conclusion is that one cannot be successful -- in the long run -- running any social media activities if not one or more activities around enterprise optimization analysis is deployed - and I hope that the above somehow indicated and inspired which direction to go in.. sorry for the partly long post (essay) :-)

Labels: , , , , ,

Web Reporting vs. Web Analysis

posted by Dennis R. Mortensen
Saturday, June 9, 2007
Bookmark: Web Reporting vs. Web Analysis

I believe very strongly that one have to understand that reporting on any metric at pure face value within a very limited context is unsound and should not be confused with actionable insight from analysis! – To wrap up that statement I tend to conclude that:

  • Reporting is produced by tools
  • Analysis is done by people
  • Processes are deployed by organisations

Which leads us to following daring fact:

The right tools +
The right people +
The right processes +
= Data Driven Organisation

Watch my presentation on the matter and find out (or at least watch my take on it) how you tell the difference between reporting and analysis.



Download the PDF here: IndexTools-Dennis-Reporting-vs-Analytics.pdf
... OR ask me to do a presentation on the subject for your organisation. :-)

The conclusion is worth repeating - analysis is all about:

  • Developing KPI’s
  • Creating Insight
  • Taking action

I would like to recommend the following great and in depth posts about the subject:

It does not get any better than that!

Labels: , , , , , , ,

Measuring dynamic internal search performance

posted by Dennis R. Mortensen
Tuesday, May 22, 2007
Bookmark: Measuring dynamic internal search performance

As a follow-up to my previous post on how to use Web Analytics to determine the width of your Internal Search Query box – I thought it would be suitable for me to deploy an internal search utility on my blog. Looking around for a solution (beyond just adding a link to technorati) that could provide us with quick, accurate and measurable results – I ended up deciding to use the Google AJAX Search API. First, because it is free (no money is made on this blog), second because I could “code” it to my taste... and it is actually enjoyable being geeky from time to time :-)



I caught myself in trying to code the search utility for optimized tracking and not for optimized usability – this, as in I wanted the “normal” Search Query Box functionally:
  1. type in full query
  2. refresh screen with a set of results (where the search query and results kpi’s are tracked)
However; Internal Search utilities are changing and I finally decided to go with the potentially better “Web 2.0” solution (time will tell). A solution where instant results are shown on an unchanged page after each keystroke.

Let us try to assess the results before talking about WHAT to measure: Searching for e.g. “emetrics san francisco 2007” we get:



Search results - “Web 1.0”
1. Emetrics San Francisco 2007
2. My comments to Emetrics London 2007
3. NEW Google Analytics launched at Emetrics SF Today!
4. Notes from ad:tech San Francisco
5. 9 Ways to Make Money on Analytics

Setting up a similar and comparable list is simply not possible in a dynamic and instant web 2.0 results environment (as the one deployed on this blog) – as we have instant results after each keystroke. We can on the other hand look at the numbers of results per keystroke:

Search results - “Web 2.0”
1 results for the phrase “e”
0 results for the phrase “em”
0 results for the phrase “eme”
0 results for the phrase “emet”
0 results for the phrase “emetr”
0 results for the phrase “emetri”
0 results for the phrase “emetric”
8 results for the phrase “emetrics”
8 results for the phrase “emetrics ”
5 results for the phrase “emetrics s”
0 results for the phrase “emetrics sa”
3 results for the phrase “emetrics san”
5 results for the phrase “emetrics san ”
3 results for the phrase “emetrics san f”
0 results for the phrase “emetrics san fr”
0 results for the phrase “emetrics san fra”
0 results for the phrase “emetrics san fran”
0 results for the phrase “emetrics san franc”
0 results for the phrase “emetrics san franci”
3 results for the phrase “emetrics san francis”
3 results for the phrase “emetrics san francisc”
5 results for the phrase “emetrics san francisco”
5 results for the phrase “emetrics san francisco ”
0 results for the phrase “emetrics san francisco 2”
0 results for the phrase “emetrics san francisco 20”
0 results for the phrase “emetrics san francisco 200”
5 results for the phrase “emetrics san francisco 2007”

So dependent on how quick I am to type and how quick my internal search utility is to respond I could end up performing 27 searches. If I am semi fast in typing in the query I might end up doing a random 10 or 15 searches, furthermore I might not have the qualification to look at the screen while I type and then as a usability fact “only” see the final result. Another interesting fact is that, if I were in fact searching for “Emetrics San Francisco 2007“ and that I would type and see results at the same time, I would stop after “emetrics”, but even more challenging, this would only count as a success search phrase keyword until I write a new post about Emetrics which would then be on top and then the “Emetrics” term might not be enough to guide people to the “emetrics San Francisco 2007“ post.

This is really fascinating!? :-)

Let us try to have a look at the standard KPI’s (a off the top of my head list - so do not take this as a final list of the most used Internal Search KPI’s, however; the list still rings true and must still pretty much be expected of anybody doing decent internal search tracking)

KPI’s for Internal Search - “Web 1.0”
  • Search action (whether a search have submitted)
  • Search phrase
  • The number of search results
  • No results action
  • The number of searches per visit/visitor
  • Search successful (as in clicking on a result)
  • The number of search attempts before success
  • Advanced search action (narrowing down the search)
  • Search location

KPI’s for Internal Search - “Web 2.0”
  • ?
I have plenty of ideas, as I think it is somehow obvious that the standard static internal search Web 1.0 KPI’s are not very suitable for a Web 2.0 environment, but I have no data as we speak, so it is time to deploy (from a tracking point of view) a few of them and see how actionable the results are. And this might even give me an opportunity to consult our Oracle “Avinash” or the smart people at OX2 as well :-) - More about the subject later when I have had some time to do a bit of research. And PLEASE do leave a comment with your ideas. :-)

N.B.
If you want great input for how to evaluate static internal search performance – go look at Gary Angel’s input on the matter. It is as usual a first-class post.

Labels: , , , ,