When should I use hits, visits or visitors in my Adobe Analytics Segments

This past week I got asked this question and although I have no doubts when I am creating my Adobe Analytics (AA) segments I still could not give a clear answer. Later on that day, I stoped and thought about my logical mechanism to create the segment. This is how I choose.

Problem that I am trying to solve
Segments is a very powerful and useful tool in AA. It allows you to create variables that were not available before. And because it is so useful, some analysts forget that they should be created to solve problems and not expand variables.

For example, let’s create a situation where I am asked to filter all conversions for a specific product, here called product bazinga. The checkout url for my commerce variables are all the same after a parameter but they all differ in the product name. No need to say that the product name is not shared in this data layer setup. This is how the confirmation url would like:

When someone buys the product cozinga:
www.companyfic.com/cozinga/checkout/orderconfirmation

And when someone buys the bazinga product we:
www.companyfic.com/bazinga/checkout/orderconfirmation

This case it’s a no brainer because I want to filter all pages that contain the string bazinga and the string orderconfirmation. And when the user hits these pages. It does not matter for me what they do after, before, in a different day. All care is that hit.

Basically I defined my problem. Defined the view I was trying to focus on and then it is clear which option I should choose.

Variables involved in the report

Now that the product is clear, looking at the variables you will be selecting in this report is also pretty useful. If the variables are more related to the hit, page views, time on page, scroll down then that’s what you are looking for. If the variables are more related to the visit (such as revenue, basket size, day of the order) then you should definitely use visit. The same with visitor.

Hit: I am focus on the page and what happens in that page. Anything after or before does not influence my segment. Ex: Page isolation, bounce rate in pages, product analysis.

Visits: I am focus on that visit, pages only matter in a visit level. Ex. what is number of page views for the product bazinga that comes SEM traffic source.

Visitors: I have a broad focus and I am interested in users. It does matter what users did in a previous visit. Ex. I want to know how many users from BC bought the product bazinga in the last 30 days.

 

Watchouts

Be careful when using segments that were created for different purpose, it might influence results. Ex. If you have your bazinga purchase segment and the business asks you to answer a question related to traffic source, you might have a higher number than you were supposed to. You are applying a hit segment to a problem that requires a visit segment for instance.

 

 

 

 

Visitors versus Unique Visitors

One of the concepts that people usually misunderstand when speaking about digital data is the terminology used in the platforms. For example, I often hear these questions:

  • “Are you giving me unique visitors or visitors?”
  • Or even “do you have unique visits too?”.

They exemplify how the concept is not crystal clear and therefore, they end up misused.

Visitors

Visitors are defined as each and every person that accesses the website. For example if we were talking about a physical store, the unique visitors would be the amount of people that are visiting the store in a specific time frame. In Omniture this is clear because it has unique visitors metric break down by month, quarter and year. No one says “I had 50 unique clients in my store this afternoon” because clients are all unique. But we do say unique visitors when we refer to the digital store – most of it because of Omniture terminology – but they actually refer to the same concept: clients.

New Visitors

New visitors in Omniture are associated with the visit number, any visitor that has the visit number equals 0 is considered a brand new one. And to have the new visitor equal 0, it would have to be the absolutely first time ever you have been to that site using that device. Well, it is cookie based.

Also, as a consequence of using the cookie register, if he/she deletes/cleans the cache, we are no longer able to identify the customer history. Therefore it would count as a new visitor.

Returning Visitors

Because we have new visitor, we also have returning visitor. Basically, it is anyone that does not have a visit number 0 attached to the cookie. Or anyone that is entering the store/website for a time that is not the first.

Visits

Going back to the context of a physical store, visits are the number of times someone enters the store. If they leave and come back, it’d be considered a new visit even though it’s the same client. In this context, every visit is unique too. Well, what would be a visit that is not unique, if such term exists? Exactly! There’s no such thing as returning visit because what it returns is the visitor.

So putting in a context, a visitor comes to the site for the first time and within the same day visits the site again and buys something and leaves.

This is the table of key metrics for that day:

New visitors 1
Unique Visitors 1
Visits 2

 

So in conclusion the clients (unique visitor/visitor ) comes (visit) the store for the first time (new visitor) on the day one, and come back (returning visitor) on the day 2.

I hope this can help people to understand better this concepts.


 

What’s behind A/B tests

Well often I’m asked how an A/B test work.

So my first saying, which is the most important thing that I’m going to mention here, is: A/B test is not a simple comparison of frequencies. So don’t ever use the terminology A/B test when you are just comparing 2 numbers, like EVER.

Now that we drew the line let’s go back to the post.
There are several tools around (most of them being embedded in your web BI solution like Adwords, Adobe Site Catalyst, etc), but in fact you don’t really get to understand how it’s calculated.

So here I’ll show (and share in my github account) some methods to do A/B tests.

Using R
Because I don’t have any real life data that I can share, I’m going to use the ToothGrowth dataset from R (please find more details about the dataset here). Basically they treated 60 guinea pigs with 6 variations of treatment and they collected the size of their teeth after a while. Now we are interested in understanding what’s the best treatment.

We can bring this example to any kind of industry that is not medical/health related. For example, let’s say you are running a campaign for sofas and for that you’re using 2 different digital channels: Facebook and Display. But you decided to have 3 types of campaign: a small banner, a big banner and a flash animation. Then you gave 6 variations of treatment just like our guinea pig example.

So going back to our dataset. We have 2 delivery method for vitamin C (Acid Ascorbic and Orange Juice) and 3 different doses – 0.5mg, 1mg, 2mg. This is the reason for the 6 variation. To make it easier (and possible see more in here), we are assuming the the whole dataset has normal distribution.

So first, let’s find out if there’s difference between the delivery methods.
It’s always a good idea to check the stats for A/B sample, like mean and variance. You can find these stats on the bottom of this page

So the first question: is there a difference between guinea pigs that took ascorbic acid and those that took orange juice?

$$H_0$$ Null Hypothesis: $$\mu_{orange juice}$$ = $$\mu_{ascorbi cacid}$$ or $$\mu_{orange juice} – \mu_{ascorbi cacid} =0$$

 

$$H_1$$ Alternative Hypothesis: $$\mu_{orangejuice} \neq \mu_{ascorbic acid}$$ or $$\mu_{orange juice} – \mu_{ascorbi cacid} \neq 0$$

So in most of the cases, we use the p-value as a comparison for our test. Being higher than 0.05 considered high. If you’re not familiar with the concept of p-value, please check here.

The calculation behind the test is considering our null hypothesis true, what’s the probability of having an extreme value outside of standard variation?

Basically we create a new variable called $$OrangeJuice-AscorbicAcid$$ that has mean as:
$$\mu_{orangejuice} – \mu_{ascorbi cacid} $$ and variance as
$$\frac{S_{orange juice}^2}{n_{orange juice}}+\frac{S_{ascorbic acid}^2}{n_{ascorbic acid}}$$

In a gross way explaining, the number you should calculate is

$$\frac{\bar{X}_{orange juice} – \bar{X}_{acid ascorbic}}{\sqrt{\frac{S_{orange juice}^2}{n_{orange juice}}+\frac{S_{ascorbic acid}^2}{n_{ascorbic acid}}}}$$

That has T-distribution if $$n_{orange juice} + n_{acid ascorbic} < 40$$ and Normal distribution otherwise.

Back to our test, first we filter the variables by Ascorbic Acid (VC) and orange juice (OJ)


vc_subset oj_subset

then we can apply the t-test


t.test(vc_subset$len, oj_subset$len)

Which results into
test1_vc_oj

P-value is higher than 5% so we decide against the null hypothesis.

The rest of this dataset analysis you can see in the github repository that I shared in the bottom of this post.

Now assuming we have 2 campaign running using different templates.

For the first campaign the click through was 25% and for second one the click through was 40%. It’s hard to tell with only these two piece of information if there’s difference between the campaign. For example, let’s say that the first campaign we have 100 people seeing it, while the second one only 10. Even having a much higher click through rate, the second campaign has a very low number of impressions and it’s hard to say if there’s difference between them.
In the chart below you can see that we have to show to at least 55 people keeping the 40% to say that the there’s difference between the samples. And at least 40 to conclude that the campaign 2 is better than the campaign 1.

pvalue_chart

Currently there are several links that can be used to do an A/B test, such as:

  • https://vwo.com/ab-split-test-significance-calculator/
  • http://www.hubspot.com/ab-test-calculator

I’m also uploading a spreadsheet with the proportion and the mean tests here.

 

And for more information about how to implement t-tests in R please access this link