SemAngel

The Visit is Dead, Long Live the Visit

In my last post on building a digital data model for the analytics warehouse, I described the concept of statistical ETL and argued for its critical importance in creating a robust and easily used visit-level data model. That post drew an interesting comment from a friend of mine arguing not with the concept of statistical ETL, but against the idea of a visit-level aggregation. Here’s the comment (from John Stansbury):

Gary, As usual, great blog. It just seems that we've got to get past the notion of visits and sessions. Those entities only exist for the convenience of measurement tools--consumers don't consciously think of a "visit" to a site, and they certainly don't think, "well, it's been 30 minutes of inactivity so if I click now it will be entirely different activity from just a little while ago." Marketers and Analysts have got to address customers in terms of their goals. I know it's difficult, but until Marketers demand it the tool vendors won't change paradigms. Customers don't visit, they shop. And if your customers' shopping crosses time and technology, you had better market to them that way. 

There’s a lot in here that I agree with. John’s surely right that the “visit” as we know it as largely an artifact of the convenience of our technology vendors. The thirty minute time-out, in particular, is so clearly arbitrary that not much except convenience can be said in its defense. Not that I would completely denigrate convenience, particularly when it comes with an element of plausibility. We draw these arbitrary lines in the sand all the time (draft age, voting age, drinking age, age of consent) and then we gradually begin to assume they have some weightier meaning than convenience and custom. But it’s also true that we often HAVE to draw some lines in the sand and if we don’t invest them with meaning then we haven’t really got a line.

So the more interesting question is do we need to draw this particular line at all – is the concept of a visit useful?

I think it is, but for a concept I described in my last post as central, I’m rather less confident of that than you might expect.  In Mobile App measurement, for example, we’ve experimented with the notion of replacing the idea of a “visit” with something I call a “unit-of-work”.

What’s the difference?

A unit-of-work is the complete set of activities over any period of time focused on a single task. If, for example, I was planning an international trip, the unit-of-work on a travel website would include ALL of the activity dedicated to booking that flight. That unit-of-work might span months, numerous periods of activity, and many, many different actions. At the same time, if I use a travel site to book my plane travel and then my hotel, I may have two units-of-work in one sustained period of activity lasting only a few minutes in time.

The unit-of-work paradigm is, therefore, quite distinct from the visit. What we call a visit might contain a part of a unit-of-work or two or three complete units-of-work. For a unit-of-work, we’d likely measure the number of periods of activity, the total calendar time elapsed, the actual activity time spent, and many other metrics that don’t exist when you think about visits. I happen to think these metrics are often more interesting than the visit-constrained ones we typically use.

There’s one advantage to the unit-of-work concept that is particularly attractive to us here at EY. It eliminates the problem of multi-purpose sessions in a two-tiered segmentation while fitting that segmentation scheme beautifully in every other respect. A unit-of-work just is a use-case – the 2^nd tier of digital segmentation. And by eliminating visits, the problem of how to classify those visits with two or more clearly different use cases is also eliminated.

Interestingly, you can ONLY switch to a concept like unit-of-work if you deploy something similar to our two-tiered segmentation and statistical ETL.  

Now as John rightly points out, in our digital analytics tools, we are locked in to the concept of a visit. But in the analytics warehouse, we have no such constraint. Just as I recommended unpacking some Adobe data feed records in my first post on building a clean event-level feed, there’s no reason why we can’t re-organize our data into a unit-of-work level aggregation in the warehouse.

So should you scrap the concept of a visit in your warehouse?

For a few types of Websites, I think the answer may just be yes. If you have lots of “mixed” sessions or lots of multi-session transactional behavior, then it seems to me that the unit-of-work concept might work better than the traditional visit. For other Websites, however, I think the visit still makes sense as a logical way to organize the data.

Think of a visit as a sustained short-term period of activity on a Website. As such, I think it captures an important and real aspect of digital behavior.

As a sports-fan, I usually keep a sports-site in a tab on my browser. Periodically throughout the day, I’ll come back to that tab, refresh the main page, and check scores or look at new stuff. Each time I do that, it seems to me that I’ve started afresh – giving the site a new opportunity to engage me. Do those events necessarily happen thirty minutes apart? Sadly, no (at least on Football Sundays and other sports-heavy days). On the other hand, we don’t necessarily have a good way to capture and track when a browse session actually has focus and even if we did track detailed focus events, we’d still probably be reluctant to describe very short losses of focus as creating distinct periods of activity. If I flip away from my sports site for 2 minutes and then come back, it’s quite different than when I’ve been away for 2 hours.

Which brings us back to drawing lines and the thirty minute boundary question. Is 30 minutes a good boundary line? Probably not for most sites, but that’s a question to be answered empirically not in the abstract. You have to draw the line somewhere and the right place is going to be arbitrary but justifiable as being largely correct and conservative in whatever direction it seems most important to protect.

Indeed, it might make sense to blend the traditional time-based visit concepts with the activity based unit-of-work model and think about some time period in conjunction with some change in activity. If, for example, I pop the Gametracker for the Giants v. Dodgers into a browser window, then go away for 35 minutes and come back to check the score, it hardly seems like I’ve changed my activity in any significant way. It feels to me like the same visit as long as I’m tracking the same game. But if I pop that same Gametracker, come back in 15 minutes, refresh to the home page and read an article about the NFL draft, then I’ve changed my activity pattern and probably generated a new visit. On the other hand, if I do that same pattern inside a minute, I’d be inclined to call it all part of the same “mixed use-case” session.

It’s complicated!

I think blending time and activity might yield a model that comes closest to representing real digital behaviors in an intuitive fashion. But don’t kid yourself – it will still contain some largely arbitrary distinctions and will regularly misrepresent certain types of browsing behaviors. That’s the price we nearly always have to pay for creating an abstract model of any complicated reality.

So is the visit dead? Maybe. But long live the visit!

Related articles

Big Data Analytics and Digital Detail Data

Statistical ETL and Big Data

A/B vs. MVT – are we asking the right question?

Guest Post by Kelly Wortham

I’ve been involved in several conversations lately regarding the pros and cons of MVT vs A/B for isolating and understanding results of optimization testing. The conversations go something like this:

• “A/B gets you an answer so much faster with easier to interpret and explain results!”
• “MVT is the only way to understand each element’s interaction with the other elements!”
• “MVT can create nonsense combinations that simply waste traffic and unnecessarily lengthen test time while also risking customer experience! At least with A/B you can be confident you’re avoiding that!”
• “If you’re not running MVT, you don’t have an advanced testing program!”

And so on. The thing is – these concerns may be valid, but they all miss the point. It’s not about one vs. the other. It’s about designing the test and test program to achieve your goal. And if your goal ISN’T insights that help you take action – it should be, because without insights, you’re still making decisions blindly and without the ability to take action, you just have some really knowledgeable analysts wasting resources.

But if your ultimate goal is to achieve actionable insights, you have more options than just MVT vs. A/B.

1. You can manually create an A/B…N test with recipes designed to test all relevant combinations of elements to isolate element and interaction impacts while avoiding the nonsense recipe risks and wasted traffic inherent in tool-automated MVT. Runtime will be longer than traditional A/B tests simply due to the higher number of recipes but shorter than a full factorial MVT with reduced customer experience risk and content creation demands due to the removal of the nonsense recipes. Results should be as easy to interpret as A/B and equally easy to communicate.
2. If you need to run multiple tests concurrently (such that the same customer can qualify for more than a single test in the same visit), you can either combine the tests as an MVT to understand impacts of the different combinations of interactions or you can conduct an analysis on the back end looking at each group separately (ex: Test 1 – Control, Test 2 – Control = “real control” while Test 1 – control, Test 2 – variant = true test 2 variant performance and Test 1 – variant, Test 2 – variant = variant interaction impact, etc). Both options will require same amount of traffic and results should be about as easy to understand and communicate. However, setup may be more complicated if you try to combine into MVT depending on your particular testing tool’s capabilities. Therefore, unless you are confident the two tests have a high likelihood of interacting with the results of the other, its likely best to simply use traffic splits and keep the results separate. If both variants win – you can always roll out a follow-up test combining the variants to value interaction impact.
3. Best of all solutions – do not focus on interaction of a few elements or tests. Instead, use strategy to design a continuous optimization roadmap with pre-planned A/B…N iterations and concurrent tests where reasonable utilizing traffic splits to ensure only the planned interaction tests are run on the same traffic. Remember, concurrent testing analysis as described in #2 becomes infinitely more complex as each additional test and recipe is added to the analysis and can quickly become unmanageable for both analysis and required traffic.

The optimal test type is much less about the tool and much more about your program strategy and overall goals.

Do you have a brand new radical redesign to test? MVT isn’t necessary (nor would it be helpful). Use A/B – primarily to measure impact of your new design.

Are there several elements you want to test and you have data to help you design the optimal combination of each? Then you don’t need MVT to do it for you. You can just create it yourself!

Are you lacking data to tell you where to focus for a specific page redesign? Are there several elements in the template and you’re largely ok with the current template but would like to understand best combination (and/or location) of elements? MVT might make the most sense for you to more quickly focus your future A/B testing efforts on the elements with the greatest potential impact.

Do you have the data to help you design smart A/B tests but just want to do “advanced testing” and think MVT is the next logical step? You might want to rethink that goal. The most advanced programs focus much less on the type of test and much more on if the test program is delivering actionable insights - something MVT is not particularly well known for.

So next time someone asks you which is better, MVT or A/B, you can answer none of the above! Or is it all of the above? MVT might help you focus your efforts when data is limited or theories abound, but A/B drives learning and action.

So good luck and get testing! And as always – would love to hear your thoughts on this topic and all things testing.

Statistical ETL and Big Data

I began this series with a short essay on the meaning of big data and why digital data is paradigmatic of big data. In digital, the unit of meaning lies above the level of the individual records we collect and cannot be represented as a simple aggregation of those records. A page view, taken in isolation, is largely without meaning, and no simple count, sum or average of page views will capture the meaning inherent in the behavioral stream. This lack of meaning at the detail level is a huge challenge.

In my last post, I described a re-formulation of the event-level digital data to make it more compact, better structured and more robust in supporting queries around sequence and time. That’s a huge step forward in supporting digital analysis, since sequence and time are often critical to uncovering the meaning inherent in the data stream.

Even with these improvements, however, the lack of meaning at the individual record level and the challenges in understanding what the data in the stream means make querying the data extremely difficult.

To understand why this is, I’m going to circle back on one of the fundamental concepts we’ve brought to digital analytics. The idea is simple: to build effective digital analysis you need to have a basic idea of who somebody is and what they were trying to do online. We call this Two-Tiered segmentation (the first tier is a traditional segmentation and the second tier is the visit intent) and it’s at the heart of our approach. That second part, understanding visit intent, is the key.  You won’t know – can’t know – whether your Website or Mobile App were successful in a given visit unless you understand why the visitor was there.  

Far too many enterprise Websites make the simplifying assumption that every visitor is there to buy, book, or generate a lead. That’s usually not the case – and often it’s so far from the truth as to make aggregated metrics that rely on that mono-purpose assumption completely useless. What’s worse, even when ALL your visitors are on your Website for a single purpose, there are often significant differences in the details of their intent. On an ecommerce site, a visitor may have the intent to buy a specific product, the intent to buy something in a category, or the intent to buy something in some category (a gift giver, for example). Those are three fundamentally different use-cases.

This matters for almost any significant use of your digital data. Do you want to understand the quality of traffic sourced by your campaigns? You need to understand the visit intent of the sourced visitors.

Do you want to measure content effectiveness? Believe me; it’s relative to the visit intent of your visitors.

Do you want to test & personalize? That, too, is directly related to visit intent.

We tell all our clients that a best-in-class digital dashboard will have NO SITEWIDE metrics. Everything will be segmented by audience and intent.

Well guess what, the same story is pretty much true for your analysts! Four times out of five, they don’t need to see all the detail inside a session. What they need to understand is what the session is about and the extent to which it was successful.

But this brings us back to the problem I first attacked - creating visit level aggregations. Because those aggregations are, fundamentally, an answer to what a visit was about. And here’s where things get sticky. If you just leave all your data in the raw event form, then every time an analyst wants to use the data, they’ll have to create some way to getting meaning out of that detail. You can bet they’ll all find different – and probably very simple and unsophisticated – approaches to doing this.

Let me put the problem in a different context.

Suppose you’ve got a database of fitness readings from one of the many wearable devices now extant for that purpose. At the detail level, those readings are not dissimilar to digital data. There’s a timestamp, a geo-location and perhaps a set of physiological readings. In this form, the data is very hard to use. You have to use logic to infer what activity the wearer was engaged in. Was the person walking, riding an elevator, flying in an airplane, stuck in traffic on a freeway, or cooking in the kitchen? Each of these types of activities will be defined by some pattern of motion and physiological readings.

You could leave all the activity in its raw form and then every time an analyst queried the data they could choose a different pattern recognition scheme to delineate some set of patterns.

You could do that. But it would be really, really stupid.

The right approach, of course, is to create a rich analysis that matches behavioral signatures to real world activities and then tags the data so that every analysis uses a shared understanding of what each pattern means. Indeed, you probably wouldn’t just tag the raw data, you’d create what amount to session-level aggregations. Those aggregations, in turn, would be the raw detail data that an analyst would use to create patterns that classify who somebody is and how they exercise.

That’s exactly the way it works in digital data too.

The types of aggregations I suggested in my previous post on session level are far from ideal. They used content categorization and basic counting and summing techniques to create, in my view, very lumpy and imprecise descriptions of visit intent.

When we create a real two-tiered segmentation, we generally build something much more sophisticated and data driven. First, we take a large sample of visitors, survey them, investigate visit intent thoroughly, and tie that to their visit behavior. We then use this to create a statistical model of the behavioral signatures for each type of visit we want to track. These behavioral signatures are then used to score every single web visit and assign a visit intent.

Built off the detail data, this segmentation model will include content categorization (similar to what I described in my data model post but not constrained by the necessity to restrict the columns), but it will take advantage of a much wider variety of classifications. We’d typically use EVERY available taxonomy and classification to at least test to see if they are predictive. What’s more, we’d typically include consumption information (how long someone spent inside every classification) and, at least, some basic level sequencing information about when things happened in the visit.

Applying this type of model is far more complex than traditional ETL counting and summing. However, it's an essential part of building a really good visit-level detail file. Having a single “best-guess” model of visit intent will make a huge difference in the usability of the data. Not only will it make interpretation of the data vastly easier, it will standardize and dramatically improve the overall quality of that interpretation. This is an analysis you can redo, refine and improve constantly – and all that work will pay off across countless queries and uses of the data. With this form of statistical ETL to create a visit intent description as part of your visit-level detail file, you can put a tremendous amount of effort and sophistication into that core intent model and then you can leverage that effort in every single query you run on the subsequent session-level file.

As I hope I’ve made clear, similar Statistical ETL processes are likely to be essential in ANY true big data application. The very nature of big data is that the detail data needs to be patterned before it can be used effectively. In the vast majority of cases, the patterning should be a foundation for subsequent analysis, not a part of every query you make on the data.

So when I say digital is a paradigm case of big data, I really mean it!

Related articles

The Big Data Analytics Warehouse and Digital Data Models

Big Data Analytics and Digital Detail Data

Taking the Fast-Boat to China - Using Analytics to Improve ePerformance in China

Not too long ago I enjoyed a fascinating conversation over lunch with Loren Hadley on our team here at EY around some work he's been doing analyzing Web performance in China. This seemed highly germane to many of our clients, so I asked Loren if he'd be willing to redo the conversation in written form. I wrote up the questions, Loren provided the insight...

 

You’ve been working on some analytics projects that are China focused and part of that has been comparing China to other regions and countries – not just the U.S…what’s your sense of how different China is? Is China's digital experience unique and if so, to what extent when compared to other major Global markets?

While every country’s Internet use patterns are unique to some extent, there are a lot of similarities across the Western world, including more western oriented Asian economies like Japan.  The US has been the driver and largest market and has basically pulled the rest of the world along in its wake. 

That’s where China becomes very interesting. In 2014, roughly 41% of China’s population was on-line amounting to roughly 618M Internet users compared to roughly 280M US Internet users which accounts for about 79% of the US population. Chinese internet consumers have grown up on locally optimized sites and logistics systems which differ from the western model.   They have lots of options that work for them, so it seem likely to me that digital channels will continue to bend to meet the Chinese consumer’s needs rather than successfully reshaping the Chinese consumer into the Western mold.

When you think about something like a “China” analysis, where do you start and what do you focus on?

Like any analysis, there are business questions being asked and stakeholders with specific interests.  That provides an anchor point for the analysis.  After that we started at the top-level metrics, segmented as many ways as we could think of and looked for anything interesting or anything we couldn’t explain with our current knowledge of the site.  We compared what we were finding to other countries, which often led to broader based research to understand the Chinese norms  

Our work is part of an ongoing optimization program so we’re looking for the lowest hanging fruit. Every iteration should see us diving deeper and building off of what we’ve learned in previous analysis. 

 

We often try to think of site analytics as fundamentally use-case driven. Is your sense that the use-cases of visitors in China wouldn’t be much different than here in the US but the success and details of what happens inside that use-case are going to be different OR do you think some completely different kind of use-cases emerge? 

While the opportunity certainly exists for new use cases to evolve, our experience has been that the Chinese consumer is similar to Western consumers at a fundamental level.  They are trying to find information, shop or be entertained.  The exact details of the checkout experience or funny cat video may differ, but the basic need is the same.   

We’ve found some differences in how navigation features are used, but it isn’t clear yet to what extent this is something fundamentally different about Chinese consumers’ use of the web, versus opportunities for localization and optimization.

Network speeds in China are much slower on average, yet the expectation is that sites will offer the rich experience we are used to.  As a result there are a lot of challenges delivering a winning user experience while keeping page load times at a bearable level.

Consumer expectations around payment and shipping of eCommerce goods are much different than they are in the US.   Knock-off products, grey market and fraudulent sites are reportedly more prevalent in China and “caveat emptor” colors user behavior.  Chinese consumers appear to shop more readily at trusted sites and seem to seek out trust signals like reviews before committing to new shopping experience.  Most commerce tends to be through third party payment systems rather than providing payment information directly to the merchant and many Chinese payment options include an escrow feature that reduces the potential for fraud.  COD deliveries are also common.

You talk about how different payment systems and usage propensities are (and that’s not just in China) from the U.S. But what are the analytics and funnel implications to that and what sort of measurement challenges does widespread use of 3^rd party payment systems create?

When almost every transaction is completed on a third party site, there’s a big black box in the middle of the funnel.  The user is handed over to another site with a different experience and the tracking path ends abruptly before the order is truly completed.  In our experience in China almost all orders used a 3^rd party payment system which creates a lot of uncertainty around the key conversion metrics.  Our situation found that the web tracking accounted for more orders than the transaction records which counted fully paid orders.  That’s not a big surprise for anyone that has tracked fallout in third party payment systems, but it complicates analysis and optimization.  As a work-around, we found with large enough numbers and some historical data, we were able to apply experience based modifiers that got us fairly close to the real-paid numbers.

 

What’s the biggest gotcha when it comes to doing analytics on a China-based digital property?

As analyst’s we’re used to figuring things out, recognizing patterns we’ve seen before which help us to make sense of the data.  But in our Chinese analysis we often found patterns that left us scratching our heads. 

We spent a lot of time hypothesizing then looking for ways to test those hypotheses and validate our assumptions.  We are fortunate in that we had an in-country team that could provide some background, and the assistance of a culturally and linguistically bi-lingual analyst that could help us understand the nuances of localization as well as the general mindset of the Chinese web users.  The cultural part was really valuable.  Although the insights were often anecdotal, it helped us develop hypothesis that as US based, non-Chinese analysts we would never have landed on by ourselves. 

 

What are the implications for a shared global architecture in digital eCommerce given what you’ve generally seen when you looked at China behavior patterns? What’s your general sense of the right balance between standardization and localization when it comes to China?
 
Shared global architecture often seems to mean taking the optimized Western experience we are most familiar with then providing some degree of localization while maintaining underlying technology and structure.  This makes a lot of sense from a management and development perspective and in most cases works fine. 

As I mentioned earlier, China is not really in the western orbit but has enough mass that it can develop in its own way.  My sense is that western companies will need to decide the level of effort and expected ROI as they plan their China strategies.  If they want to add an additional revenue stream by skimming the China market, then localization for language will probably suffice, and there is an interest in western products and culture.  If however the company wants to become a major brand in China and really harness the potential of China’s massive Internet market, they may need to go much further with developing a unique Chinese oriented experience and content. 

 

Are there other differences people should expect when they think about analyzing eCommerce in China?

Well, we’ve already covered a lot of ground, but a lot of the surprises were in the details. 

As sites advance, support for older browsers often suffers and that can degrade both experience and performance. We’ve seen that the market share for old IE browsers in China as a whole is much, much higher than we are used to seeing.  That is dropping over time, but our hypothesis is that there are numbers of newer Internet users that haven’t bother to or don’t realize you can and should update browsers.     

Another area is mobile use.  Mobile penetration in China is high and text messaging platforms like WeChat (600M+ registered users) have a huge following.  However, desktop platforms still deliver the preponderance of traffic.  Mobile use in China is integrated into the on-line ecosystem and social media, but it seems to revolve around SMS rather than the smart phone usage in countries like South Korea and Japan.  Figuring out how to leverage mobile in China may be a key to success for some companies.  

I guess the short answer is that there will be a learning curve for non-Chinese analysts.  Understanding what is unique to the site we were analyzing and what is a systemic factor in China took more time than usual.

 

Is it your sense one of the challenges is a wider variety of existing UI – less standardization of experience than is common in the US – you talk about how users are newer and as I think back to the early 2000s there was much less conformity in UI here in the U.S. which made navigating EVERY website harder. Same in China right now?

I’m not sure what we would find looking across the entire spectrum of Chinese sites, but looking at major Chinese retailers and corporate sites we actually found a fair degree of consistency.  It appears that the key players that have defined the mainstream Chinese market have looked at best practices and designed systems that work in the Chinese environment. 

 

How challenging is it work with the analytics when so much of the key stuff is Chinese – and even looking at the source content is hard? What are some of the ways a team can solve those problems?

If the tagging for your project is well implemented and standardized in English, then the actual in-analytics experience isn’t so different. Without that, an analytics project would be nearly unmanageable for a non-Mandarin speaker.  As it is we rely heavily on our Mandarin speaking teammates to help us sort out the detail and work through the customer experience.  On a micro-level on-line translation tools are pretty handy.  The translations are often clumsy, but you can figure out that the big button says roughly “Buy Now!’ and the little one says “Log-in”.  At a macro-level it is much harder since you are puzzling out the details so it is more difficult to see the big picture issues.  Again, having a Mandarin speaking compatriot lets us step back, ask questions and vicariously develop an understanding of the page.

 

I suppose that's one part of the content that we don’t always think about - the language itself. A lot of global companies just do straight translations of their site and that has potential pitfalls. What sorts of things have you found around that and how did analytics help identify potential problem areas in translation?

We often find instances where improvements in translation offer easy opportunities to improve aspects of the user experience.  Sometimes they are the same sorts of issues we deal with in English: validation prompts may be more confusing than helpful to consumer success.  At other times they are due to more nuanced aspects of the language.  For example, a term that was used successfully on the English language site is presented in a technically correct translation.  However, it has a more formal meaning in Mandarin that doesn’t seem to fully reflect the brand's intent.  This provides a great opportunity for testing alternatives that will resonate more with the Chinese consumer.

We’ve all read package inserts or instructions that are translated to English from another language.  It may help you get the product put together, but it doesn’t always generate confidence or reflect positively on the quality of the product.  This highlights that a technical translation may suffice, but it probably won’t convey the same meaning as copy written for the audience by a native speaker.  Companies will need to weigh the cost against benefit of a translation program versus the recreation of content in a linguistic and cultural way that resonates with the Chinese consumer.  The later is a much greater undertaking, but it may pay real dividends for sites doing business in China. 

 

You touched on some specific challenges around China with payments, forms and fraud – does any of this impact tagging requirements for digital?

Use of alternate tools that don’t rely on cookies and beacons can help track exits and returns to the funnel from the 3^rd party sites.   This may offer deeper insights into what was happening, but it can complicate analysis when you have part of the story in one data set and part in another.   

Having robust form tracking is going to be important for anyone trying to optimize transactional or registration based performance.  As you find the balance between the Chinese consumer’s expectations and the demands of your internal systems, knowing exactly what is and isn’t working will be key.

Tracking and tools that will allow you to determine the optimal mix of 3^rd party partners around payment and fulfillment will likely be of critical importance.  It will help you home in on the partners that are most used by your audience and potentially weed out the ones that are primarily cluttering the experience.

 

Last thoughts on analytics targeted to China-based sites? Anything we haven’t covered?

If you will be responsible for analyzing a Chinese site, get involved as soon as possible.  As with any project, the quality of the implementation will dictate what you’ll be able to look at and this will be exacerbated by the language issues and predominance of 3^rd party tracking.

Develop relationships with in-country team-members and/or make sure your team has access to people who are fluent in Mandarin as well as current Chinese culture.

Have fun with it.  It will likely be a fascinating and challenging analysis and will become increasingly relevant to global businesses as China’s economy and Internet base continue to grow.