Tracking IP Addresses Across Microsoft UXC

Nick Rawitsch
1/3/2018

Background

The recent addition of browser fingerprinting to Microsoft User Research’s participant recruitment process has created an opportunity to study how monetary incentivization leads participants to overstate their use of Microsoft products to qualify for compensation.

Goal

The goal of this study is to determine what percentage of survey respondents overstate their qualifications to participate in Microsoft User Research, and what influences them to do so.

Research Questions

Who is our user?
Who is the desired user?
What trends do we see?
If the desired user does not match, what can we do about it?
What are others in our industry doing about it?
How has this research design influenced our products?

Methodology

To answer the question of who we are recruiting, versus who is desired, Research will examine 671 responses to a Qualtrics survey designed to qualify respondents to participate in Microsoft User Research studies.

We will then conduct a literature review to determine what tools are in use to combat deceit in survey data in peer reviewed scientific journals. Finally, we will conduct a competitive analysis of research practices at Amazon, Google and Apple.

Who is our user?

When researchers posed the question, “Which Microsoft products do you use at least weekly,” 0/671 complete survey responses reported non usage of Microsoft products, despite being given the option to choose “None of the Above.”

Within product segments, 68% of participants reported they have Windows 10 – yet according to external market research analysts, Windows 10 users should only make up .07% of the total Windows market share.

In a randomly selected sample of computer users, only .63% should use Windows 10 when adjusted to include competitor’s market share, leaving us with the question, “Does every single person taking our survey truly use Microsoft products, and could our recruiting efforts possibly result in over 1000 times as many Window 10 users as you would expect to find in a random sample?” We wanted to get a closer look.

Within this segment of self-identified Windows 10 users, a significant proportion said they used only Windows 10. If their self-reported software usage were true, how could users whose computers came loaded with many of the other products given as options, not select others from the list?

In contrast to the self-identified users, those users whose browser fingerprint automatically collected Windows 10 as their operating system, universally marked more than one product. As our collection of browser fingerprint information is universalized, we expect the user’s tendency toward self-interest to become even more clear.

Who is our desired user?

Many of Microsoft’s products cater to a wide range of customers, but there are others that would be completely inoperable to a non-user. Our user anticipates this expertise gap, and in self-interest avoids studies that seem too technical.

For Azure, a developer platform, there was only one questionable response from a “Face Painter” – compared to hundreds of unlikely self-identifications for consumer-oriented and widely understood Microsoft products like Windows 10 and Bing.

This might help to explain why these trends have not been previously detected, and demonstrates that our users are smart enough to know not to identify themselves for a task they would not be able to complete. Our user simply does not exaggerate product usage for products that could expose them. Often, they even know what profile we are searching for in advance – allowing them to qualify while still avoiding the embarrassment of being caught.

These non-users, that are totally undetectable to the researcher, become the desired user out of utility of the recruiter and the expectation of the researcher that they have been properly vetted. In reality, the majority of our studies depend on unverified, self-reported data, delivered without the oversight of trained UX professionals and designed around a business model that incentives volume over precision.

The result is that nonusers constantly slip through the cracks due to research designs that conveniently, but incorrectly assume that all users are honest. Clearly, they are not.

What trends do we see?

Self-reported data varies drastically from what was automatically collected
Users tend to overstate the number of products they use, except when they anticipate that the product would be unusable based on their skill level.
Fewer items does not equate to honest behavior. Guilt and self awareness arising from the selection of products they don’t use drive users to report fewer items.
To minimize this guilt, they randomly select items that seem easy to use
Nobody wants to disqualify themselves. If the user has taken the time to do the survey, they want to be selected. This desire, coupled with a sense of anonymity, drives users to mislead researchers wherever they can get away with it
A user will never willingly supply information that they think would disqualify them

If the desired user doesn’t match, what can we do about it?

Use automatically collected data as a substitute to self-reported data wherever possible
Mix in questions that give users an opportunity to contradict themselves as a flag for bad data
Force responses, disable the back button, and prevent users from completing the survey twice by requiring a sign-in with a unique identifier
Substitute cash gratuity with another incentive to combat selection bias by diversifying rewards

What are our competitors doing about it?

Microsoft’s closest competitor, Amazon does not offer any cash or cash gift card incentives on its user research signup page. To assure their incentives resonate only with true users, they only advertise Amazon gift cards as gratuity. For higher security projects, Amazon takes this a step further by withholding the company’s identity throughout the entire user research process.

If Amazon’s participants are contacted through a third party, they are only told that a market research firm believes they might be a good match for a user research study. Researchers participants interact with even give pseudonyms so they cannot be found by participants online. Most importantly, the name of the company and product in development are withheld.

Apple also hires outside market research firms, and claims to “brand our research whenever possible.” Their privacy policy states that personal customer data can be used for recruitment into usability studies, even if you opt out of email. Apple makes no public solicitation for research participants, and judging from the information they provide, only ever draws their participants from customer records.

Google is clearly aware of the hazard that comes with cash incentives. The gift cards they promote their user research program with show amounts of $10-$25, and do not even reveal at what retailer they can be redeemed in fear this could influence the results. They also offer compensation in the form of charitable contributions – which safeguards their research from being too attractive to grat-seekers.

The back buttons in their survey are disabled, they ask questions twice to confirm consistency, and verify your responses against a record of your usage of google products, deanonymizing your responses by using your google account sign in information.

Status-Quo

Unlike any of its direct competitors, Microsoft announces the desired user, participant gratuity, and amount paid per study on the first page of its user research site. Their individual surveys tout the monetary rewards of participation, which often far exceed what a working person would make in the same time at their job.

The result is a group of basically anonymous testers, motivated by gratuity, undeterred by threat of verification, guided by advertising of a specific study or product, that happen to be available on a workday.

In no simpler terms, Microsoft gives users all the information needed to cheat and all of the incentive to do so. This creates a business problem when recruiting non-users, but is at every other moment a profitable, but flawed methodology for vendors who get paid per qualified participant.

Therefore any question that could disqualify participants is avoided by vendors. Verification only takes place when a researcher requests it, contradictory information is taken at face value, and because no customer data is available to researchers to cross reference, the user ends up disproportionately represented by people in low income demographics, not representative of Microsoft’s target market.

In either case, whether or not the participant is a true user is an afterthought, eclipsed by the logistics and business model of recruiting. Changes are on the way, and Microsoft will soon use user’s live ID as a signup verification – but this alone will not change the profile of the users being recruited, as all of our competitors have learned.

This behavior has both a reputational and monetary cost, the former the risk of losing business to UX research specialized companies, and the latter resulting from anytime recruiters are asked to find non-users, that will not self-identify as such (cost quantified per user below.)

How has this research design affected our products?

Overtime, design evolves to accommodate its users– which has made our software more usable for non-users and novices, but less focused on the needs of more experienced users. Take the evolution of Word for example:

In its present iteration, Word has been configured not to hide any functionality. Every option is available within a single click on the top menu. Compared to past iterations where users selected what toolbars they needed themselves.

The trade-off is a steeper learning curve for users of past iterations, and a disincentive for current users to upgrade, in exchange for more accessibility for beginners. Were it not entirely accidental, the choice could be strategically sound. Office has also lowered the barriers to entry for new users by jumping to the SaaS model, and preserved some of the revenue stream despite the attack of Google Docs.

Interestingly enough, Google Docs has capitalized on Microsoft’s focus on making the old new again, maintaining the original Microsoft Word design as its template for the Docs interface.

Their now decade old offer of free access to Docs has completely destabilized the value proposition of Word – which saw a 62% decrease in subscriptions for 2016, following the initial success of conversion to a SaaS model, our explanation being, non-users make decent test subjects, but terrible customers.

Conclusion

Microsoft maintains a powerful oligopoly on its Windows OS, with a market share near 90% - but the age of desktop computers are numbered. If their user research pipeline is any indication of the company’s R&D focus, Microsoft will continue to focus on existing products, packaging and repackaging them as upgrades at the lowest possible cost, all the while investing their cash in acquisitions.

As the revenue streams for these software products dry up, cost cutting becomes a greater priority than the development pipeline – and existing users find alternatives to products designed for ROI not UX.

—
The Business Case for Response Verification

Executive Summary
300 calls to schedule a single tech cautious profile, and the impossible task of finding non-users, quantified.

Assumptions
$16 hourly caller wage
1 minute per call (one 10 minute call per 10 calls)
$100 revenue per successful recruit
450 minutes per shift ( 8 hours minus two 15 minute breaks mandated by law)
Benefits = 1/3 of total wage
36 square feet of office per employee
$39.59 per square foot yearly commercial rent

Cost Benefit Analysis