When I joined Meta in 2025, the developer tools org had almost no dedicated UXR coverage. A Workplace post from the performance team had explicitly called for research — not as a courtesy, but because product decisions were being made without it. Both new and experienced creators and developers use profiling and simulation tools to build, test, and optimize VR, MR, and 2D applications. Key pain points had been named anecdotally: inconsistent terminology across tools, steep learning curves, no actionable guidance, simulators that couldn’t keep pace with what creators actually needed. But no one had synthesized what was known, mapped the competitive landscape, or run primary research in this space at Meta before.

XR Simulator 2.0 was launching in December 2025. A major hardware generation was coming. The question wasn’t whether these tools mattered — they did. The question was which of Meta’s existing positions were worth building on, which were liabilities, and what the next hardware generation would expose before the team had time to fix it. I had weeks, not months.

Two developer personas anchored the research across all three studies. They weren’t demographic categories. They were different relationships to the tools.

The distinction matters because the tool that works for Persona 01 (RenderDoc, Perfetto) is the tool that fails Persona 02, and vice versa. A research question framed at the level of "how do developers use profiling tools" misses this entirely.

Three gaps, each with a compounding cost if left unaddressed before the next hardware generation.

Performance Analyzer had a return on investment sitting unclaimed. Only 3% of organizations used it before submission, but those organizations saw 35% higher app approval rates. A tool with that effect size, with 3% penetration, is a growth lever the team hadn’t pulled. The barrier wasn’t the tool itself. It was onboarding and discoverability.

Meta’s competitive position in MR was the right bet, but the team needed research to confirm it. Apple led on polish, Android led on flexibility. Meta led on MR and multiplayer simulation, carrying the most usability debt of the three platforms. The research needed to say: own that position, don’t chase parity on someone else’s terms.

Hand simulation was about to become a bottleneck no one had surfaced yet. Six of seven usability participants independently asked for the same thing: custom, recordable, uploadable hand gestures. Apps were already being built that required them. If the simulator didn’t support them before the next hardware generation shipped, porting would stall. The research named it early enough to act on it.

Three studies, each calibrated to a different knowledge gap. AI was the methodological through-line, not a convenience, but as a deliberate choice about where researcher time is most valuable when the domain has no prior coverage and the clock is running.

The knowledge base for XR performance tools was fragmented across 15+ internal Workplace posts and research reports with no prior synthesis (the Workplace call-for-research post had flagged this explicitly). Sequential reading would have taken weeks. I uploaded the entire corpus to NotebookLM and ran structured queries to surface patterns, contradictions, and gaps across sources simultaneously. The synthesis identified the 24-point satisfaction gap, the Performance Analyzer 3%/35% finding, the two-persona framework, and the four key challenges (tool fragmentation, misaligned metrics, insufficient actionable guidance, and onboarding barriers) that shaped the research agenda for everything that followed.

The cross-functional readout was built using AI-assisted infographic generation: translating dense quantitative findings into a visual artifact the design and PM teams could work from directly. In a domain where most stakeholders are engineers, making the research readable is its own methodological problem.

Developer communities for XR tools are active and technically specific: Reddit threads on simulator bugs, YouTube tutorials surfacing workflow pain points, Meta Community forum posts on feature requests. Manually reading and synthesizing that volume of unstructured feedback would have taken weeks. I ran two structured prompts through Vocal: one on simulation tool competitive analysis across Meta, Apple, and Android; one specifically on gaze and hand input issues across platforms, drawing on social mentions, Quest Store reviews, bug reports, YouTube, and customer service tickets across a seven-week window.

This produced the competitive positioning finding (Meta leads on MR and multiplayer, carries the most usability debt), the XID ambivalence finding (developers building for next-gen hardware are uncertain about the audience and monetization path, which directly affects how much they invest in tool mastery), and the early signal on hand tracking as a differentiator.

Seven moderated usability sessions on XR Simulator 2.0 in the week of launch. Participants were recruited across the Ainsley and Wren personas: established Unity developers and Android developers adapting mobile apps for MR — to test whether the reliability improvements had landed and what remained blocked. I used NotebookLM to synthesize session notes in parallel during data collection, enabling pattern identification before the final session was complete. The full report was delivered within days of closing. In a launch-week context where the team needed findings before the post-launch sprint was scoped, that turnaround was the point.

A 24-point satisfaction gap across the tool suite, with adoption and reliability problems concentrated in the tools developers need most as they scale.

Performance is continuous, not a one-time fix. Creators and developers address issues both proactively and reactively throughout the build lifecycle, averaging 20 to 30 minutes per profiling session excluding setup.

Tool satisfaction and reliability vary widely. OVR Metrics leads at 77% CSAT and 85% perceived reliability. Perfetto sits at 53%. RenderDoc %Bad is at 40% against a 20% EoY target, signaling significant UX debt.

Developer experience determines tool needs. New and generalist creators and developers rely on lightweight tools like OVR Metrics. Experienced and specialized creators and developers need RenderDoc and Perfetto, but these carry the highest onboarding barriers.

Performance Analyzer drives app approval rates. Only 3% of organizations used it before submission, yet those organizations saw 35% higher approval rates, signaling strong untapped value.

The social listening surfaced something the product questions hadn’t anticipated: a layer of ambivalence about building for XR at all. Developers building for next-generation hardware weren’t just uncertain about which tools to use. They were uncertain about whether the audience would be there, whether the monetization path was viable, and whether the investment in tool mastery would pay off. This matters for the research because it reframes the adoption problem. Low adoption of Performance Analyzer or XR Simulator isn’t just a discoverability or UX problem. It’s partly a motivation problem. Developers who aren’t confident in the platform aren’t going to invest in learning tools they might not need.

The implication for the product team: fixing onboarding and documentation is necessary but not sufficient. The tools need to visibly reduce risk — by making it easier to hit quality gates, easier to port existing apps, easier to test scenarios that would otherwise require multiple headsets. Each of those is an argument for building for XR, not just an argument for using a specific tool.

Meta leads on MR and multiplayer simulation, but adoption lags and competitors are closing the gap. The strategic position is clear; the execution gap is onboarding and discoverability.

Simulation tools appeal to indie studios and 3D creators, but mobile developers remain hard to convert. 47% of game developers globally focus on 3D development, making them well-positioned to build for XR. Indie studios and solo creators are a key segment, motivated by audience growth and new creative possibilities, but hard to attract without strong monetization and discoverability support.

XR Simulator drives publish rates but adoption lags. XRSim has a +0.1% effect on app publishing but only 40% adoption within 6 months, below Link’s 50%. WATM@14 sits at 18.27% against a 24% target.

Spatial Simulator is a major relief for Android creators and developers. Early dogfooding feedback described dramatically faster iteration speeds. No more don/doff. No dead batteries. Development anywhere on the go.

Competitors set a baseline expectation. Apple leads on polish. Android leads on flexibility. Meta leads on MR and multiplayer testing, but faces the most usability challenges of the three.

The social listening component was the methodological move that made the competitive analysis possible at the speed it needed to happen. Using Vocal with structured prompts, I ran targeted queries by platform, by tool, by developer type, and synthesized findings across thousands of posts into a competitive signal the team could act on. Three platforms, distinct competitive positions, Meta’s MR advantage as the strategic bet. All of this came directly from that synthesis.

The competitive data told us where Meta stood. The developer interview data told us how developers actually used simulation tools inside the build process. These were different questions, and together they shaped the strategic recommendation: own the Build phase, because that is where simulation earns its place and where Meta has the strongest differentiated position.

Simulation tools appeal to indie studios and 3D creators, but mobile developers remain hard to convert. 47% of game developers globally focus on 3D development, making them well-positioned to build for XR. Indie studios and solo creators are a key segment, motivated by audience growth and new creative possibilities, but hard to attract without strong monetization and discoverability support.

XR Simulator drives publish rates but adoption lags. XRSim has a +0.1% effect on app publishing but only 40% adoption within 6 months, below Link’s 50%. WATM@14 sits at 18.27% against a 24% target.

Spatial Simulator is a major relief for Android creators and developers. Early dogfooding feedback described dramatically faster iteration speeds. No more don/doff. No dead batteries. Development anywhere on the go.

Competitors set a baseline expectation. Apple leads on polish. Android leads on flexibility. Meta leads on MR and multiplayer testing, but faces the most usability challenges of the three.

7 of 7 participants praised the new UI. The standalone app and clearer panel organization were described as less clunky, more intuitive, and accessible enough that "anybody can use this." 4 of 7 noted improved stability and fewer crashes.

Versioning and discoverability remain blockers. 3 of 7 participants were confused by two coexisting versions (v81 and v83) and the split between Unity package manager and standalone website installation, creating friction at the moment of adoption.

Gaze and eye tracking readiness is low. Most creators and developers had no hands-on experience with gaze inputs due to Quest 3/3S hardware limitations and EU privacy concerns, creating uncertainty about next-generation hardware readiness.

Hand simulation is the top feature request. 6 of 7 requested custom, recordable, and uploadable hand gestures, critical for creators building sign language apps, escape rooms, and gesture-driven experiences. 4 of 7 needed complex multi-hand interactions their apps required but could not simulate.

Hands are the interface for next-generation XR, and 6 of 7 participants asked for the same thing in different ways: custom gestures, two-handed interactions, controller-to-hand mapping. Not edge cases. Load-bearing requirements for the apps creators were already trying to build.

Three studies, one argument: AI-assisted research can build a strategic knowledge base in a product area with no prior coverage, then validate it in the launch window.

The performance tools study defined the 2026 tooling strategy and surfaced a specific AI integration opportunity: embedding AI-driven suggestions directly into performance tools to surface root causes and actionable guidance, not just flag issues. Developers, particularly new and generalist ones, consistently hit a wall where a tool would tell them something was wrong but not what to do about it. The recommendation was to close that gap with AI. That finding was funded for the 2026 roadmap. The broader fix — standardizing terminology across the suite and rebuilding the onboarding layer — addresses a structural problem that no individual tool fix can solve.

The simulation study established MR as Meta’s strategic competitive bet and defined the H1 2026 research agenda for discoverability and adoption. The recommendation was to double down on MR and multiplayer rather than chasing Apple’s polish or Android’s cross-platform flexibility. The XID ambivalence finding added a dimension the team hadn’t framed explicitly: adoption is partly a motivation problem, not just a UX problem, and the tools need to visibly reduce risk for developers who aren’t yet confident the platform is worth the investment.

The usability study shipped 13 features and 3 bugs into the post-launch sprint and surfaced hand simulation as the top feature request, named independently by 6 of 7 participants, before it could become a porting bottleneck for the upcoming hardware generation. The recommendation was immediate: resolve the version confusion between v81 and v83, consolidate the installation path, and add hand simulation capabilities before the next hardware generation ships. The H1 2026 hand simulation roadmap had been waiting on a research signal that didn’t exist. It does now.

The methodology is the other outcome. NotebookLM, Vocal, and AI-assisted synthesis are a reusable reference model for building research coverage fast in a domain with no prior dedicated coverage, applicable to any new product area the team enters next.

When customers contact Google support, they’ve usually already failed at self-help. They want short, precise resolution: not a robot, not a script. The bar they set is high and specific: “I will find the solution from beginning to end. And if it’s not something I can help you with, I will jump through the hoops and I will explain it to someone else.” That’s what a good support experience feels like from the customer side. A committed human who owns the problem. As Google moved to scale support with generative AI, the design question wasn’t just whether the AI worked technically. It was whether the customer on the other end of the chat would still feel that.

Google’s support infrastructure was organized by channel. Voice had its own team, its own tooling, its own research agenda. So did chat, and email. A cross-channel synthesis study ran alongside the lifecycle research to ask whether that structure was serving the customer. What it surfaced was a convergence finding. Voice, chat, and email agents, working in separate teams with no visibility into each other’s work, surfaced the same anxiety independently: the AI didn’t know their customer. Different channels, different features, the same structural problem. The question the research kept raising was whether the infrastructure was built for that customer, or for the org chart.

Google was shifting its 1:1 support model from fully manual to AI-assisted to AI-supervised, with the goal of scaling support capacity without degrading customer satisfaction. The research program was designed to track whether that shift was working, and for whom: agents, customers, and the organization.

The cost of getting it wrong was threefold. For customers: a support experience that felt less human the more automated it became, eroding the trust that converts a frustrated user into a retained one. For agents: a tool that added cognitive load rather than reducing it, creating rational resistance to adoption and undermining the productivity gains the system was built to deliver. For the organization: shipping an AI layer that didn’t know the customer, then building the next generation of the platform on top of that same structural gap.

A mixed-methods program of 25 studies over two years, spanning the full product lifecycle from pre-Beta through launch, alongside a parallel cross-channel synthesis study with 17 frontline agents and 2 workforce leads across voice, chat, and email. Primary methods included prototype testing, in-depth interviews, shadowing, focus groups, surveys, NASA-TLX cognitive load assessments, chat transcript analysis, and ethnographic field research with frontline agents in Manila. The 25 studies weren’t a single research push: each one was sequenced to where the product was, what the team needed to know next, and what could move the agent experience forward.

ML researchers and engineers need to identify and debug architecture, quality, and performance issues in large models, especially for on-device deployments where conversion and optimization processes can significantly alter a model from its original state. Existing tools couldn’t render large models and had significant usability problems incompatible with modern model architecture.

The two studies were each built around a specific decision point. The first: where does visualization actually earn its place in the on-device ML lifecycle? The second, a year later: does this IDE prototype solve the fragmented-tool problem well enough to ship? Different questions, different methods, the same underlying discipline — research tied to an action, not an open brief.

Two studies, one year apart, each calibrated to a different product decision. Study 01 was foundational: a literature review of approximately 15 internal research reports followed by eight contextual inquiry interviews with on-device ML developers, conducted against a hard pre-I/O 2024 deadline. The goal was to map the actual model development workflow end-to-end before the team committed engineering resources to a direction. Study 02 was evaluative: six prototype testing sessions to answer whether the Model Explorer IDE was ready to ship, or whether the team was building confidence on a scenario that didn’t reflect how developers actually worked.

For both studies, AI-assisted synthesis compressed the analysis cycle. In a pre-launch context where findings needed to reach the team before decisions were already made, turnaround speed was part of the methodology.

Eight contextual inquiry interviews with 1P and 3P on-device ML developers (LLM and non-LLM specialists), conducted ahead of a hard pre-I/O 2024 deadline, preceded by a literature review of approximately 15 internal research reports. The goal was to map the actual model development workflow end-to-end and surface where visualization tools added load-bearing value, not just where they were nice-to-have.

The research established four core ODML stages developers follow to launch models on device: Build, Adapt, Integrate (Optimize), and Release (Deploy). Each stage has distinct tasks, evaluation targets, and tools. Developers frequently revisit earlier stages as they learn more about quality vs. performance trade-offs. An XFN alignment workshop followed the contextual inquiry, with results shared pre-I/O and a metrics dashboard monitored post-launch.

A year later, the team built a prototype Model Explorer IDE: a unified environment combining visualization, quantization, numerical analysis, and layer modification in one tool. The hypothesis was that an IDE would solve the fragmented-tool problem the first study had surfaced. Six participants experienced in quantization and performance debugging were shown a demo video of the prototype, which walked through an engineer quantizing a small image segmentation model: opening files, running quantization without and then with "numerical diff" enabled, and using the exclude-selection feature to address problematic nodes.

The ask was simple: how would this fit into your existing workflows? The answer was more complicated than the demo suggested.

Servomatic is the Google internal tool ML developers use to serve models in production. The quarterly CSAT survey had been running since 2022, started by a previous researcher. When I inherited the program in 2024, it produced a single number and a stack of free-text responses no one had time to read systematically. The number was useful as a temperature check; the responses were where the actual signal lived, but only if you could make them tractable.

The question wasn’t whether users were satisfied. They mostly were. The questions were what was driving that satisfaction, whether the drivers were stable, and whether there were structural friction points the product hadn’t addressed. And lurking in the open responses, a third question the survey couldn’t answer at all: whether a proposed change to how developers moved between dev and production environments would actually be adopted. Each question needed a different instrument.

The survey was already running. What it lacked was a way to read the open-ended responses at scale. I designed the coding schema inductively, working through responses and clustering them until the categories stabilized, rather than imposing categories from the outside. That produced 25 categories rolling up into four top-level areas: Documentation, Ease of Use, Support, and Performance. The decision to hold the schema fixed once it stabilized was deliberate: you can’t trend data across categories that shifted mid-program. Every quarter’s responses from that point could be coded and compared against prior quarters, turning a one-time report into a longitudinal signal.

Each quarter, invites were sent automatically via a Python script to active Servomatic users through internal chat, consistently producing response rates around 47%, high by survey standards, and a function of the internal distribution channel rather than anything special about the instrument. What made the responses useful wasn’t the volume. It was the schema.

When the longitudinal data surfaced a recurring undercurrent about dev-to-prod readiness that the survey couldn’t answer, I ran eight moderated qualitative interviews. The stimulus for the concept test was a technical one-pager describing the proposed API integration. Not a prototype. The fidelity matched the question: we weren’t testing whether an interface worked, we were testing whether developers would adopt the concept at all. A higher-fidelity prototype would have answered the wrong question.

With the schema in place, every quarter’s data could be read against prior quarters. CSAT held steady at around 77%, consistent across the program. That stability was itself a finding: users found Servomatic reliable and trustworthy for the common case. The product wasn’t broken.

The volatility was in the open-response category mix. Four rolled-up categories accounted for most of the movement quarter to quarter.

The 25-category schema was designed to be exhaustive at the coding layer but tractable at the reporting layer. In practice, the open-response signal consolidated into four rolling themes that behaved differently across quarters. Documentation and Ease of Use stayed consistently high, meaning these weren’t one-off complaints but structural friction points the product hadn’t addressed. Support and Performance moved more, often spiking after specific incidents. The schema let me distinguish persistent friction from incident response: two categories of problem with very different fixes.

1. 01 Documentation. The most persistent area. Editors flagged out-of-date or missing docs for edge cases like multi-label classification, turn-up/turn-down procedures, and validation errors. The consistent ask was for more usage examples they could copy and adapt.
2. 02 Ease of Use. Friction clustered around deployment and configuration: deploying to production, validating models, accessing logs, and understanding resource allocation were all consistently difficult. Users wanted clearer, more actionable error messages and faster iteration on configurations.
3. 03 Support. Issues spiked around specific incidents rather than persisting across quarters. Slow troubleshooting and limited coverage for custom binaries were recurring complaints, alongside requests for more responsive channels and on-call coverage in US-friendly timezones.
4. 04 Performance. Moved most quarter to quarter, often tied to specific outages or releases with regressions. Users wanted more proactive communication about outages and gradual rollouts to non-production environments before full deployment.

The categories also raised a question the survey itself couldn’t answer: a recurring undercurrent in the open responses suggested that production readiness wasn’t being considered early enough in the development cycle, creating costly rework downstream.

I ran a targeted qualitative study in April 2024 to answer the question the survey couldn’t: if and when production readiness gets considered in the development cycle, and whether developers experience downstream losses because they don’t factor it in early enough. Eight 45 to 60-minute moderated interviews with XManager and Servolab users. Half the session was about current workflows and pain points; the other half was concept testing against the proposed API integration, using a technical one-pager as the stimulus.

Dev-to-prod is structurally siloed. Experiments are run to find the best model for the project; that model is then handed off to a different team or person to handle production readiness. The skillsets aren’t the same. The handoff is where time, context, and quality decisions get lost.

Participants were receptive to the proposed API integration in principle, but raised practical concerns about how it would fit existing org structures and developer skill sets. The proposal was technically sound. The adoption barrier was organizational: the integration would require cross-team coordination that didn’t currently exist.

The XID finding made this concrete. The integration proposed a unified experiment ID shared between XManager and Servolab. Participants had no strong preference either way. It wasn’t opposition. They didn’t understand what benefit it would bring. That ambivalence was the signal: if developers couldn’t articulate why the unified ID would help them, the integration hadn’t cleared the bar of solving a problem they actually felt. No product-market fit for the dev-to-prod simultaneous experimentation concept as scoped.

Turned an inherited CSAT program into a research engine: the schema as the bridge between quant breadth and qual depth.

The schema made 1.5 years of quarterly data comparable for the first time, surfaced Documentation and Ease of Use as structural friction points rather than noise, and gave the product team a signal they could act on rather than a number they could only watch. The 2024 roadmap was directly informed by the category trends.

The qualitative study produced the more consequential finding. Testing a one-pager for a proposed API integration rather than a prototype was the right call: the question wasn’t whether the interface worked, it was whether the concept would be adopted. It wouldn’t. The adoption barrier was organizational, not technical, and a prototype would have answered a different question. The no-product-market-fit finding saved two quarters of engineering work and redirected capacity to higher-confidence bets. This is the same outcome as the Model Explorer v2 study: stopping a launch before it ships against unvalidated assumptions is a recurring and undervalued form of research impact.

When I joined the product area in 2021, research operated reactively. Designers filed requests, research triaged them, studies landed weeks later, and insights were filed in decks only the research team knew how to find. As the team grew and AI product work accelerated, the gap between when a designer needed a signal and when research could deliver one widened. Research was becoming the bottleneck it was supposed to relieve.

The program ran across three distinct user populations, each with different recruiting logistics, session protocols, and legal review requirements. Frontline support agents operating AI-assisted tooling in live customer interactions. The customers those agents served. And internal operations staff, including QA auditors, team leads, and subject matter experts who shaped how the product got used in practice. Standardizing the recruiting pipeline across these populations was one of the quieter high-leverage moves in the program. It was what made weekly cadence possible across all three user types without forcing designers to reinvent the operational work for each study.

The obvious fix was hiring more researchers. The better fix was recognizing that not every research question required a researcher. Tactical concept tests and early usability sessions could be run by designers themselves, given the right scaffolding. What they couldn’t run on their own: participant recruiting, legal compliance, methodological judgment on when their question was outside tactical scope, and the synthesis discipline that turns sessions into institutional knowledge. Those were the things a researcher could own at scale if everything else was self-service.

This program was built before GenAI was permitted for UXR work, so every piece of it was manual: the facilitation, the synthesis, the templates, the debriefs. The constraint mattered. It meant every scaffolding decision had to earn its place. If a template didn’t save a designer real time, it was dead weight. That discipline shaped the program more than the eventual tooling would have.

I treated the program itself as a research problem. Before designing the scaffolding, I ran structured interviews with cross-functional stakeholders: designers, PMs, program managers, and design leadership. Four questions shaped the intake: what did they want the program to look like, how would they know it was working, how would insights flow across functions, and what hesitations did they have about participating in or relying on the program? The answers shaped every subsequent design decision.

The operating model that came out of that research: weekly self-service sessions, run by designers against pre-built templates, with me as coach and quality gate. I handled participant recruiting, legal review, and methodological consult. Designers handled facilitation, note-taking, and synthesis with template support. A 15-minute cross-functional debrief ritual converted sessions into decisions and surfaced duplicate work before teams accidentally re-tested the same question.

Two existing literatures grounded the design. Teresa Torres’s work on continuous discovery (Continuous Discovery Habits, 2021) argued that weekly customer touchpoints should be standard product team practice rather than a research-team privilege, which shaped how I positioned cadence and cross-functional participation as non-negotiable. Nielsen’s small-sample usability research (Nielsen and Landauer 1993; Nielsen 2000) established the quantitative floor: five participants surface roughly 85% of usability issues in a given round, and even three participants surface the majority of findable issues. For the tactical and directional questions the program was designed to answer, those thresholds were sufficient to iterate confidently. The small-N design wasn’t a shortcut. It was a principled match between sample size and question type, which let designers run weekly rather than quarterly without trading rigor for speed.

To keep the program honest, I ran two complementary measurement instruments. Qualitative stakeholder interviews quarterly throughout the two-year run, which surfaced friction that hadn’t yet shown up as a complaint. And a satisfaction and perceived-efficiency survey sent to all program participants every six months, producing four comparable data points across the program’s lifetime. The survey measured satisfaction on a percentage-agree scale, asked whether designers felt their iteration speed had actually improved, whether they saw the program as replacing or supplementing researcher-led work, and how confident they were in the synthesis coming out of their own sessions. It also included one open-ended question: “What could be improved about the Continuous Discovery process?” Each round’s free-text responses went into the next cycle of program iteration, which is how most of the six insights in Figure 01 originated. Running the survey twice a year rather than once gave me a trend instead of a snapshot, and made it possible to see when program changes landed (Q3 2022 peak, after the v2 template library). The program that existed in Q3 2023 was meaningfully different from the one I launched in Q3 2021. Six of the iterations that got us there came from a single cycle of stakeholder research and are documented below.

The most consequential finding from the stakeholder research: designers and PMs thought the program was for tactical concept and usability testing at the end of the design process. That was how I’d initially scoped it too. But the interviews surfaced a missed opportunity. Continuous discovery could run across the full product lifecycle, not just before launch. The question wasn’t whether to test an existing design; the question was what phase of discovery the design was in.

The three-phase framework came out of that reframe. Each phase calibrates the question, the method, and the expected outcome to where the product actually is, not to what the program had conventionally been used for.

The single highest-leverage artifact was the v2 research template. It was a copy-ready Google Doc with sections for research plan, interview guide, note-taking, post-session debrief, and a synthesis email. Each section had prose guidance explaining what belonged there and why, with worked examples. The template let a designer walk from research plan to stakeholder-ready synthesis in a single document without needing to invent the structure.

The post-session debrief section was the hinge. It asked for four specific observations per session: pain points, surprise findings, least surprising findings, and key insights. The distinction between “surprise” and “least surprising” mattered. It forced designers to separate what they’d learned from what they’d already suspected, which kept the research honest and made the synthesis durable. A debrief that only listed confirmations wasn’t discovery; it was validation. The template surfaced that difference without me having to name it in every session.

This quote was the signal I watched for. Not “the research was insightful” but “I wouldn’t be able to iterate so quickly.” The program’s purpose wasn’t to produce more research. It was to shorten the loop between a designer having a question and having an answer they could act on. When designers described their own iteration speed, the program was working. When they described the research, it wasn’t quite landing.

The program was designed for tactical and directional questions. It wasn’t designed for deep foundational research, and trying to use it that way produced thin findings. When teams asked questions that needed more than a 60-minute session, more than five participants, or more than a single designer’s synthesis capacity, I pulled the question out of the program and ran it as dedicated research. Protecting that distinction mattered. A self-service program that tries to handle everything ends up handling nothing well.

The other constraint was capacity. With manual ops, the ceiling was five active designers. Beyond that, I couldn’t maintain quality on recruiting, coaching, and the debrief ritual simultaneously. I knew this when I built it; it was the explicit tradeoff for keeping the rigor high. But it meant the program was a supplement to researcher-led work, not a replacement. Senior stakeholders sometimes wanted it to be the latter. The right answer was always to surface what the program could and couldn’t do and let them decide, rather than let the scope quietly drift.

The manual overhead that capped the program at five designers can now be compressed. In 2026, I would redesign three of the operational bottlenecks with AI in the loop, preserving the researcher’s role as the judgment layer rather than the execution layer.

1. 01 Living knowledge repositories. Product-area notebooks in a tool like NotebookLM, built from the accumulated session notes, synthesis emails, and debrief outputs. Instantly queryable by researchers and cross-functional partners. No more hunting through old decks or asking “did we test this?” The notebooks become the program’s institutional memory without requiring anyone to maintain a knowledge base manually.
2. 02 AI-assisted synthesis with researcher review. LLMs draft initial themes and findings from interview transcripts; researchers review, revise, and ground the output in the raw data. The researcher’s judgment is applied to pattern validity, not to transcription labor. This is where the Human-in-the-Loop principle matters most: AI outputs that go out without researcher grounding carry the authority of research without the accountability.
3. 03 Smart participant screening. LLM-drafted screener questions, scored responses against ideal participant profiles, edge cases flagged for human review. Recruiting was the biggest ops bottleneck in the original program. Most of it is pattern matching against known criteria, which LLMs handle well. The researcher applies judgment to edge cases and to the small number of recruits where fit matters more than screener alignment.

The underlying principle: AI scales and supplements UX researchers. It doesn’t replace them. The program should adapt to each team’s needs while remaining cognitively safe for researchers and participants. What changed between 2023 and 2026 isn’t the problem. It’s the ceiling on how many teams one researcher can support while holding the quality line.

Over two years, the program supported 20+ product initiatives through 100+ sessions, run by five designers with me as coach and quality gate. Four signals tracked whether it was working. Designers reported faster iteration on the biannual satisfaction and perceived-efficiency survey, with self-reported confidence in their own synthesis rising alongside it. PMs reported more confident direction-setting in the quarterly stakeholder interviews. The cross-functional debrief ritual embedded itself into team workflows beyond research sessions. And the model was adopted by adjacent teams that hit the same bottleneck.

The program also became the artifact that shaped how I think about research infrastructure in general. Research teams spend a lot of energy arguing that research matters. The bet this program placed was that research matters more when the surface area of who can do it gets wider and the researcher’s role gets more leveraged. Two years of data suggests the bet was right.

Bottom-of-the-page recirculation is one of the highest-leverage moments in the NYT reader experience: it guides someone from one story into the next, and converts casual readers into subscribers. The editorially-curated tool for that moment was called Related Links: a block of related stories at the bottom of an article, picked by hand by an editor.

But most readers weren’t seeing it. The data showed that most editors barely touched the tool. The product team’s working theory was that the newsroom didn’t see the value.

Contextual inquiry and shadowing with five editors across five desks, watching the full publication workflow rather than just the Related Links step. Shadowing was the right method because the question was about workflow integration, not tool usability in isolation. Synthesis used affinity diagramming across three lenses: business value, reader value, and editor workflow.

The answer wasn’t about motivation. It was about time and cognitive load. The tool required 10 to 15 minutes per article at a stage in the workflow when editors were already under deadline pressure and had mentally moved on to the next story. The behavior the product team read as disengagement was rational adaptation to real constraints.

The workflow took 10 to 15 minutes per article. Editors manually found 3 to 7 article URLs, picked a display option, wrote a title and description. By the time editors got to it, they were exhausted.

It was a final-step task with no clear ROI. Adding Related Links sat at step 9 of an 11-step publication process, after editors had already coordinated with print, visual, SEO, and copy desks. Without metrics to justify it, it dropped to the bottom of the list.

Post-publish updates were the worst friction. Every time a new article published in the topic area, editors had to go back and manually update every previous Related Links block. The maintenance burden compounded over time.

Editors had no shared mental model of the strategic value. Was Related Links a display feature? A recirculation engine? A subscriber conversion tool? Editors gave conflicting answers, which meant the product team and the newsroom weren’t aligned on what success looked like.

No guidelines on how, when, or how much. Editors didn’t know which display style drove the most CTR, how many links to include (3 or 7+), or which articles deserved the treatment. Decisions were made by gut, not data.

Editors didn’t know what else existed in their topic area. Without a discovery layer for related coverage, editors couldn’t make connections across the archive, even when they wanted to.

Manual selection led to inconsistent packaging. The same news event could end up with three different Related Links blocks, each curated by a different editor, none aware of the others. India’s COVID coverage was a vivid example.

Two distinct user groups, same frustration. News Editors needed speed and consistency. Features Editors wanted to compete with the news desk for placement and reader attention. Both wanted to find, save, and reuse previous blocks. Neither could.

Feature desks treated their story pages as mini sections. For desks like Styles, Related Links was a way to showcase the breadth of their coverage and create a destination for readers. The most strategic users were also the most underserved.

Some reporters used it for self-promotion. A use case the product team hadn’t imagined: reporters added Related Links to their own articles to showcase their previous work to readers.

Usage tracked workflow fit, not tenure. The 13-year veteran used it on 1 of 5 articles. The 3-year editor used it on 3 of 5. Familiarity didn’t fix what the workflow couldn’t accommodate.

The research pointed toward Prism. The legacy CMS could absorb incremental improvements, but the workflow problem was structural: Related Links required a manual step at the wrong moment in the publication process. Prism, still in development, offered a different architecture: one where the recirculation step could be repositioned, automated in part, and stripped of the cognitive load that was causing editors to skip it entirely. Figure 03 shows what that workflow looked like in practice.

Reframed the product team's understanding from "editors aren't adopting this" to "the workflow is broken, and that's why readers aren't getting the recirculation experience the strategy depends on."

The findings directly shaped the design of Prism, NYT's new CMS, built around the actual publication workflow rather than the idealized one. Bottom-of-page recirculation is one of the highest-leverage moments for reader conversion, which made the workflow fix a direct lever on subscriber growth. The synthesis methodology (affinity diagramming across business value, reader value, and editor workflow) became a template for adjacent CMS research at NYT.

NYT was working toward 10 million subscribers by 2025, and mobile apps were the highest-converting consumer surface. But the company hadn’t launched a new mobile feature for readers in two years. I led research on a proposed new tab aggregating breaking news, top stories, and personalized alerts into a reverse-chronological feed going back 72 hours, built for readers who wanted a fast catch-up.

The underlying question wasn’t really about the feature. It was about how readers form habits with information products. A reader who opens the NYT app has a mental model of what’s there. Adding a new tab doesn’t automatically update that model. The research question was: what does it take for a new surface to become part of how someone actually uses a product, rather than an option they never discover?

Full product lifecycle research spanning four phases: a competitive analysis to validate the problem space; pre-launch usability testing with a hi-fi prototype and internal dogfooding; an A/B framing test, in-product survey, and four-week diary study at launch; and a post-launch assessment measuring against KPIs. The diary study was chosen for post-launch because it captures behavior in natural context over time, which survey and interview methods cannot replicate for a habit-formation question.

The A/B test itself was designed, not just executed. Working from the July 2020 iOS baseline (5.1M WAU, 34% tap-through rate), I used NYT’s internal Abra Experiment Duration Calculator to model the tradeoff between audience allocation, test duration, and minimum detectable lift at a 95% confidence interval. The modeling made the calibration choice explicit: a 1% minimum detectable lift would need ~918K users and 27 days; a 3% lift would need ~103K users and 16 days. The team committed to detection sensitivity that matched the scale of effect we could act on, rather than over-powering the test for a lift that wouldn’t change the product decision.

Every competitor called their equivalent feature some version of Alerts. The category had converged on a pattern, including the name. A reverse-chronological feed, bell icon, 48 to 72 hour timeline: the design language was settled. That convergence became the question worth testing: if everyone called it Alerts, was that the right name, or just the inherited one?

Pre-launch usability testing with 24 participants across registered users and paying subscribers. Unmoderated remote sessions with a high-fidelity prototype. Participants moved through the user flow without prompting, found the feed readable and straightforward, and were open to seeing personalization added later. No major red flags surfaced. Subscribers wanted a morning-and-breaking-news habit. Registered users found the reverse-chronological framing made the feed feel timely and live.

15% of the iOS audience, equally split across two variants and a control. Both variants had identical design and content. The only difference was the tab name. The headline question: does framing a tab around a reader need (Catch Up) or a service (Alerts) drive more adoption?

Reader-need framing beat service framing on every metric. Pre-elections, Catch Up averaged 14,300 weekly active users (5.9% of eligible users) versus 12,500 for Alerts (5.1%). During election week, both variants roughly doubled, with Catch Up reaching 9.58% of eligible users versus 8.96% for Alerts. Tap-through rate followed the same pattern: 16.45% for Catch Up versus 14.14% for Alerts. The in-product survey put it most starkly: 67% of Catch Up users rated the experience Good versus 56% for Alerts, a gap that held across the entire test window.

But two findings complicated the picture. First, stickiness was identical across variants at 18% DAU/WAU — the name affected discovery and first impressions but not whether readers came back. Second, the tab was cannibalising existing behaviour rather than growing it. Sessions with the Catch Up or Alerts tab showed a slight decrease in For You tab activity. The tab wasn’t creating a new habit. It was redistributing an existing one.

The bigger insight was what the data couldn’t fully explain: some users confused the tab with For You, others discovered it by accident weeks after launch. Passive discovery wasn’t enough. New tabs need a major news driver, active onboarding, or promotion to overcome the inertia of an existing mental model.

13 participants split between both variants, mixing registered users and subscribers, over four weeks. The goal was depth: understanding habit formation and engagement over time, not just first impressions. Catch Up users incorporated the tab more consistently than Alerts users, but several felt ambivalent about it. The defining entry came from a participant who had had the tab for weeks without knowing it was there.

A great feature discovered by accident, weeks after launch. The finding pointed directly at what would solve it: a major news driver large enough that users couldn’t miss the new tab.

The research memo laid out three options. Rebuild the tab as a webview to allow faster iteration outside the native release cycle. Pause the tab and fold the alerts functionality into the broader personalized notification strategy. Or pause the tab and explore the concept of a live/breaking news destination as part of Storylines work. The memo recommended options two and three.

The reasoning was direct: the core hypothesis was right (readers want a place to track timely updates), but the MVP had delivered a lukewarm experience and native implementation was too slow to iterate toward a compelling one. The Elections Tab had already demonstrated that webview-based tabs could perform well. The question was whether to keep iterating on this specific concept or let it inform something bigger. The team chose the latter.

The 2020 US election was exactly the kind of news driver the Catch Up research had identified as the missing ingredient. Working from that hypothesis, I led the launch of a temporary Elections Tab: a focused destination for election coverage that ran for approximately six weeks. The hypothesis going in was that a pop-up, time-bound tab centered on a major news event would not disrupt habituated users and would drive new users to develop the habit the Catch Up tab had failed to instill on its own. Both predictions held.

Two million weekly active users in election week, and a research-led template that ended a two-year drought of mobile product launches.

The Elections Tab reached nearly 2 million weekly active users in election week, ran for approximately six weeks, and drove measurable lifts in registration and subscription conversion among readers who entered through the tab. It confirmed the discoverability hypothesis at scale: a major news driver solved the adoption problem that passive launch couldn't.

The research plan, experiment design, and post-launch memo were reused for two additional mobile launches including the Covid-19 Tab, making this the template for how mobile feature research was run at NYT going forward.