Restricted category To view older posts, you must first submit an approved post in that category

Conflicting goals

Hi- would like to gather your feedback on this.

Let’s assume hypothetical question: For the core news feed, should we display more video watches or more events?

  1. Unifying Metric: Would overall active users for newsfeed be the right metric to determine / or should we compare click through rate for video versus event (and calculate each product’s active user uplift)? How to think about relationship/ structure between topline metric vs north star metric in this case? What could be an unifying metric?

How to deal with this type of comparison case where north star metric is different across products? (video : time spend, event: people RSVPed, goal oriented and time spend is less critical)

  1. Testing for solve: would multi-variant testing be right approach ? (Scenario 1: no change S2: more video watch, S3: more events). How long the test should be run? What are trade-offs for this type of testing?
1 Like
  1. Unifying Metric: Would overall active users for newsfeed be the right metric to determine / or should we compare click through rate for video versus event (and calculate each product’s active user uplift)? How to think about relationship/ structure between topline metric vs north star metric in this case? What could be an unifying metric?

Active users (DAUs) overall (not just on newsfeed) since newsfeed would have such a strong influence across the platform.

  1. Testing for solve: would multi-variant testing be right approach ? (Scenario 1: no change S2: more video watch, S3: more events). How long the test should be run? What are trade-offs for this type of testing?

It is true that sometimes the products are fairly dissimilar (watch time vs. RSVP) but generally if they’re all social products the unifying metric is DAUs (or time on site).

Regarding how many variants: good to mention if you want, but this not critical and squarely in the “optimizations” bucket.

For example, here you could have 5 variants (a lot of video, a little video, treatment, a little events, a lot of events) or 10 variants or 100 variants if you want — more variants will get you closer to the optimal solution but will take longer to run (less statistical power). You can assume you’ll be working with a DS to define how many variants — your job is not the implementation but the clarity on goals.

On duration: generally the “novelty effect” lasts for a few weeks at most and so for most tests 6-8 week is plenty. If you have a product that you expect will have long-term effects (e.g. people get bored of it or perhaps that it will take a long time before it’s useful), make sure to emphasize that you’ll run a long test (8+ weeks).

1 Like

Thanks a lot! It’s very helpful!!