27. June 2023

230627 Entry Level Digital Marketing - Experimentation Overview

  1. Ideate Use data (user behaviour, feedback from surveys and interviews, competitor research, etc.) to pinpoint potential areas of improvement, or even problems to solve. You can also use Pirate Metrics (Acquisition, Activation, Retention, Revenue, and Referral) to brainstorm experiments that optimise each step of the funnel.
  2. Prioritise Use ICE scores to help - more details in an upcoming module! Prioritising experiments is important because when you lose focus, you can lose momentum. Then, nothing gets done and growth stagnates.
  3. Create a hypothesis Make your best guess about what you think will happen, based on the data you have. Fill in the blank: Based on [insert data or observation], I believe [insert what you’re experimenting with] will impact [Pirate Metric] by [insert goal here]. Example: Based on 35 users saying they need more information before signing up for a course, I believe adding a syllabus to the top of the website will impact acquisition by increasing sign up rates to 25%.
  4. Plan and run the experiment Write down a description for the experiment. Include how you will run it - ideally, experiments are quick and do not cost many resources. Consider what success looks like and decide on important metrics to track.
  5. Analyse results Was your hypothesis correct? What did you learn? Should you implement the changes on a wider scale? Remember to share what you learned with the rest of your team.

Connect it with Pirate Metrics

Connect it with Pirate Metrics

Test Funnel Source Description
Emojis vs. no emojis in the subject line Acquisition Link Reaching out to or nurturing a potential lead
Longer vs. shorter subject lines Acquisiton Link Reaching out to or nurturing a potential lead
Hero Image Activation Link Clearer message to value proposition
Offer Revenue Link More compelling offer
Banner design Acquisition Link Simplify text

Experiments

Acquisition

Description: Test out a thumbnail Experiment Goal: Determine a better layout of thumbnail for YouTube video recipes marketing content. Hypothesis: Thumbnails that have a key ingredient get more views. Control: SHow image of final dish Variant: Show image of single ingredient

Things I would look for:

  • Views
  • Traffic from YouTube link
  • Watch length

Activation

Description: Test out a magnet Experiment Goal: Assess the impact of signups from emails with recipes with a link to the Play Store on the bottom Hypothesis: Users who have read one recipe would want to find more Control: Users without emails. Variant: Users with emails.

Things I would look for:

  • Signups from the emails
  • Signups to the emails

Revenue

Description: Test pricing Experiment Goal: See what kind of pricing users are willing to pay Hypothesis: Users from the website who are given a discount, may be more willing to buy. Control: Regular web traffic Variant: Users are presented with a 20% discount for signing up on web (Since we don't need to pay the app store)

Things I would look for:

  • Sales

Retention

Description: Push Notification reminders Experiment Goal: See if push notifications make users come back to the app Hypothesis: Notifications on the phone would help the user remember that the app is on their phone Control: Regular usage Variant: Users are sent notifications on their phone to try out new recipes with leftover chicken

Things I would look for:

  • Clicks onto the notification
  • User retention
  • LTV

Referral

Description: Test Sharing Experiment Goal: See if sharing a recipe and ingredients with partner would make the user more likely to share with others Hypothesis: Nurturing users with a small behaviour of sharing may get them sharing more with others Control: Regular usage Variant: Provide users to share a link with hero image, recipe, and ingredients with significant other.

Things I would look for:

  • Share counts
  • User retention