RecSys 2009 Keynote: Top 10 Lessons Learned Developing, Deploying, and Operating Real-World Recommender Systems
Presented by Dr. Francisco Martin, Founder and CEO of Strands, the first keynote at the 3rd annual ACM Conference on Recommender Systems was delivered this morning and has been generating some buzz on Twitter.
The number of online services providing users with real-time recommendations has increased exponentially over the last few years. Recommender Systems that were originally only accessible to a limited number of high-tech companies are now widely available through a growing number of both technical choices and vendors. The acceptance, however, of automatically delivered recommendations by users depends on numerous factors that go far beyond the algorithms that constitute the major focus of researchers.
Over the past 10 years, Strands has been creating and operating recommender systems in a multitude of domains, ranging from digital music to fitness plans and personal finance management, and in a multitude of business settings ranging from lightweight integrations to highly-coupled integrations within secure bank environments.
As summarized by Neal Lathia of MobBlog, below are the top 10 lessons we’ve learned developing, deploying, and operating real-world recommender systems:
- Lesson 1. Make sure a recommender is really needed! Do you have lots of recommendable items? Many diverse customers?… also think Return-on-Invesment… a more sophisticated recommender may not deliver a better ROI.
- Lesson 2. Make sure the recommendations make strategic sense. Is the best recommendation for the customer also the best for the business? What is the difference between a good and useful recommendation? Good recommendations vs useful recs; obvious recommendations may not be useful; risky recs may deliver better long-term value.
- Lesson 3. Choose the right partner! Select the right rec vendor vs hire some #recsys09 students. If you are a big company the best thing you can do is organize a contest.
- Lesson 4. Forget about cold-start problems (!) …. just be creative. The internet has all the data that you need (somewhere…).
- Lesson 5. Get the right balance between data and algorithms. 70% of the success of a recommendation system is in the data, the other 30% in the algorithm.
- Lesson 6. Finding correlated items is easy but deciding what, how, and when to present them to the user is hard… or don’t just recommend for the sake of it. Remember, user attention is a scarce and valuable resource. Use it wisely! … dont make a recommendations to a customer who is just about to pay for items at the checkout! User interface should get at least 50% of your attention.
- Lesson 7 Don’t waste time computing nearest neighbours (use social connections)… just mine the social graph. Might miss useful connections?
- Lesson 8 Don’t wait to scale!
- Lesson 9: Choose the right feedback mechanism. Stars vs thumbs …. the YouTube problem. More research on implicit and other feedback mechanisms is needed. The perfect rating system is no rating system! … focus on the interface.
- Lesson 10 Measure Everything! … business control and analytics is a big opportunity here.
Keynote Takeaway
Think about application context; Focus on interface as much as algs; be creative with startup data. … the UI needs to get the lion’s share of the effort (50%) compared to algorithms (5%), knowledge (20%), analytics (25%).










