How Beekeeper optimized user personalization with Amazon Bedrock
How Beekeeper optimized user personalization with Amazon Bedrock
Publish Date: 2026-01-09 11:10:00
Source Domain: aws.amazon.com
-
Dynamic Model and Prompt Evaluation: Large Language Models (LLMs) are rapidly evolving, making it difficult for organizations to choose the best model and optimize prompts. Beekeeper solves this challenge with a system that continuously evaluates and ranks model-prompt combinations based on quality, cost, and speed.
-
Automated System Features: The system tests and ranks model/prompt pairs live, incorporates user feedback to personalize responses, and automatically routes requests to the optimal choice currently available.
-
Real-World Application: An example of Beekeeper’s LLM system in action is the chat summarization feature for deskless workers. When users return to their shifts, they can request summaries of lengthy chats, receiving concise overviews tailored to their specific needs.
-
Evaluation Criteria: The system uses quantitative and qualitative metrics to measure summary quality, including compression ratios, action item presence, lack of hallucinations, and vector comparison for semantic similarity.
-
Feedback Integration: Beekeeper incorporates both synthetic and real user feedback into the evaluation process to continuously improve the model-prompt combinations.
-
AWS Integration: The system leverages several AWS tools and services, such as Amazon Bedrock for model access, Amazon EventBridge for scheduling and orchestration, Amazon EKS for container management, Amazon Lambda for evaluation functions, Amazon RDS for data storage, and Amazon Mechanical Turk for manual validation.
-
Benefits: Key advantages include rapid evolution to meet user needs, cost control, improved quality of results, and the ability to tailor prompts to meet unique tenant requirements.
-
Conclusion: Beekeeper’s approach of automating the evaluation and optimization of LLM and prompts helps organizations use the best-performing combinations for their specific use cases, while preparing for ongoing refinements and user customization.
