Client Overview

Greater Public headquartered in Minneapolis, US is a non-profit organization dedicated to supporting fundraisers and marketers of public radio stations and television. 

The organization heavily relies on written materials for fundraising such as on-air scripts, letters, emails and social media with the aim of increasing the campaign effectiveness. This means converting the radio stations listeners into paying contributors.

In the face of the evolving landscape of artificial intelligence and Generative AI, they were eager to explore these technologies to enhance their script and letter products. Their aim was to launch beta versions of these products by the end of the quarter. To accomplish this they were seeking a skilled Prompt Engineer and Generative AI consultant. 

Business Challenge

The end customers for GP are Radio stations of different sizes catering to diverse target audiences through varied genre contents. GP has a strong team of content writers and SMEs (Subject Matter Experts) who provide the campaign specific content and advise the radio stations on their marketing approach. There are major radio stations that have large reach and have a clear value proposition that is communicated well. On the other hand there are medium and small radio stations that rely on the Greater public for this. 

The On-air scripts for fundraising written by SMEs (Subject Matter Experts) were available as a static arrangement that is accessible through the website – behind the paywalls. The template based scripts were provided in the form of google docs that could be later customized by users. The process of creating and customizing templates became increasingly complex due to multiple factors. These included diverse radio station formats (ranging from news to music), seasonal variations (fall versus spring campaigns), different donor targets (monthly supporters or major one-time contributors), ever evolving local and global affairs relevant to the target listeners in growing customer base i.e. newly onboarded radio stations. This combination of variables made template management and personalization increasingly challenging and hampered the effectiveness. 

The written pitches often became too station-centric and lacked fresh ideas. The goal was to reframe station features and unique value proposition as listener benefits. Essential was to ensure highest relevance in these short messages. The approach aimed to prompt listeners to reflect on the value they receive and encourage them to translate that value into monthly financial support.

When fundraising, it’s crucial to understand your target audience. Many stations overlook a key fact: they are addressing loyal listeners who haven’t yet become donors. These individuals perhaps already have a strong connection to their content. The challenge during pitch breaks is to engage these listeners, persuade them of the value of contributing, and motivate them to take immediate action by donating. So one of the main objectives was to expand the club of givers.

Expected outcome from AI assistant development:

• End product expertly blends user-provided personalization variables with meticulously crafted prompts, ensuring the delivery of highly relevant and impactful content to each individual customer.
• Provide input on the development of a user interface for users to input variables, which will then be integrated into prompts to generate high-quality fundraising scripts and letters.
• Ensuring the outputs meet the defined outcomes, aligning with the high standards of Greater Public professional fundraising experts. In other words, scripts should sound human-like.
• Establishing a transparent systematic process for testing, replicating, and validating results.
• Measurement and comparison of generated content against a set of predefined variables to ensure consistency and quality across diverse inputs.
• Provide insights into how AI can be further leveraged effectively.
• Assist in demystifying the AI process for Greater Public team, contributing to a better understanding of its potential and limitations.

Our Approach

Based on our years of experience at TotemXLabs, we’ve established a streamlined approach for the entire project lifecycle, from initial onboarding through development to final delivery. This standardized yet adaptable process ensures smooth and efficient solution implementation.

Our methodology includes

1. 1. Discovery phase: Develop domain cognizance, stakeholder engagement, identify requirements,problems,limitations,expectations, understand the customer segments and establish KPIs.
   2. Data: Access, Gather, Extract, Verify, Validate, Transform, Process, Engineer, Generate. Most important step in AI product design and development.
   3. AI solution design: User journey designing, identify the tech stacks and frameworks selection, Clear prioritization of essential and nice-to-have features, Timeline. Outcome of this phase is a development plan aligning with the vision and goals of all the stakeholders . 
   4. Development: PoC, MVP, Iterative improvements, Key milestones, Tackling challenges, Cross-functional collaboration, 
   5. Deployment: Scaling, Feedback, Maintenance, Upgrade

Discovery phase

In the cutthroat world of AI solutions, our client initially fell for the siren song of an off-the-shelf, one-size-fits-all approach. They onboarded a top-shelf proprietary Large Language Model (LLM) faster than you can say “disruptive innovation.” However, this AI wunderkind turned out to be more “artificial” than “intelligent” when it came to script generation. The problem was the content being generated was failing to meet mission-critical criteria such as human-like tone, relevance, meet the baseline pitch quality, understand the value proposition and communication, follow the prescribed practices or reach at least halfway to benchmark standards of content delivery from GP.

The real ROI lies in bespoke AI implementations. To address this pain point, we initiated a deep-dive discovery phase. This strategic alignment session served as a bilateral knowledge transfer conduit, facilitating the exchange of domain-specific insights, data availability, core competencies, existing tech stack, key stakeholders, KPIs, and ROI metrics.

We engaged in a 360-degree learning process with Greater Public, absorbing critical information about their industry vertical, radio script templates, customer segmentation, and the value propositions of their B2B clients (radio stations). We also identified their mission-critical requirements for an AI assistant to optimize their workflow bottlenecks.

In return, we provided a comprehensive overview of data requirements, elucidated the limitations of generic LLM models, and presented a panoramic view of available open-source and proprietary LLM options. We also outlined our proposed tech stack, user experience journey mapping, and potential AI implementation strategies to maximize business value and drive synergies across their operations.

Discovery phase in our approach determines dos and don’ts and ensures we’re not just thinking outside the box, but redefining the box itself in our quest for AI-driven excellence. 

Data

In the data-driven paradigm of AI product development, high-quality data isn’t just a nice-to-have – it’s the rocket fuel for your AI transformation journey. Leveraging relevant, adaptable datasets can catapult your AI solutions from mere proof-of-concept to market-disrupting juggernauts, driving exponential ROI and unlocking unprecedented business value.

Prompt engineering, the unsung hero of AI alchemy, is where the rubber meets the road in human-LLM interactions. It’s the art and science of fine-tuning the conversational interface between user queries and AI responses, akin to teaching an eager puppy to fetch the Financial Times instead of your slippers. The endgame? Crafting prompts are so razor-sharp, they could slice through the Gordian knot of ambiguity, ensuring the AI’s output aligns with user objectives like a well-oiled machine. Understanding and structuring the data plays a pivotal role in designing such prompts.

In our initial development phase, we commenced with a baseline implementation – a minimalist prompt engineering approach. We deployed system prompts with limited specificity, initially overlooking the critical factors of data variance and bias. The outcomes were subjected to rigorous evaluation by our Subject Matter Experts (SMEs), resulting in a suboptimal average rating of 2.5,5, indicating significant room for enhancement.

Leveraging these insights, we pivoted to a more refined methodology in subsequent iterations. We recalibrated our system prompts, incorporating high-fidelity data examples while simultaneously implementing robust strategies to mitigate bias and variance. This data-driven optimization yielded substantial improvements, culminating in a remarkable average rating of 4.5,5. These quantifiable results underscore the transformative impact of our iterative, data-centric approach to prompt engineering and AI model fine-tuning.

AI Solution Design

In the realm of AI product design, we employ a comprehensive strategy that addresses the quintessential W’s (what, why, who) and H (how) paradigm. What would the user journey look like? Why to select the particular tech stacks and frameworks? How to prioritize essential and nice-to-have features? Who are the stakeholders? When to expect the first version to be ready for rollout? This holistic approach ensures alignment with both business objectives and technological feasibility. 

Our user journey mapping process leverages design thinking methodologies to optimize the client experience. In prioritizing feature sets, we utilize the MoSCoW method (Must have, Should have, Could have, Won’t have) to streamline development and maximize ROI. Stakeholder identification and engagement are facilitated through RACI matrices, ensuring clear accountability and communication channels. 

To design a user journey for our AI-driven copywriting assistant, we initiated by conducting a thorough analysis of script customization variables. The UI,UX design process for the conversational interface incorporated dropdown menus for key parameters such as format (News,Jazz,AAA,All Music,Classical), drive time (Any,Spring,Fall,FYE,CYE), and donor segmentation (monthly,one-time contributors). This user-centric approach significantly reduces cognitive load and enhances overall user experience, culminating in improved productivity metrics.

Given the rapid evolution of the LLM ecosystem, we implemented an agile evaluation framework to benchmark emerging models against established solutions. This iterative testing protocol, validated by Subject Matter Experts (SMEs), ensures continuous improvement and future-proofing of our AI solution.

Building an ideal UI is a time consuming process. We at TotemXLabs believe in quick wins at every juncture. To accelerate time-to-market and optimize resource allocation, we leveraged the Streamlit Python library for rapid prototyping of the chatbot interface. This strategic decision exemplifies our commitment to lean development principles and cost-effective solutions.

While we champion AI-driven innovation, we maintain a steadfast belief in the human-in-the-loop paradigm as a critical success factor. We instituted a robust performance evaluation process, incorporating SME feedback through a quantitative scoring system and qualitative commentary. This invaluable input serves as a cornerstone for our prompt engineering efforts.

By implementing the above AI solution elements we could commit and manage the project timeline, allowing for rapid iterations and continuous improvement. Despite the evolving landscape of LLM technologies, our team successfully navigated challenges and delivered the minimum viable product (MVP) within the projected quarter. The initial rollout exceeded expectations, with the beta version demonstrating significant improvements in copywriting efficiency and quality.

Development

As we approach the denouement of our case study, one might anticipate a narrative replete with formidable obstacles, innovative problem-solving methodologies, and intricate stakeholder management strategies – the hallmarks of a typical software development lifecycle. However, we must subvert these expectations. Thanks to much of the lifting done in previous phases, we had a development journey like riding a supercar on a German autobahn. 

The comprehensive discovery phase yielded a strategic roadmap that significantly de-risked our execution. Our data processing stage, executed with precision, culminated in a refined knowledge base, primed for the engineering of sophisticated system prompts. Subsequent phases confirmed the framework for testing multiple LLMs, determining key input parameters and evaluation through feedback loop was hot to trot. 

We went through four major iterations, or as we like to call them experiments. 

Experiment Template: 

Step 1: Transforming the data from the Airtable knowledge base into structures consumable by LLMs

Step 2: Structuring prompt design

• SystemPrompt ingredients
  • Given instructions from meticulously selected from Greater Public articles on how to write ideal scripts
  • Explained the task and priority.
  • Explained the inputs to be expected, definition of format, tones, what are radio scripts and what is it supposed to do.
  • Input Examples from randomly selected from existing scripts
    • Input variable 1 : Format: AAA
    • Input variable 2: Drive time: Fall
    • Input variable 3: Tone: Energetic
    • Output: It’s amazing to think of all the music [STATION] shares with our community every single day…

Step 3: Generate responses, parse into readable formats

• Test Example
  • Format: All Music
  • Category: Spring
  • Tone: reassuring, persuasive, community-oriented

Step 4: Evaluation and feedback by SMEs

We cast a wide net, evaluating five proprietary LLMs: Gemini Pro 1.0, Claude instant, Claude 2.1, GPT-3.5-turbo, and GPT-4-turbo. This barebones approach served as our digital litmus test, gauging raw LLM performance with minimal prompt engineering—akin to sending a toddler to pitch a Fortune 500 company. As anticipated, the results were about as impressive as a PowerPoint presentation without animations.

First experiment, our maiden voyage, was a vanilla implementation strategy. We just went ahead with choosing random examples from available scripts. We cast a wide net, evaluating five proprietary LLMs: Gemini Pro 1.0, Claude instant, Claude 2.1, GPT-3.5-turbo and GPT-4-turbo. The reason for running a very basic experiment was to evaluate raw performance of LLMs with minimal prompt engineering – akin to sending a toddler to pitch a Fortune 500 company. As anticipated, the results were about as impressive as a PowerPoint presentation without animations.

The second experiment saw us pivot faster than a startup changing its business model. We leveraged SME-curated, crème de la crème scripts as input examples. Recognizing that ‘Tone’ as a variable was as unpredictable as the stock market, we bid it adieu and introduced donor segmentation as our new north star. Additionally, we prompted models to write a little more informal and creative scripts than the provided examples. In this round, we welcomed Claude 3 Sonnet to our LLM portfolio, deploying it through Amazon Bedrock with an eye on future scalability. The results of this experiment ranged from very good to below average.

Our third experiment targeted the underperformers with the precision of a heat-seeking missile. We fine-tuned our LLM lineup based on the previous round’s American Idol-style reviews. Step 3 evolved into a demo application—a veritable playground for our client to test-drive the AI. This digital sandbox featured model selection options, predefined inputs, and a “Creativity” slider that went to 11 (metaphorically speaking). By the experiment’s conclusion, we had scripts that not only met but pole-vaulted over SME expectations. We had our MVP—Most Valuable Prompter.

After careful analysis of previous iterations and feedback from SMEs, with the third experiment we targeted the below average cases and we fine-tuned our prompts. Step 3 evolved into a demo application—a veritable playground for our client to test-drive the AI. 

Step 3: Generate responses through demo app

• Demo app consists of following components
  • Options for models selection
  • Inputs given as predefined options
  • Sliding input Creativity hyperparameter (0-1.0)
  • Based on selected options generate script by clicking a button

This digital sandbox featured model selection options, predefined inputs, and a “Creativity” slider that went to 11 (metaphorically speaking). By the experiment’s conclusion, we had scripts that not only met but pole-vaulted over SME expectations. We had our MVP—Most Valuable Prompter.

We locked on the default LLM to be used in the final version, defining final modifications for UI,UX. 

The final experiment shifted gears smoother than a luxury sports car, focusing on transforming our copywriting prodigy into a full-fledged conversational savant. We deliberately kept this aspect under wraps until the eleventh hour to avoid opening Pandora’s box of complexities. This phase was all about perfecting our prompt engineering strategy—like teaching a parrot to recite Shakespeare—and putting the final touches on our UI/UX.

In a mere four iterations we had a market-ready product. Our agile approach facilitated rapid adaptation and problem-solving, resulting in an AI-powered copywriting assistant that combines efficiency with creativity. The result? An AI-powered copywriting assistant poised to revolutionize the industry faster than you can say “paradigm shift.”

Deployment

To ensure seamless integration with Greater Public’s existing infrastructure, the AI assistant was deployed on Amazon Web Services (AWS), leveraging its robust cloud ecosystem. We implemented a comprehensive CI/CD pipeline using GitHub along with AWS CodeDeploy and CodePipeline, orchestrating automated testing and deployment processes. This DevOps approach facilitates rapid iteration and version control, allowing for efficient upgrades and rollbacks as needed. This ultimately makes pushing upgrades as easy as pushing buttons on a coffee machine. 

The final deployment of LLM agent:

ScriptPro AI

Client Testimonial

The below screenshot is just before the launch:

Why Choose TotemXLabs

Our meticulously orchestrated approach, underpinned by agile principles and data-driven decision-making, facilitated a product development that was not just efficient, but transformative. By front-loading complexity and leveraging cutting-edge AI methodologies, we effectively mitigated common pitfalls, resulting in a streamlined development cycle that exceeded both temporal and qualitative benchmarks.

Do we need to write any more to make you believe in our commitment for excellence! Still in doubt? Give us a call.