My Journey to Building an AI Agent: Extracting Insights from Social Media

Introduction 

Artificial Intelligence (AI) is transforming industries across the globe, and its influence on social media management is particularly noteworthy. AI agents can analyze vast amounts of data, uncovering patterns and trends that are otherwise difficult to detect. As part of my Master of Science degree program in Information Systems at the University of Utah, I faced a pivotal decision: complete a certification course or take on a capstone project. Drawn to the challenge and potential for growth, I chose the latter. Under the guidance of Prof. Rohit Aggarwal from the Information Systems Department at the David Eccles School of Business, I embarked on an exciting journey to build an AI agent capable of revolutionizing social media posts. Little did I know that this project would push me far beyond the boundaries of my classroom knowledge and into the realm of practical, cutting-edge AI application.

Seizing the Opportunity: The Beginning of My AI Journey 

The project's scope was ambitious: develop an AI system that could analyze historical social media content and generate future posts to drive engagement. I chose Instagram as the primary platform, focusing on strategies employed by industry leaders like Ogilvy, BBDO, AKQA, and MCCANN. These companies, known for their expertise in brand promotion, provided valuable insights into audience engagement that I could leverage in my AI agent.

My task involved collecting data from company websites and social media platforms through web scraping, processing it, and utilizing AI models to extract meaningful themes. The ultimate goal was to generate future posts that would drive engagement and align with each company's brand. This project would test not only my technical skills but also my ability to understand and apply marketing strategies in a digital context.

The Journey: Building the AI Agent

Data Collection and Preprocessing

My first major hurdle was data collection through web scraping. While I had some exposure to Python in my classes, this project demanded a level of expertise I hadn't yet achieved. I spent countless hours poring over YouTube tutorials, documentation, and online forums to master the intricacies of web scraping. I learned to use Python libraries like Instaloader to obtain data from Instagram pages, and Selenium and BeautifulSoup to scrape company websites. This process yielded valuable information, including captions, shares, likes, and comments from each company's Instagram account.

After collecting the data, I moved on to preprocessing. I cleaned the data by removing duplicates and null values, converted dates to a datetime format, and prepared it for analysis. Ensuring that the data was accurate and well-organized was crucial for setting a solid foundation for theme extraction. This step taught me the importance of data quality in AI projects, a concept that was only briefly touched upon in my coursework.

Theme Extraction and Grouping

With the data ready, I conducted theme extraction using the Gemma2b model from Ollama. This was a significant leap from the basic machine learning concepts I had learned in class. I employed zero-shot prompting, a method where I asked the model to perform tasks it hadn't been explicitly trained on. By providing suggested themes, I guided Gemma2b to extract relevant themes from the Instagram posts, such as 'Product Announcement' and ‘Customer Story.’

Once I extracted the themes, I grouped and normalized them. I used Gemma2b to categorize the themes into more concise groups, ensuring that similar themes like 'Customer Story' and 'Customer Stories' were treated as one. This normalization was essential for scaling the data effectively, teaching me about the nuances of natural language processing and the importance of context in AI-driven text analysis.

Engagement Analysis and Generating Future Posts

Next, I conducted an engagement analysis by calculating scores for each theme based on likes, shares, and comments. Summing up these metrics helped me identify the top 10 themes across all companies. This analysis revealed which themes were driving engagement and how companies like Ogilvy and AKQA were leveraging these strategies. This step required me to blend my understanding of social media metrics with data analysis techniques, bridging the gap between marketing concepts and technical implementation.

Armed with this analysis, I used Gemma2b to generate future social media posts. I crafted these posts based on the successful strategies I identified, with suggestions for images, videos, captions, and hashtags. I also included a predicted engagement score for each post, aiding social media managers in planning their content effectively. This phase of the project was particularly exciting as it allowed me to see the practical application of AI in content creation, a concept far beyond what I had learned in my classes.

To make my AI agent accessible, I developed an interactive interface using Streamlit. This user-friendly platform allowed social media managers to interact with the model, generate posts, and visualize engagement predictions. Creating this interface pushed me to learn about web application development and user experience design, areas that were entirely new to me but crucial for making my AI agent practical and usable.

Challenges I Faced

Throughout this project, I encountered numerous challenges that pushed me far beyond what I had learned in my classes:

Web Scraping Implementation: Despite my theoretical knowledge of web scraping in Python, this project demanded practical application at a much higher level. I had to enhance my skills through intensive study of YouTube tutorials and comprehensive reading on the subject, including its legal implications to ensure compliance.

Model Selection and Deployment: I initially explored quantized models for local execution, gaining extensive knowledge about their capabilities and limitations. After considering various options, including GPU-dependent models, I settled on Gemma 2b with Ollama due to its compatibility with my local machine's resources. This decision came after attempting to use Google Colab's enhanced GPU environment, which proved financially unfeasible for my project's scope.

Development Environment Setup: Setting up the working environment posed its own challenges. I opted for Visual Studio Code, which provided a robust platform for code structuring and debugging the large language model. This choice significantly improved my workflow efficiency, but required me to learn a new development environment.

Data Processing and Analysis: Data cleaning and merging CSV files presented initial hurdles. I overcame these by developing Python scripts to streamline these processes. The most significant challenge was extracting themes from the large dataset using Gemma 2b, which required substantial computational time. To address this, I utilized a high-RAM system and implemented checkpoints in my code to manage the process more effectively.

Model Fine-tuning and Result Validation: To ensure the extracted themes aligned with the desired format, I implemented a training method using sample themes. This was followed by a meticulous manual review process to verify the accuracy and relevance of the extracted themes.

Post-processing and Application Development: Once I extracted themes, I leveraged the model to categorically group them and align them with engagement metrics. Additionally, I used Gemma to generate weekly posts designed to resonate with the target audience. The final step involved developing a Streamlit application to generate prompt responses, providing a user-friendly interface for accessing the project's insights.

Lessons Learned and Conclusion

Despite the difficulties, this project provided me with invaluable lessons. I honed my coding skills, mastered the intricacies of web scraping, and gained hands-on experience with machine learning models. Additionally, the project emphasized the importance of adaptability, communication, and project management—skills that are crucial for success in any professional setting.

Building this AI agent was a transformative experience for me. It not only equipped me with technical skills but also prepared me for future roles in AI and data analytics. My project demonstrated the potential of AI in enhancing social media management and underscored the importance of understanding data to make informed decisions.

Looking ahead, I'm excited about the possibilities AI offers and the role I can play in shaping this technology. This experience has not only provided me with technical skills but also ignited a passion for creating AI solutions that can make a real difference in how businesses understand and interact with their digital audience. My journey of building my first AI agent has laid a solid foundation for future projects, and I have a strong desire to continue learning and growing in this dynamic field.