Back to Blog

Always Prototype, Always Learn: Part One

5 min readWired Sixth Intelligence
#machine-learning#internship#data-engineering#prototype#learning

Always Prototype, Always Learn: Part One

By late 2024, I'd realized my technical skills had plateaued. The patterns I was using felt familiar, maybe too familiar. The same day-to-day data engineering work had become muscle memory. Meanwhile, my curiosity about machine learning was growing, but I didn't have a structured way to bridge the gap between what I knew and what I wanted to learn.

The problem wasn't technical. It was time. When do you find space to learn something new when your primary job already demands everything? My answer was to mandate it: I made "always be prototyping, always be learning" one of my pillars.

That's when Megan joined us as an intern. She had a theoretical foundation from her coursework, while I could anchor things with business context and technical guardrails. This combination worked for a prototype sprint.

Here's what we built together.

The Opportunity

We had years of historical social media data sitting around—TikTok, Google Ads, Meta campaigns with engagement metrics, impressions, clicks, and conversions scattered across different platforms. It was there, but untouched.

I pitched an idea: "What if we built a model to predict campaign engagement? Help the business estimate future ad spend?"

We started with weekly syncs, though we had to stretch it to every two weeks when our main data engineering pipeline work caught up with us.

Figuring out what to measure

Our first lesson: keep it simple.

Beginners often over-engineer feature sets. We focused on dimensions the business actually cared about:

  • Ad formats
  • Campaign objectives
  • Movie genres
  • Markets
  • Campaign phases

The insight here was simple: business relevance beats technical sophistication every time. Megan had read about dozens of advanced features, but we started with the basics. The fancy stuff can come later.

Facing the dirty data reality

This is where reality hit.

Dirty Data: The Outlier Problem

Before we even got to EDA, we had to deal with the garbage in our dataset.

We found campaigns with 200-300% engagement rates—literally impossible numbers. These were:

  • Paused campaigns that spilled over into our data
  • Test buys across native platforms that got logged incorrectly
  • Human errors where someone ran a campaign with broken tracking

Before we could even start proper EDA, we had to filter out this garbage. If we didn't, our model would learn from nonsense.

Data Flow: From Raw to Clean

We pulled our data expecting clean tables ready for EDA. What we found instead:

  • Null values in places we didn't expect
  • Ridiculous outliers (200-300% engagement) from paused campaigns and test buys
  • Human errors that contaminated otherwise clean data
  • Broken joins across channel-specific tables
  • Missing IDs that required careful reconciliation

Even data that had gone through our ETL pipeline needed significant cleaning. We had to go back and fix our data pipelines before we could even start proper exploration. We learned that "clean" data from ETL is never really clean.

The EDA doom loop

Once we had somewhat cleaner data, we dove into EDA. This is where the iteration really started.

EDA Iteration Cycle

  1. Run EDA → discover data isn't balanced
  2. Decide: normalize, log-transform, or remove outliers?
  3. Go back and fix the data
  4. Run EDA again

Repeat. A lot.

Each cycle taught us something new about our data quality and the real-world messiness that exists even after processing. We had to make decisions: include or exclude? Normalize or leave it? Remove outliers or keep them?

The iteration rollercoaster

By the time we finally had "clean enough" data for an MVP model, we were already several weeks into the project. The excitement of "we're ready to train!" met reality pretty quickly.

Model Performance Progression

Our initial model performance was disappointing. We went back to feature engineering. We went back to EDA. We iterated again.

When we finally looked at R² scores and accuracy metrics, we realized something important: the data size mattered. After aggressive aggregation and cleaning, we'd gone from 10 million rows to just a few hundred thousand. Was that enough to train a meaningful model? Probably not ideal. But it was something to start with.

The Long Haul to Production

Here's where things got really interesting—and where the project took an unexpected turn.

The 6-Month Reality

Six months in, Megan's internship came to an end. We'd made great progress on the model, but the deployment work was still ahead. The project could have ended there—a nice learning exercise, but no real business impact.

That's when Levi stepped in.

Continue to Part Two to see what happened next with Levi and the AutoML breakthrough.

This is Part One of a two-part series. Part Two covers the AutoML breakthrough and deployment to staging.