socialcrawl

Lessons learnt from building a social media API

Building is a blast until reality hits

It’s 2026, product building is EASY. Claude Code, Codex, OpenClaw, Hermes Agent, you name it. Give it some specifications, install some powerful skills, throw in some testing scenarios, and ask it to build a product, ola! Congratulations, you’re now officially a founder too.

After consuming enough caffeine to hear your own heart drumming, and even watching the night owl going to bed before you, you know you are onto something. Working on client projects to our very own projects, building small here, another a bit bigger there, or implementing a feature a user requested, whatever it was, building has been a blast. But when it comes to actually distributing it? That is another story.

After all these nights, what is the point when no one uses your product? How do we distribute the great product you’ve built that you know solves a huge problem when no one knows you exist? In the olden days, the most obvious option would be shouting that about your new product on the streets. Interested people will come and listen to you, and potentially buy it. Well in 2026, I believe that is social media.

Realising that influence takes time

Growing your social media account is not easy at all. Yes, you need to make great content and posts, but you also need to drive high engagement with other posts. This takes time and a lot of effort. There are at least 7 popular social media platforms to reach your users. These include X, Threads, LinkedIn, Reddit, Instagram, TikTok and YouTube. Assuming you’d spend around 20 minutes a day on each platform writing comments, that would add up to 140 minutes or 1 hr and 20 minutes daily! When you start to prepare for content, that becomes a whole new issue.

So I did what developers do when they hit a wall – I over-researched it. Roughly 500 hours of YouTube marketing content, a pile of Reddit threads, and a few podcasts on loop. The conclusion was that every marketer who actually scales has a system they run. Not vibes, not posting whenever they felt like it, a repeatable loop that could be scaled and be replicated.

Then I thought I could also make an AI that watches X, Reddit, Threads, and Instagram on a schedule, spots posts worth replying to, tracks which ones are blowing up, and drops the opportunities into my Slack every few hours. This would at least save me an hour a day. Maybe two.

Building Socialcrawl

I immediately started building. However, I quickly faced a significant problem. Every social platform had a different API, and even worse, returned data in all different shapes and formats. I dug through Apify, RapidAPI, and random GitHub repos looking for anything close to what I needed. The pieces existed, just scattered across a dozen providers with different auth flows, response shapes, pagination, and billing relationships. The fragmentation wasn’t just annoying for me; it was actively making the AI worse. **Plus** none of them were actually providing skills or MCPs for my AI agent to access their docs easily. This ended up with Claude burning tokens, re-learning every endpoint, and every new platform meant teaching it a fresh schema from scratch.

There had to be a better way.

One of my favourite tools on the internet is Firecrawl. It is a YC-backed startup that allows inputting any URL and get a clean AI-ready markdown back. It also provides different functionalities like AI-search, multipage crawling, all from a single API. My biggest reason for loving this service is that it provides a single shape of data, which eliminates parsing headaches. Inspired by it, I kept coming back to the same question: why doesn’t this exist for social media data?

So I built SocialCrawl, on a mission to provide the cleanest, AI-ready social media data on the internet. One API key, one response shape, 21+ social platforms, purpose-built agent skills and MCP servers on top. Then I rebuilt the original agent on top of the new API. The agent now works flawlessly.

See part 2 – What can we do with Socialcrawl?


Share this post!

Leave a Reply

Your email address will not be published. Required fields are marked *