Web Scraper
A browser automation service that is core to Optimeleon's AI product, not a side tool. It moved off a restart prone hosted platform onto our own in-house browser management, and its slowest endpoint, element and CTA detection, moved off AI onto pure code I built, cutting response time from minutes to seconds and making it reliable for the first time.
Optimeleon's AI product needs to see a customer's website the way a person would: a clean screenshot, plus a correct read of its calls to action and key elements. This scraper is core infrastructure the product depends on, not a side tool. Early on it ran on a hosted platform (Apify) that rebuilt its entire runtime environment and launched a fresh Chromium instance on every single request, so one response could take one to three minutes. Once basic scraping was solved, a harder problem surfaced: getting a machine to correctly find CTAs and key elements on real websites, most of which do not use clean semantic HTML.
- I led moving element and CTA detection off AI and onto pure code inside the scraper itself. Elements are resolved into legible sentences rather than matched against raw HTML tags, since a lot of real sites don't use correct semantic markup. This is the highest leverage piece of work in the project.
- I built pop-up removal, backed by our own browser extensions, so screenshots and page captures come back clean instead of showing cookie banners and consent modals.
- I built lazy-load handling so pages that load content on scroll are fully rendered before capture, instead of coming back with blank sections.
- Day to day, I owned keeping the production server reliable, held a share of ownership of the service as a whole, and made a number of smaller performance decisions along the way.
- The bigger architectural calls, leaving the hosted platform, the browser engine migration, the anti-bot stack, and adopting Kernel and BrightData, were our CTO's. My ownership was concentrated on the data fetching and extraction layer, and on keeping the service reliable in production.
Element and CTA detection now run in under 30 seconds, down from over 5 minutes on AI, and rarely fail even a year on, where the AI based version broke every 1 to 2 weeks. The scraper has processed more than 100,000 ecommerce, landing page, and product detail websites, and the extraction logic it runs is core to the Optimeleon AI product today.
- AI is not always the right tool for a well defined extraction problem. Once the target output was fully specified (find this CTA, find this element), deterministic code beat an AI call on speed, reliability, and cost.
- Real world HTML is not semantic. Extraction logic has to resolve elements into meaning, not just match raw tags, to hold up across sites that don't follow correct markup conventions.
- Reliability is made of small, unglamorous things. Handling pop-ups, lazy-loaded content, and malformed pages mattered more to real-world reliability than any single clever optimization.
- Owning a production service, not just a feature, changes what you optimize for. Thinking about what breaks at scale, restarts, timeouts, browser leaks, mattered as much as making any one request fast.
