I maintain a Python scraper that collects daily front-page headlines and metadata from Argentine news outlets, supporting sociological research at OSPO (Observatorio de Medios, Facultad de Humanidades, UNMDP). That data feeds monthly analysis of the city’s headlines, plus occasional deep dives into specific topics, and we’re now enriching it with AI, tagging each headline with the actors, topics, and locations involved. It didn’t start on a server I own, it started on AWS Lambda + S3. A few months ago I moved it to a self-managed VPS, and the reasons behind that move ended up more interesting than the move itself.
Why AWS Lambda First
Lambda made sense at the time. No server to provision, no OS to patch, just a function that wakes up on a schedule, scrapes a handful of sites, and writes the results to an S3 bucket. For a research project with an uncertain future, that low commitment was the right call.
Why I Actually Moved
I didn’t move because Lambda was technically wrong for the job, and I didn’t do it to save money either. A VPS isn’t necessarily cheaper than what I was paying on AWS, sometimes it’s about the same, sometimes it’s more. What forced the decision was the free trial period running out. Once I had to choose, I chose to pay for a VPS instead of continuing with AWS. What I was really buying wasn’t a lower bill, it was a flat, predictable price instead of one that changes depending on what I decide to run next. Not having to think twice before trying something new mattered more to me than saving money.
I Already Had Somewhere to Put It
That decision was easy because I wasn’t starting from zero. Picking the VPS provider itself took me a while, comparing options before settling on Netcup, a German hosting provider. I’d already been building my own environment there, documented in a separate server-docs repo, for personal projects, dotfiles, and experiments. Moving the scraper there felt less like a new infrastructure project and more like bringing one more tenant into a house I was already maintaining.
The migration itself wasn’t hard at all. The scraping code moved over, the cron schedule moved with it, and instead of writing to S3, results now sync out to Google Drive with rclone as a lightweight, good enough backup and access layer.
That last part mattered more than I expected. The researchers using this data aren’t developers. They don’t want an AWS console or S3 credentials, they want a folder of CSVs they can open. Syncing to Google Drive turned “ask me to pull the files” into “they’re already in your Drive,” a small technical change with a bigger effect on how usable the data actually is for the people it’s for. AWS probably has a smoother answer for non-technical access than handing someone raw S3 bucket permissions. I just never needed to find it, because Drive was already the tool my collaborators knew.
Infraestructure Started to Feel Like Mine
It became the home for more than one project
Once the VPS was running my scraper reliably, it stopped being “the machine for one project” and became the place I run things in general.
The clearest sign of that is the second scraper sitting next to the first, with nothing to do with it. Where the OSPO one collects Argentine headlines from a plain crontab and syncs CSVs out to Drive, this one covers Basque and Spanish outlets, fires from a systemd timer twice a day, and writes into a Postgres container instead of flat files. Same box, completely different shape. Two Telegram bots run there too, and what started as a single Grafana turned into Grafana plus Loki plus Promtail.
That last part is the piece that ties everything else together, and it’s the one I’d have had the hardest time justifying on Lambda. These four projects don’t log the same way at all. The two scrapers write to files, the two bots go through journald as systemd services. Promtail collects both, parses timestamps and log levels out of the raw lines, and tags everything with the project it came from, so by the time it lands in Loki I can query across all of them with the same labels. Four things that agree on nothing, readable in one place.
None of that was planned. Owning the box lowered the barrier for the next automation idea, and then for the one after that. There was nowhere else those ideas would have gone.
Not everything ends up there, though. Shortly after writing this I put together a watcher for a government appointment site and deliberately kept it on my laptop instead, because where the request comes from mattered more than having it run 24/7.
The terminal got a lot more comfortable
I open the VPS often enough that I gave it its own tmux shortcut, using mosh for a connection stable enough that flaky wifi (looking at you, Argentina’s Claro) doesn’t kill the session:
bind S if-shell 'tmux has-session -t vps01 2>/dev/null' { switch-client -t vps01} { new-session -d -s vps01 -n mosh "mosh vps01 -- sh -c 'tmux a || tmux new'" new-window -d -t vps01: -c "$HOME/repos/server-docs" select-window -t vps01:1 switch-client -t vps01}One keybinding and I’m inside a live mosh session on the server with server-docs open in a second window. It’s a small thing, but it’s the kind of small thing that decides whether I actually check on a system or just avoid it.
AWS’s real convenience
I want to be fair to Lambda here. Not having to think about the environment at all is a real advantage, and for teams shipping fast or without ops capacity, it’s often the right trade-off. It just wasn’t what I was optimizing for. I was already comfortable owning a Linux box, and honestly, part of the migration turned into pure procrastination getting Neovim properly configured against Debian’s packaged versions of everything. Time that had nothing to do with the scraper and everything to do with enjoying the environment I’d just taken over.
That comfort wasn’t a coincidence. I’d used Debian as my daily desktop OS for about a year before this, so none of it was new. I already knew I’d need to compile a couple of tools from source instead of grabbing them from a repo, and I already had the habit, learned via PEP 668, of not touching global pip install. Putting Debian on the VPS wasn’t a new experiment, just familiar ground.
The tests CI can’t run for me
Most of the suite runs in GitHub Actions on every push, and I want to be clear that this has nothing to do with the migration. It worked exactly the same when the scraper was a Lambda. Where you deploy and where you run unit tests are separate questions, and I’d be taking credit for GitHub’s work if I said otherwise.
The interesting part is the tests CI deliberately doesn’t run. The integration ones make real requests to all nine news sites and check they still respond and still parse the way the scrapers expect. They’re marked integration and excluded from CI on purpose, because a GitHub runner hitting Argentine news sites from a datacenter somewhere else tells me very little about whether my scraper works. I’d get flaky failures, rate limits, and the occasional block, none of which reproduce the conditions the scraper actually runs in.
That’s the case where owning the box genuinely changes something. The machine the cron job runs on is a machine I can SSH into and run the suite on directly, over the same network path, from the same IP, in the same environment. I’ll be honest that I haven’t turned this into a scheduled job yet, it’s still something I run by hand when a site changes its markup and things start coming back empty. But it’s an option that only exists because there’s a persistent machine to run it on, and it’s the thing I’d reach for first when a scraper starts returning nothing.
Watching it run without opening the AWS console
I run Grafana on the VPS, and it gives me a dashboard I actually open, a quick view of whether things are running and healthy. The AWS console never earned that habit from me. It wasn’t uncomfortable in any dramatic way, just enough friction that for something I want to check often, it lost out to a dashboard that’s one bookmark away.
Working on the server from local AI tools
Being able to open the VPS directly from Claude Code or OpenCode in my local terminal has been useful for configuration and troubleshooting, with the obvious caveat that I read every command before it runs against a production box. That habit has saved me from at least one bad idea, and helped me fix a real issue faster than digging through docs alone would have.