My First Job

I learned to code on the job without any training in a production-only environment. Looking back, it’s a miracle that I survived. My team was able to run the Django servers for our company’s websites locally, but that’s about where our separation from the real services ended. All of our integrations with other apps were hooked up to primary instances, and, worst of all, we developed while connected to our production database. This meant we tried every table migration for the first time on real data, as well as every new feature that included a write-operation.

Many of us were aware of these problems, but sadly it wasn’t as simple as just hopping into the configuration and changing a database connection. Everything was hardcoded. The database credentials were locked into our settings files (where anyone with access to the git repository could take a peek), and we even had logic inside the code dependent on those particular credentials. Addressing this wasn’t just a simple refactor that could be knocked out when someone on the team had the time; our codebase needed a full shakedown, with thousands of lines of legacy code potentially affected. And many of us were too scared to touch it since no one understood how the live server worked as a whole.

Linux, Apache, MySQL, and Python (PHP, Perl)

After about a year of working in this high-stress environment, I was entrusted with the responsibility of maintaining the Django servers discussed above. Once the keys were in my hand and my username applied to the right LDAP group, I quickly set to work on learning more about the architecture so that I could improve it. I wanted to escape the rough position we were in and make maintenance easier—not just for myself, but for the developers who would work on the code after me. We had a typical LAMP set-up: a Linux virtual machine with Ubuntu installed, Apache for serving our code over HTTP, MySQL for our database, and the business logic written in Python (using the Django web-framework). With all of this common technology at play, I knew there had to be already existent solutions for separating our development environments from production; I just had to dig, read, and be patient.

The Legacy System

With my newfound power, I SSHed into the live servers and started taking notes. I came across things that I had no idea existed, but without which our whole infrastructure would have come crashing down. The more I found, the more I realized how fragile our setup was—and the more I understood why no one wanted to fix it. Some applications only worked because someone had SSHed into the server a few years back, installed some program to fix a problem or enable a feature, adjusted some environment variables, fiddled with the .bashrc, and signed out without making note of any of this. What was left was a landmine for posterity to stumble across when a change was made that inexplicably broke what should have been an unaffected feature.

A few weeks later, with my mind more seasoned and my attitude surprisingly more positive than before, I had developed a new philosophy: we needed to be able to run the entire server locally without a single reference to production—to be able to connect to a local database and run migrations and test code that performed write-operations without even grazing our real data. We also needed to be able to spin up development and staging instances of our servers where our code changes could be tested in a production-like environment and shown to our non-technical coworkers—meaning no more deploying a feature to the live website just to get feedback on it!

The Refactor

It wasn’t long before I had what I wanted. Though it took a lot of effort and a couple of months of work (where not a single new feature was made), I finally had everyone on my team off of the production database for development. I rotated and encrypted the credentials so that the interns couldn’t accidentally drop a live table, and we even had staging instances set up with isolated datasets so that testers could try out our new ideas worry-free. The codebase was now configuration agnostic. Production grew more stable, maintenance became easier, and development went from stressful to fun. We could also ship out new features more quickly because we didn’t have to configure a specific data state in production just to try out our code; we just ran a command to set our local SQLite files to a state useful for development.

I was nervous about justifying spending all of this time on code that had already been written to my superiors, but luckily my team backed me up and we soon had everyone convinced that following my new tenets had been worth it. For those familiar with Greek philosophy, I had made an appeal to Plato and decided that forms were more true than objects, that the shape of the server was more important than the actual server—and that our actual server would work better if we improved the structure behind it. Our servers were no longer something that only worked when the stars aligned; they were now something “recreateable,” the particular virtual machine running our Django code no longer sacrosanct but simply a manifestation of a higher form.

But many of these values were challenged and stretched when I changed jobs and started working at Purple Technology. While the underlying principle about forms being more important than objects remained the same, it was taken to new levels when I learned my new company’s codebase, and I quickly recognized how small of a step towards this principle my efforts at my last job had been. The tech-stack we were using at Purple Technology was like nothing I had used before, and I soon learned about the concept of infrastructure as code. This idea had been what I was unknowingly reaching for back when I had attempted to reshape our systems for my Django gig, but since the only architecture I had ever known was LAMP-based, it had never occurred to me that a completely different paradigm existed—one that took my ideas much further.

Serverless

At the new job with Purple Technology we wrote everything in Typescript and served everything in the application via the Serverless Framework. Our infrastructure was a complex network of AWS microservices and Node containers encapsulated in a single YAML file. There, we specified everything about the DevOps process you could imagine: API endpoints, static asset management, frontend configuration, permissions policies, cron jobs, state machines, and database table schemas. To anyone who’s ever spent hours configuring Apache, this is an impressive show of force. Needless to say I was taken away, but I was quickly disappointed the first time I wrote code for a new feature and asked my lead, “How can I run this locally?”

“You can’t,” was the answer. “You need to deploy it to a development instance and try it out there.”

Since my feature involved several different AWS services, testing the full application from front to back locally on my laptop with the serverless-offline plugin wasn’t going to cut it. Though it was at least isolated from production, this felt like a major step back from my efforts at the last job to make sure every piece of code could be run locally on the developer’s workstation, and I contemplated this disappointment for the entirety of the 30 minutes it took for AWS to spin up a new CloudFormation template that contained my measly 20-line code change. And it didn’t make me feel any better when, after finally testing the code change in the deployed environment, it didn’t work because I had a boolean backwards in one of my if statements. I then had to wait another half hour after making a one-line-fix to run it again and make sure my code worked. To say I was feeling mixed emotions regarding the architecture transition would be a polite way of putting it. I was starting to miss Django.

The Era of Microservices

What I had to realize was that when your architecture involves both your code and integrated microservices, you have to give up on the idea of running the entire application locally on your laptop. While this concept was precious to me with the Django servers, it wasn’t tenable for something like an AWS-based system. As modern software development continues to evolve, integrations will become more and more essential to the core of our applications. In the same way we hardly write our own sorting algorithms anymore (since many have already been mathematically perfected), we will one day completely stop writing our own code for common patterns like user sign-up and email verification. Instead, we’ll integrate our applications with services that do it for us. And when that day comes, it will likely be because most development teams will have moved on to cloud-based systems, where pulling in such an integration will only require a few lines in your serverless template.

Similar approaches of course exist in the LAMP world to a certain extent. You can use something like Ansible to map out your Linux server’s environment or write a bunch of bash scripts to automate set-up and configuration tasks, and you can easily spin up multiple instances by creating docker or LXC containers, but if you’re willing to do all of this work, you’ll still come across the pains that come along with babysitting a physical server or virtual machine: software version upgrades, fighting the Apache or NGinx configuration, managing your logs, implementing settings you don’t understand at the behest of your security team, and whatever other problems you come across due to managing your own hardware. The alternative is a YAML file and an AWS account.

New Development Patterns

Before long I had a work-around to my frustrations with the AWS deploy times being the bottleneck in testing my code. The answer was modularity, something which I had always been a proponent of, but of which I became an even stronger advocate once I saw how it enabled development with Serverless. While it’s true that there are some things you just have to deploy to fully try out (Cognito triggers and Step Functions, for example) if you’ve divided up most of your code into separate files, classes, and functions, it shouldn’t be difficult to run the bulk of what you’re building locally. As for the part where you need to deploy something into the cloud to test it, that can be kicked off before your coffee or lunch break. While not ideal, it’s a small price to pay.

Though there were more growing pains after my first experience with the first development deployment, it wasn’t long before I began to see how much closer towards my original philosophy this new infrastructure was. Our set-up was even more agnostic than the Django system I had previously built. While I had been so proud that those LAMP servers could be run anywhere, our systems at Purple Technology can run nowhere. We don’t need to provision any virtual machines or configure Apache, much less care about a physical server: we just need our repo. An entire application with a frontend, RPC and GraphQL APIs, list of scheduled tasks, queuing and notification systems, user pools, and state machines can spawn into existence by running a single command.

Plato’s Forms

Different approaches to architecture of course have different advantages and disadvantages, and the right solution always depends on the particular application in question, but as far as my initial goal is concerned—my dream of being able to spin up my team’s codebase whenever and wherever I wanted—the Serverless framework is definitely a step further in that direction. In answer to the question of, “What is more real, the form or the object, the class or the instance, the idea of something or the thing itself?” the Serverless framework answers with the former. The truth lies within the form of something, not its manifestation.

At my first job, I had worked tirelessly so that the form of our server would just be a git repo, an Ansible playbook with our Linux and Apache configuration, and a virtual machine provisioned with the right Python version—but at Purple Technology with the Serverless Framework, that higher form is just a simple serverless.yml.