February 6 on vAMSYS - or - how not to update PHP

[Lukas Jankauskas]

I had to double-check, as I thought it's the time for monthly update, but I posted one just two weeks (give or take) ago on 24th Jan. You can read it here btw if you have not already. 

For this out-of-order update I thought I would keep it light and blog-like (read - unstructured). I sometimes like talking to ether and it seems some of you like to read my ramblings.

vAMSYS has always been a love-hate relationship for me. It was not something I started or wanted. It was foisted on me in away and I accepted it out of duty and the potential it had. Little did I know back then that this joy would lead to headache years later. Back then it was something I just had to do... This sense of duty has remained - i.e. Lukas, you cannot shut down vAMSYS and just live your life. I think that has, in a way, led to dissatisfaction and disinterest along the way. I need not lie, as you can see the code change history, that there have been months without code changes at a time.

I don't know about you, dear reader, but recently I have yet again found myself looking at vAMSYS with enthusiasm and untapped potential which I have not felt for... let's say a while. And it has been good. Welcomed even.

So lemme talk about what's new since two weeks ago, yet feels so long ago that prompted me to write this post. 

 

Bye bye Bugzilla, hello Vision! 

As I recently alluded in Discord, I was never a huge fan of bugzilla - it did it's job and I appreciate George setting it up, maintaining it and writing forum posts about it - It just never clicked with me - it was something out of 90s and more functionally dense than what we actually needed, yet not in the way I wanted it. 

And so it was that bugzilla was my third cousin I never liked, but had to interact with because it's how we chose to organise vAMSYS communication. Not saying it was a bad idea - vAMSYS has grown to the stage where we do need a centralised bug tracking system. I just always felt we could do better.

Vision by vAMSYS was this little side project I started to get even more acquainted with Tailwind, Livewire and Alpine with the help of Laracasts (speaking of, if you ever wondered how I know what I know - it's cause of Laracasts. I will leave you to your own conclusions whether I know a lot or very little) to help progress with v4ification of vAMSYS. And I - maybe you too - love Vision. 

It's one of the reasons I feel so up-beat about vAMSYS again - I got to look over some good ideas I can implement and over the 48 hours since it's been launched Vision has already lent itself to very productive discussions on Discord over the direction of vAMSYS. Not to mention I like how it looks and the fact it is tailored to our needs - issue with voting we had in the past, for example, with Ideas system - eliminated in Vision as Owners and Staff now vote on behalf of their airlines. Vision is a nice little system with potential to change as our operations do. I loved coding it and I love to see you using it and discussing the ideas it results in.

 

How to improve vAMSYS and break it at the same time

Or - to rephrase - vAMSYS works really well for me and loads super fast when nobody else can use it. This will be a short story how a long Sunday goes awry very quickly and how bad decisions lead to prolonged downtime.

Let's start at the beginning. At home, I am lucky enough to have a spaceship of a PC setup:

For those interested... It's... Well... decadent and I am very lucky to be in this position - AMD 7950X, nVidia 4090 FE, 128 GB of RAM, one LG super ultrawide and 2 LG ultrawide monitors. I will not lie - it is amazing to have so much screen real-estate when working on anything.

Back on topic - I do most of the coding at home during my downtime. Quite often I will find myself tweaking some code on my HP Envy laptop whilst at work or away or on the way to home. This leads to the need of having a common development environment between the computers I use. Up till Vision came along, I used Laragon - it has been amazing.

However, with so much compute power and other benefits it entails, WSL - Windows Subsystem for Linux - became a better choice and it was my go-to when developing Vision both at home at on the move. I fell in love - the power of Ubuntu (same strong stuff powering our beefy servers) running on a windows machine just made things so much easier and faster. Naturally, Vision is running on PHP 8.2. Why that became a problem will soon be obvious.

Fast forward to Sunday yesterday - it's time to host vAMSYS Sunday. I do the few bits with Vision and it's time to move on to vAMSYS and VDS work. Naturally, I try and git clone Thunder (codename for vAMSYS codebase) into the WSL ubuntu, but it does not seem to clone for some reason. No biggie - we still have Laragon, so let's work there. Awesome - three hours later vAMSYS Sunday affair is done - thank you everyone who has joined! Time for my break to clear my brain a bit. As is always the case with vAMSYS Sundays, if I have no other errands, it's back to coding. This time, let's figure out what is going wrong. Screw PhpStorm clone tools, we'll do this properly this time and clone in command line. So far so good. 

I return. Have some issues with npm install - FontAwesome Pro is giving some issues as it's not configured at the WSL instance - I muddle through and I do composer install to ready up Thunder. In the anatomy of 'Where things went wrong' this is the first entry. Since it was a brand new copy, there was no composer.lock and it just picked up the latest and greatest compatible stuff with php8.2 which is running on my WSL - it has been a while, all dependencies we use are 8.2 compatible by now so composer does not give a squeak or a beep. Problem is - vAMSYS is not tested on 8.2 - it's last configured to run on 8.1...

So as one goes, I do my thing here and there working on all sorts of stuff (Later on that day I improved my select VA page load times by a factor of a lot) 

It's now time to commit something I was working on. Envoyer does it's job and all of a sudden... Oops.

 

Ok. Panic mode. Do we revert to last working version, which will take around 30 seconds or just update the system? The correct answer is, has been and always will be - you guessed it - revert to last working version. You wanna guess what I did? Well, you already know:

I thought simple. Quickly Teleport into 3 VMs responsible for web rendering, apt-get install php8.2, update and restart nginx to use 8.2-fpm and you're done - 5 mins tops, no biggie. 

Do I wish I was right. The entire affair took 39 minutes and not all of it is to be blamed on me, I must say. PHP 8.2 install went well - no issues there - installed, FPM running seems good. Switch nginx and stuff seems to be working. Yay. But no yay - my alerts are going off - it works on one of the web servers. Ok - let's re-deploy. Re-deploy gets stuck. Restart server - still stuck. It times out after 5 mins. Ok - worst behind us now. Let's redeploy. Redeploy fails - missing PHP extensions - fffffreaking awesome. Ok, let's install php8.2-pgsql on 3 Virtual Machines and redeploy. Redeploy gets stuck again for server 2. I am now rethinking my life choices and looking at skyscanner for flights to Cuba. Deploy times out again - third time lucky - yes, deployed successfully. I take a breath of air and stop mid-gasp as the familiar cloudy page with 500 error is replaced by the one of chrome default where the page could not resolve. I cry inside.
Couple of minutes going through nginx and php logs - file read access denied? What gives? Ooooh, our php needs to use vamsys and not www-data user. Let's fix php config; at the same time I remember to fix execution time, max upload and post sizes in settings - it would've been a problem later on if I did not do it now; whilst there-ish, let's also fix pool config so more than 10 people could load vAMSYS - and presto 39 minutes later we are back fully online (mostly).

 

Now, it is a jumble up there - and if you read it fast enough you will get closer to wearing my shoes that evening - it was busy, confusing and incoherent - welcome to server ops. As a courtesy to you I have omitted all the dead-ends I went to and details of log perusing which led me astray. I will say one thing - had something like this happened 2 years ago - it would have taken longer than 39 minutes to recover.

Nonetheless - lessons have been learned - till next php update that is.

 

OpsGenie to the rescue

Before I wrap up today, I just wanted to touch on our reporting system real quick and hopefully demystify the thing. 

George has hooked us up with Newrelic APM - Application Performance Monitoring. We have it tied up with OpsGenie. The result is - if vAMSYS is down - whether it's something we did or a server or a component of it failed - we get to know about it really quickly by notification in private discord channel and a phone call. There is no way we can be unaware of vAMSYS being down unless George and I are both on a flight or in a tunnel at the same time. 

That is why George and I have never met and never travel together - it's a security risk to vAMSYS 😄

In all seriousness, most of the time we know that vAMSYS is down before you do or your pilots mention it on your Discords. If it happens at 5am, it might take us a little to get out of our beds and fire up the PC to write a message in the #announcements, but when there is a total failure - we know and have been woken up because of it.

So let's all do ourselves a favor and stop playing the game of FIRST! and instead just:

 

 

That's all from me this time - again, sorry if it was poorly written with spelling and grammar mistakes all over the place - I did say this one is gonna be more blog-like as it's not a monthly update - I tend to care more about those 😛

See you next time!