I created this article with the intent of explaining the migration journey from deploying a legacy application with manual steps to an automated Kubernetes deployment with proper DevOps practices. Its intent is not to help you understand Kubernetes deeper (there’s an abundance of materials out there already).
As a Cloud Solution Architect for Microsoft, every week I work with our partners to assist them towards containerization and Kubernetes. I’ll use AKS and discuss it’s strengths and weaknesses without holding punches. Disclaimer: Given I work for Microsoft, I am self-aware of my bias. So in this article, I will make an effort to be more critical of Azure to balance that out.
Beginning With the End in Mind, I created the following outline:
Intent
Duckiehunt is secure, monitored and deployable with the least amount of manual effort, cost and code-change.
Purpose
I wrote Duckiehunt in 2007 as a LAMP website. It embodies many of the customer requirements I see:
Old code, using legacy tooling
Want a reliable, resilient infrastructure
Want to automate deployment
Don't want to re-write
Migration should involve minimal/no code change
Need to update to modern standards (e.g. HTTPS, MySQL encryption, private DB instance with backups)
Outcomes
CI/CD (Code Check-in triggers automated tests and pushes to Production)
Monitoring cluster + app (visualization + alerts if down)
HTTPS enabled for duckiehunt.com (CA Cert + forced redirection to https)
Running on Kubernetes (AKS)
Managed MySQL
Milestones: (in reverse order of accomplishment)
Production DNS migrated
Azure Monitor + Container Monitoring Solution + LogAnalytics
Distinct Dev + Prod environments
VSTS + Github integration
Securely expose UI + API
Integrated MySQL instance
Installed on AKS
Test in Minikube
Migrate App to Container
From here on, I’ll explain my journey as steps fulfilling the milestones I created. I’ll list my estimated time, as along with my actual time to compare. The times below are not “Time to get X workingâ€, but “Time to get X working correctly and automate as if I had to support this in production†(which I do). As a result, they’re much higher than a simple success case.
Migrate app to Container
Estimated Time: 4 hours. Actual Time: 10 hours
I wrote this in 2007 using a PHP version that is no longer supported (5.3) and a framework (CodeIgniter) that is not as active. I didn’t want to re-write it yet. Thankfully, 5.6 is mostly backwards compatible and I was able to find a container using that.
I would have been done in ~4 hours; however, I lost an embarrassing amount of hours banging my head against the wall when I automated the docker build. (I would always get 404) I learned this was because Linux’s file system is case-sensitive and OSX’s is not, and the PHP framework I chose in 2007 expects the first character of some files to start with a capital letter. grumble* *grumble
Test in Minikube
Estimated time: 12 hours. Actual Time: 10 hours
Now that I got my PHP app running in a container, it was time to get it running inside Kubernetes. To do this, I needed to deploy, integrate and test the following: Pod, Service, Secrets, Configuration, MySQL and environment variables.
This is a pretty iterative approach of "This, this…nope…how about this?...Nope...This?...ah ha!...Ok, now this...Nope." This is where Draft comes in. It’s a Kubernetes tool specifically designed for this use case, and I think I’ve started to develop romantic feelings for this tool because of how much time and headache it saved me while being dead simple to use.
Install in AKS
Estimated time: 8 hours. Actual time: 2 hours
Creating a new AKS cluster takes about 10 minutes and is instantly ready to use. Because I had done the work on testing it Minikube the hard-word was already done, but I expected some additional hiccups. Again, this is where my love and adoration of Draft started to shine. I was almost done in 30 minutes, but I took some shortcuts with Minikube that came back to bite me.
Integrated MySQL instance
Estimated time: 2 hours. Actual time: 3 hours
Azure now offers MySQL as a Service (aka Azure Database for MySQL) and I chose to use that. I could have run MySQL in a container in the cluster; however, I would have had to manage my own SLA, backups, scaling, etc. Given my intent of this project is to have the least amount of work and cost, and the cost is still within my MSDN budget, I chose to splurge.
I spent an hour experimenting with Open Service Broker for Azure (a way of managing external dependencies, like MySQL, native to K8S). I really like the idea, but I wanted one instance for both Dev + Prod and needed a high control over how my app read in database parameters (since it was written in 2007). If I was doing more deployments than one, OSBA would be the right fit, but not this time.
Steps taken:
Create the Azure Database for MySQL Instance
Created the dev/prod accounts
Migrated the data (mysqldump)
White-listed the source IPs (To MySQL, the cluster traffic looks as if it's coming from the Ingress IP address)
Injected the connection string to my application (Using K8S Secrets)
Then I was off to the races. OSBA would have automated all of that for me, but I'll save that for a proverbial rainy day.
Securely expose UI + API
Estimated time: 4 hours. Actual time: 20 hours
This was the most frustrating part of the entire journey. I decided to use Nginx Ingress Controller with Cert-manager (for SSL). There’s lots of old documentation that conflicts with recommended practices, which led to lots of confusion and frustration. I got so frustrated I purposely deleted the entire cluster and started from scratch.
Lessons’ learned:
nginx-ingress is pretty straight-forward and stable. Cert-manager is complicated and I had to restart it a lot. I really miss kube-lego (same functionality, but deprecated. Kube-lego was simple and reliable)
Put your nginx-ingress + cert-manager in kube-system, not in the same namespace as your app
You might have to restart cert manager pods when you modify services. I had issues where cert-manager was not registering my changes.
cert-manager might take ~30 minutes to re-calibrate itself and successfully pull the cert it’s been failing on for the last 6 hours
cert-manager creates secrets when it tries to negotiate, so be mindful of extra resources left around, even if you delete the helm chart
cert-manager injects its own ingress into your service for verifying you own the domain. If you don’t have your service/ingress working properly, cert-manager will not work
If you’re doing DNS changes, cert-manager will take a long time to “uncache†the result. Rebooting kibe-dns doesn’t help.
There’s no documentation for best-practices for setting up 2 different domains with cert-manager (e.g. dev.duckiehunt.com; www.duckiehunt.com)
AKS's HTTP application routing is a neat idea, but you cannot use custom domains. So you're forced to use its *.aksapps.io domain for your services. Great idea, but not useful in real-world scenarios
To summarize, I was finally able to get development and production running in two different namespaces with one ingress controller and one cert-manager. Should have been simple, but death-by-1000-papercuts ensued with managing certs for each of them. Now I’m wiser, but the journey was long and frustrating. That might involve a blog post of its own.
VSTS + Github integration
Estimated time: 4 hours. Actual time: 2 hours
VSTS makes CI/CD easy. Real easy. Almost too easy.
I lost some time (and ~8 failed builds) because the VSTS UX isn’t intuitive to me and documentation is sparse. But now that it’s working, I have a fully automated Github commit -> Production release pipeline which completes within 5 minutes. This will save me a tremendous amount of time in the future. This is what I’m most excited about.
Azure Monitor + Container Monitoring Solution + LogAnalytics
*Estimated time: 3 hour. Actual time: None. *
This was the surprising part. All of this work was already done for me by setting up the AKS cluster and integrated into the portal. I was impressed that this was glued together without any additional effort needed.
That said, here’s some “gotchasâ€:
The LogAnalytics SLA is 6 hours. My testing showed that new logs showed up within 5 minutes, but after a cluster is newly created, initial logs would take 30 minutes to appear.
The LogAnalytics UX isn’t intuitive, but the query language is extremely powerful and each of the pods logs were available by clicking through the dashboard.
Monitoring and Logging are two pillars of the solution; however, Alerting is missing from the documentation. That integration is forthcoming, and will likely involve another blog entry.
The “Health†tile is useful for getting an overview of your cluster; however, the “Metrics†tile seems pretty limited. Both are still in Preview, and I expect to see additional improvements coming soon.
Production DNS migrated
Estimated time: 1 hour. Actual time: 1 hour
Since I did the heavy lifting in the “Securely expose UI + API†section, this was as easy as flipping a light switch and updating the DNS record in my registrar (dreamhost.com). No real magic here.
Summary
This has been a wonderful learning experience for me, because I was not just trying to showcase AKS/K8S and its potential, but also using it as it is intended to be used, thus getting my hands dirtier than normal. Most of the underestimated time was spent on a few issues that “rat-holed†me due to technical misunderstandings and gaps in my knowledge. I’ve filled in many of those gaps now and hope that it saves you some time too.
If this has been valuable for you, please let me know by commenting below. And if you’re interesting in getting a DuckieHunt duck, let me know as I’d love to see more take flight!
P.S. The source code for this project is also available here.
Emerging civilizations naturally gravitate towards beds of water. Growing up in lower Louisiana, the Mighty Mississippi was where my ancestry settled. It…
I was recently invited to participate in the Microsoft Partner blog where I shared my love of containers.
I'm especially passionate about container technology because of how much it makes the developer's life easier. Unfortunately, it's one of those things that must be experienced to truly understand. I tried to boil my thoughts town to just a few paragraphs here. Check it out and let me know what you think!
https://blogs.technet.microsoft.com/msuspartner/2017/11/13/how-i-learned-to-stop-worrying-and-love-the-containers/
TL;DR: Size matters.
After Oracle's surprise announcement of their containerization of Oracle DB, Oracle WebLogic and a few of their other core technologies, I decided to test it out for myself. (Speaking authentically, I'm leery of their commitment; however, I recognize that I work on Open Source at Microsoft, so who am I to judge?)
My end-goal is to get Oracle DB 12.2 running in a container on Kubernetes inside Azure Container Service. This is Part 1 of my walkthrough from 0 to operational.
Build and Verify the Container
Unlike most Docker projects, Oracle does not have a public image on Docker Hub. To get started, you'll need to:
Clone their Github repo
Download the Oracle DB instance
Run their buildDockerImage.sh from the Github Repo
Start the container
Clone the github repo
`git clone git@github.com:oracle/docker-images.git
...
Receiving objects: 100% (5643/5643), 425.77 MiB | 5.41 MiB/s, done.
`
Wait...what?! 425MB?!
After some sleuthing, it appears they once included the OracleLinux binaries in the git repo but have not purged them. Poor Github. I have a tremendous amount of appreciation for their architects and support engineers. Below is the SHA1 of the blob, the # of bytes of each file and the path.
`git clone git@github.com:oracle/docker-images.git
Cloning into 'docker-images'...
remote: Counting objects: 5643, done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 5643 (delta 12), reused 0 (delta 0), pack-reused 5607
Receiving objects: 100% (5643/5643), 425.77 MiB | 5.41 MiB/s, done.
Resolving deltas: 100% (3164/3164), done.
git:(master) git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| awk '/^blob/ {print substr($0,6)}' \
| sort --numeric-sort --key=2 | tail -7
35eda80405d711ae557905633d9f9b8d756afb94 42358832 OracleLinux/7.0/oraclelinux-7.0.tar.xz
e359def3dde981199ea692bbb26c24bd37e6fd68 42765288 OracleLinux/7.1/oraclelinux-7.1.tar.xz
0956d25bcb27f804cfc37f2a519a5cfb35af0955 43951872 OracleLinux/6.8/oraclelinux-6.8-rootfs.tar.xz
6de0b5011f509e53623ab0170fbc72e8bb53b501 43953520 OracleLinux/6.9/oraclelinux-6.9-rootfs.tar.xz
b05b9f4971b6d28330545fadc234eb423815dd59 47275816 OracleLinux/7.2/oraclelinux-7.2-rootfs.tar.xz
9b07a976e61ed2cf3a02173bf8c2d829977f2406 49130232 OracleLinux/7.3/oraclelinux-7.3-rootfs.tar.xz
3b7610a3df4892e9cf4f5d01eb3d55bcd3f2ad54 50369896 OracleLinux/6.7/oraclelinux-6.7-rootfs.tar.xz
`
Moving right along...
Download the Oracle DB instance from their website
Since Oracle does not allow anyone else to distribute their software, you must go to their site, register (Larry Ellison now has my email), and download. Unfortunately, the login process does not allow me to "wget" the file and put on a remote machine, so I must download locally via browser. I chose "Oracle Database 12c Release 2"
`-rw-r--r--@ 1 thfalgou staff 3.2G Apr 27 10:07 linuxx64_12201_database.zip
`
Another 3.2GB.
I now have an alternate version of Sir Mix A Lot's infamous song going in my head: I LIKE BIG BINARIES AND I CANNOT LIE...
Moving right along...
Run their buildDockerImage.sh from the Github Repo
The documentation isn't explicit about where to store the downloaded image. (in my case the 'OracleDatabase/dockerfiles/12.2.0.1' directory)
Now the moment of truth. From the "OracleDatabase/dockerfiles" directory, run buildDockerImage.sh
`dockerfiles git:(master) time ./buildDockerImage.sh -v 12.2.0.1 -s
...
Building image 'oracle/database:12.2.0.1-se2' ...
Sending build context to Docker daemon 3.454 GB^M^M
Step 1/16 : FROM oraclelinux:7-slim
---> 442ebf722584
...
Pages and pages of output. So much text that my iTerm buffer no longer had the initial command.
...
Oracle Database Docker Image for 'se2' version 12.2.0.1 is ready to be extended:
--> oracle/database:12.2.0.1-se2
Build completed in 658 seconds.
./buildDockerImage.sh -v 12.2.0.1 -s 3.68s user 8.15s system 1% cpu 10:57.49 total
`
10 Minutes later, the container is finally built. 10 minutes. 10!
Perhaps I'm being overly dramatic; however, the Docker Ecosystem has lots of high expectations and one of those is rapid development and deployment through small, composable artifacts. Granted, building and deploying a new version of database is not a common occurrence; however, the process it not conducive to DevOps. That said, this is their first foray into this, so I'm still excited to see the change.
`dockerfiles git:(master) docker images
oracle/database 12.2.0.1-se2 f788cd5b4b9d 4 minutes ago 14.8 GB
oraclelinux 7-slim 442ebf722584 6 days ago 114 MB
fedora latest 15895ef0b3b2 7 days ago 231 MB
microsoft/mssql-server-linux latest 7b1c26822d97 7 days ago 1.35 GB
nginx latest 5766334bdaa0 3 weeks ago 183 MB
ubuntu latest 0ef2e08ed3fa 8 weeks ago 130 MB
...
`
14GB? I take that back.
Start the container
Let's get the party started...
`dockerfiles git:(master) docker run --name oracledb -p 1521:1521 -p 5500:5500 oracle/database:12.2.0.1-se2
ORACLE PASSWORD FOR SYS, SYSTEM AND PDBADMIN:
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 28-APR-2017 03:21:48
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Starting /opt/oracle/product/12.2.0.1/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /opt/oracle/product/12.2.0.1/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/91c68ac2b2bf/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
...
Copying database files
1% complete
...
`
Huzzah! After about 9 minutes, it's finally started! Let's test it!
`~ docker exec -ti oracledb sqlplus pdbadmin@ORCLPDB1
SQL*Plus: Release 12.2.0.1.0 Production on Fri Apr 28 03:58:10 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
SQL>
`
We're in!!! It worked!
It is at this point that I realize I've already gone through 2 drams of Aberlour and I should probably stop for the night. Provided there is enough interest (and whiskey), I'll write-up Step 2 of getting this running on Kubernetes in ACS. As for now, I should stop while the world is only mildly spinning.
NOTE 1: If the database auto-generates a password with a "/" in it, I've found it doesn't work. You can change that by running:
docker exec ./setPassword.sh
NOTE 2: If you run this multiple times, make sure to run "docker system prune" as it fills up your disk fast. On my 3rd try, I hit the following error, even with lots of space on my disk.
` The location specified for 'Fast Recovery Area Location' has insufficient free space.
CAUSE: Only (9,793MB) free space is available on the location (/opt/oracle/oradata/fast_recovery_area/ORCLCDB/).
ACTION: Choose a 'Fast Recovery Area Location' that has enough space (minimum of (12,780MB)) or free up space on the specified location.
`
NOTE 3: It looks like everyone uses Docker now...
iPody Dude. Originally uploaded by SnoopyKiss. BIG UPS! to Ruby for accessorizing one of my favorite toys. 9 Guests last weekend, half of which I never…
Once
again, I decided to make game of this crazy world and venture off into the
unknown by myself. I had the best time going by myself to a hostel, so I
should have just as much fun same this time, right? Nope.
My trip to Brugges was almost cancelled because there was an accident
at Gare du Nord (my take off station). Luckily, an old Frenchman took me
under his wings and helped me get to another Metro where I was able to make
it to my train with only 5 minutes to spare. When I got to Brugges, I went
to the "Johnny Rockets" hostel, because the description and name seemed fun
enough. I took the taxi and walked inside the bar and asked the bartender
for a room. It seemed pretty noisy, but I'm sure that I would enjoy myself.
After the paperwork and credit card, I hear the words that will forever ring
in my ear as The signal that you're getting a bad room. "You're
in Room...1." Not knowing better at that time, I took the key and walked
upstairs to the unlit, stair hallway and blindly felt myself though the door
way. I finally opened up a door and the motion detector lights came on.
And true enough, I was the ONLY soul there. I found my room, opened it up
and plopped my stuff down and went off for a restaurant. Dinner was pretty
good. I then explored Brugges and found that it's a really romantic city at night with
the nicely lit buildings and waterways. Tired and full, I went back "home"
and crawled in bed...going to sleep took a lot longer because I could hear
the music downstairs going strong.
The next day, (Unfortunately, a rainy day, but that did not douse any spirits.)
I packed everything up and took a bus tour of Brugges and was quite impressed.
Regrettably, I did not see it; however, in one of their buildings, they have
in a metal container, the "Blood of Christ." They also have a beautiful area with a statue of Neptune in the middle. The city is surrounded by many canals, which split up the city wonderfully.
Points of interests in Brugges: The main attraction is the Tower.
(I forget the name.) Every 4th store is a Lace store since it is a huge
tradition in their town. And inside every other lace store is a tapestry
store. If I remember correctly, there are 20 tapestry factories in Europe
and 9 are in Brugges. So, if you're going to get lace and tapestries, go
to Brugges.
After enough touring, I decided to make a change in plans and go to Brussles.
Home of the EU (European Union) and my favorite tourist place so far: Mini Europe!
(The leaning tower IS leaning, but the picture was taken the wrong way.)
When I got into Brussles, I was completely lost (AGAIN!). I quickly found
an Ibis hotel (4 stars and in the Nortel travel recommended hotels list)
and plopped my weary arms and legs in bed.
The next morning, I had some Belgium Waffles (very good with some chocolate syrup.); visited the Grand Market; did some window shopping; and enjoyed some of the local artwork. Afterwards, I took a bus tour to the Atomium (A
100+ meter monument built for the 1950 Olympics) and then the neatest place
so far: Mini Europe. As it's name suggests, all of the monuments of Europe
are shrunked and placed in this decent sized area. There's the Eiffel Tower, Sienna, off shore rigs, Brugges, Brussles, Arc du Triumph, Venice, The Berlin Wall, Greece (?), an oil repository on fire, and a lot of other places I have not been to yet. It was; however, a really great feeling to see some of the places which I have already been to and places I have yet to see. I then hopped on the tour bus again and went to see some Chinese houses which were imported to Brussles as monuments. (Most impressive.) It made me realize that Asia is the next place I want to visit.
No journey to Brussles can be complete without seeing the "Manneken Pis".
Legend has it that when Brussles was under attack, the king's son releaved
himself on one of the attacking cannons, thus saving Belgium. A statue was
created of the boy, of course...peeing. Who knew that urine could also be
used to save lives? Also, the boy is dressed up in various costumes throughtout
the year. (Long live the King.)
At first, I thought that I would not enjoy myself as much in Belgium,
since I had this thought that it was a boring country. However, Europe has
not let me down yet. I continue to explore new places and enjoy myself each
time I venture out into the unknown. Next weekend will be great because
"Father Falgout" (aka Dad) is coming to visit for 2 weeks and we shall paint
Europe red!
Cheers!
P.S. Here's the link to the entire album. There are lots more pictures from Mini Europe.
