WARNING: SSH’ing into an agent node is an anti-pattern and should be avoided. However, we don’t live in an ideal world, and sometimes we have to do the needful.
This walkthrough creates an SSH Server running as a Pod in your Kubernetes cluster and uses it as a jumpbox to the agent nodes. It is designed for users managing a Kubernetes cluster who cannot readily SSH to into their agent nodes (e.g. AKS) does not publicly expose the agent nodes for security considerations).
This is one of the steps in the Kubernetes Workshop I have built when working with our partners.
It has been tested in AKS cluster; however, it should also work in other cloud providers.
You can follow the steps on the SSH to AKS Cluster Nodes walkthrough; however, that requires you to upload your Private SSH key which I would rather avoid.
If you’re paranoid, you can generate your own SSH server container; however, [this one by Corbin Uselton](https://github.com/corbinu/ssh-server) has some pretty good security defaults and is available on Docker Hub.
kubectl run ssh-server --image=corbinu/ssh-server --port=22 --restart=Never
Setup port forward
Instead of exposing a service with an IP+Port, we’ll take the easy way and use kubectl to port-forward to your localhost.
NOTE: Run this in a separate window since it will need to be running for as long as you want the SSH connection
kubectl port-forward ssh-server 2222:22
Inject your Public SSH key
Since we’re using the ssh-server as a jumphost, we need to inject our SSH key into the SSH Server. Using root for simplicity’s sake, but I recommend a more secure approach going forward. (TODO: Change this to use a non-privileged user.)
Using the SSH Server as a jumphost (via port-forward proxy), ssh into the IP address of the desired host.
# Get the list of Host + IP's
kubectl get nodes -o json | jq '.items.status.addresses.address'
# $USER = Username on the agent host
# $IP = IP of the agent host
ssh -J email@example.com:2222 $USER@$IP
NOTE: If you get “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” You might need to add `-o StrictHostKeyChecking=no` to the SSH command if you bounce across clusters. This is because SSH believes that the identity of the host has changed and you need to either remove that entry from your `~/.ssh/known_hosts` or tell it to ignore the host identity.
This week, I found myself in one of the most unique and challenging situations of my life. And now that it’s all over, I find myself in tears. Not because of sadness, but because I now know myself as someone who can actually make a difference is this world, despite the circumstances.
Now for a little backstory.
It should be no surprise that I love to build. I found my best friend, Lee Gibson, when a LEGO set came up at a White Elephant party and we both schemed on how to win it. I’ve created a non-profit called “The Trebuchet Society”, with the primary goal of hosting SlingFest, a (mostly) annual event designed to gather builders from around the area to create trebuchets and toss pumpkins hundreds of feet. It’s a blast and fuels my desire to build and be around other builders.
In 2014, I discovered TheLab.ms via a tweet. A budding Makerspace/Hackerspace. Its mission is to foster a collaborative environment wherein people can explore and create intersections between technology, science, art, and culture.
I found my people.
Their guiding principles were more focused on education and ethical hacking instead of building trebuchets, but that’s cool. My mom was a librarian, so education is in my blood. I just wanted to be around like-minded people.
As with all non-profits, you want awareness, engagement and members. These usually bring in new ideas and fresh blood. Sometimes in alignment with your own ideas, sometimes not. And as a father, I can tell you, there is no rage in the world like watching something happen to your baby.
Fast forward a few years and after some leadership changes, the last of the founders resigned as a board member, and a number of positions were either vacant or MIA. Then the Education Coordinator resigned. Then the President resigned. Then the Floating Board Member. And the Vice President. And the Secretary.
Their reasons were their own. And I support them 100%.
I was now in one of the most unique and confronted situations of my life. The sole Board Member of TheLab.ms. A community that I’ve been with from almost the very start and loved so dearly was fighting amongst itself. Anger and frustration was evident on a daily basis. People were burnt out.
Thankfully, I had an ace in my pocket. For the last 6 months, I’ve been registered in a course called “Team Management and Leadership Program” from Landmark Worldwide. It is a course designed around creating teams and teamwork in any situation that produce powerful results in many areas of life with freedom and ease. I called my coach and the classroom leader in tears that day. I felt completely broken down and had no idea how to make this work. Through an insightful and “tough love” conversation, I started to see a path forward.
I organized a last-minute event and invited people to create the future of TheLab. I expected about 6 people to show up. I had to hold back my emotions when the room completely filled up, including members I hadn’t seen in years. These were people who, despite the burn-out, despite the anger, despite the frustration, deeply wanted TheLab to not just survive, but to thrive. It was showtime.
In an hour and a half, we dug deep, asked some good questions and had some fun. We had some deep, meaningful conversations about the future and not the past. And most importantly, people stepped up to the plate to take on some big leadership positions. Elections are next week and I invite all of you to learn what we’re about. I have never been more proud to be part of an organization than I am right now.
Emerging civilizations naturally gravitate towards beds of water. Growing up in lower Louisiana, the Mighty Mississippi was where my ancestry settled. It was a source of commerce, livelihood and fisheries which provided sustainability that allowed the surrounding areas to flourish to the ecosystem it is now.
Technology mimics this cultural expansion and KubeCon/CloudNativeCon is the riverbed where developers and operators around the world arrive to ship and receive containers from the Kubernetes dock.
I was fortunate enough to join 50+ other Microsoft’ers and 4000+ others KubeCon/CloudNativeCon on Dec 5-8th. This hotbed of activity has flourished from the internal foundational work that Google created to a vibrant open source community. This small stream has gathered enough momentum to be undeniable in the development and operations community.
Kubernetes is software that makes it easier to run your software. Software development is hard, not just because you have to worry about your code, but you also have to worry about monitoring, maintaining, updating, scaling and more. Kubernetes was the pilot program for a larger organization called the Cloud Native Compute Foundation. The CNCF was designed to be stewards for this and other projects with the intention of making software easier to develop and operate.
This year was the year of the Service Mesh and socks.
The week was not just an opportunity to learn from other experts, but to be at the forefront of new announcements from my favorite cloud.
Virtual Kubelet – The new version of the Kubernetes connector was announced at KubeCon. This enables Azure to extend Kubernetes to Azure Container Instances (ACI), and provide our customers with per-second billing and NO virtual machine management for containers.
Ark – a migration tool which enables teams to move AWS and GCP (cross cloud Kubernetes tool) to Azure. Microsoft and Heptio (the creators of Ark) have formed a strong partnership. Ark delivers a strong Kubernetes disaster recovery solution for customers who want to use it on Azure.
Open Service Broker for Azure – We announced the open sourcing the Open Service Broker for Azure (OSBA), built using the Open Service Broker API. OSBA exposes popular Azure services to Kubernetes such as Azure CosmosDB, Azure Database for PostgreSQL, and Azure Blob Storage.
Metaparticle – Brendan Burns announced during the Keynote address, the delivery of an experimental model for coding for cloud. Metaparticle attempts to reduce the complexity and duplication of code for deploying software to Kubernetes.
Kashti – A visualization dashboard for https://github.com/azure/brigade
Other notable announcements:
Kubeflow – Machine Learning Toolkit for Kubernetes
I was recently invited to participate in the Microsoft Partner blog where I shared my love of containers.
I’m especially passionate about container technology because of how much it makes the developer’s life easier. Unfortunately, it’s one of those things that must be experienced to truly understand. I tried to boil my thoughts town to just a few paragraphs here. Check it out and let me know what you think!
Azure App Service for Linux is a pretty neat offering from Azure. You get all of the DevOps features you want (A/B Testing, Hosted Application, Tiered Support, Button-click scaling, lots of templates and more!) without the headache of managing VM’s.
9 years ago, I wrote a quacky little website called “Duckiehunt“. Unfortunately, I didn’t pay the tech debt and things kept breaking until it was abandoned. I’m now using Duckiehunt as a learning ground for Azure’s services and alternatives.
Azure App Service for Linux was the perfect fit. However, back in 2008 SSL wasn’t as ubiquitous. Now, it’s a badge of shame to NOT have it. Azure does offer an App Service Certificate, but I’d like to find a cheaper/more open solution.
Enter Let’sEncrypt from Mozilla and the EFF. If you don’t know, EFF are the unsung heroes of the internet. They fight tirelessly to support your freedom and rights on the internet. Mozilla and EFF offer Let’sEncrypt as a free way to encrypt websites via CertBot. Now I’ll dig into the technical details behind encrypting an App Service for Linux with Let’sEncrypt.
Step #1: Get CertBot
Because I’m on OSX, I was able to run: brew install certbot. For the full range of options, CertBot’s webpage has what you need.
Step #2: Create Cert locally
Before CertBot can create the certificate for you, it must first validate you own the domain. It will prompt you for a few questions, and then ask you to create a file on the webhost and add content to that file for validation.
Thankfully, Azure App Service for Linux provides a terminal access to your container so you can make these modifications yourself.
➜ sudo certbot certonly -d duckiehunt.com –manual
Create a file containing just this data:
%RANDOM STRING 1%
And make it available on your web server at this URL:
At this point, the validation is in place and it’s time to continue with Chatbot by pressing “Enter”.
Waiting for verification…
Cleaning up challenges
– Congratulations! Your certificate and chain have been saved at:
Your key file has been saved at:
Your cert will expire on 2017-11-12. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
– If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Huzzah! I’ve now got a certificate. Time to upload.
By following the instructions, I was able to explore space and move strange and dangerous cargo from distant planets. By moving the wings around, I was able to make the Batwing and fly around Gotham. (Well before anyone else realized that potential.)
This was an immensely rewarding experience that I’ve carried with me through my professional career.
Naturally, the toys of the child lead us to adulthood. I knew I wanted to spend my life building. Creating. Spawning new ideas. I wanted to physically manifest my ideas into structures that others would see, admire and even work/play/live in. When I learned that you could get a job doing this, I was elated. I knew this was exactly what I wanted to do. My mission in life was set.
One fateful day, when I was sharing my new life mission with my Godmother she informed me: “To be an architect you have to know how to draw.” Anyone who’s seen me sign a check, write on a whiteboard, or even attempt to draw a square knows artistry genes were not bestowed upon me. I was crushed. My life’s mission was aborted and I was unsure what to do with myself.
To quote my wife: “Those are people? I thought those were windows…”
I drew this. Not sure what my obsession with blue people was. That drawing is nightmare fuel for me.
In High School, when Career Day came I didn’t care about any session other than the local architect. As torturous as it was, I still wanted to know what it was like. All I remember was “hard work…something something…dedication”.
Fast forward to the last 12 months. I made an exciting and brave leap to join Microsoft, and am now a “Cloud Solution Architect”. I’m an Architect. I’m a real, bonafide Architect. (I’m literally crying as I write this as I’m so overwhelmed with a sense of accomplishment.) My bricks aren’t 8x8x9.6 mm, they’re CPU Cores. I no longer have one toychest, I have 36 datacenter regions, spanned across the world.
//build is a developer-centric conference Microsoft hosts every year. Since I never expected to work for Microsoft, I wasn’t even aware of //build. So, when my manager asked me if I was excited to attend and I told him no, I now know why that was the naive answer.
AWS has a head start on cloud services over Azure. But if this conference was any indication, Microsoft is taking this all the more serious.
Here’s some of the announcements that really caught my eye:
CosmosDB: Originally the distributed storage behind DocumentDB, CosmosDB allows not only a document store, but a MongoDB API, a key-value store and a graph database (Gremlin). That alone is pretty impressive; however, the portion that impresses me the most is how CosmosDB handles consistency. Traditionally, a database will offer either strong or eventual consistency.
However, CosmosDB goes far beyond those two models and introduces 3 more that are all available as a turn-key solution. (Bounded Staleness, Session and Consistent Prefix (a new model of their own design))
As a data guy, this is impressive to say the least. Not just because I work here, but because this is a new level of choice that I haven’t seen before and am excited about.
Speaking of being a data guy, offering Postgres and MySQL as a service made me giddier than it probably should. That said, AWS has had it for a while, so I’m more excited that we’re catching up.
AI: There’s no denying that machine intelligence is on the rise. Netflix’s $1,000,000 prize was just the start, and the pot has gotten bigger. The teams demo’ed Object detection and identification in manufacturing rooms, that led to a “sledgehammer selfie”. You had to be there.
Skype: While Skype may not be sexy technology, if it can provide an email transcript of a meeting with a list of action items (assigned by voice commands) as the demo provided, that might change.
Powerpoint + AI: Powerpoint isn’t really sexy either. Even less than Skype. In fact, I’d put it along the same sexiness as Orkut. But the demo of speech-to-text + text translation got a huge round of applause (the demo showed a Spanish presenter translated to Chinese in seconds.)
ServiceFabric: The team announced a GA for 5.6, and while it was already available, Windows + Linux containers. It can also ingest docker-compose files, which is interesting, but sent a mixed message to the OSS community.
Fluent Design: I’m color blind, so visual design is often lost on me. Other people seemed excited about it. So, that’s nice.
Lin on Win: Ubuntu Bash on Windows is nothing new. But now you can download Ubuntu, Fedora and SUSE from the App store instead of enabling “developer mode”. Oh yeah, iTunes is on App Store now too. Dude.
Hololens: Microsoft’s current Hololens is very neat, but costs ~$3000. Microsoft announced is a $399 model from Acer, which will be available in time for the holidays. Microsoft’s Hololens uses a transparent screen in front of your eyes to overlay augmented reality, and the Acer model provides a complete-view screen with cameras on the side to augment. There were 19 mixed reality experiences (vendors/partners) attending //build.
The parties: Microsoft spared no expense in ensuring that the guests enjoyed themselves. My highlight was walking around CenturyLink Field (home of the Seattle Seahawks) and screaming “Who Dat!”. Rock-aoke (Karaoke with a live-band) was a huge hit too.
Want to pretend you were there from experiencing my photos? Now you can!
After Oracle’s surprise announcement of their containerization of Oracle DB, Oracle WebLogic and a few of their other core technologies, I decided to test it out for myself. (Speaking authentically, I’m leery of their commitment; however, I recognize that I work on Open Source at Microsoft, so who am I to judge?)
After some sleuthing, it appears they once included the OracleLinux binaries in the git repo but have not purged them. Poor Github. I have a tremendous amount of appreciation for their architects and support engineers. Below is the SHA1 of the blob, the # of bytes of each file and the path.
Run their buildDockerImage.sh from the Github Repo
The documentation isn’t explicit about where to store the downloaded image. (in my case the ‘OracleDatabase/dockerfiles/184.108.40.206’ directory)
Now the moment of truth. From the “OracleDatabase/dockerfiles” directory, run buildDockerImage.sh
CLICK TO SHOW DETAILS
dockerfiles git:(master) time ./buildDockerImage.sh -v 220.127.116.11 -s
Building image 'oracle/database:18.104.22.168-se2' ...
Sending build context to Docker daemon 3.454 GB^M^M
Step 1/16 : FROM oraclelinux:7-slim
Pages and pages of output. So much text that my iTerm buffer no longer had the initial command.
Oracle Database Docker Image for 'se2' version 22.214.171.124 is ready to be extended:
Build completed in 658 seconds.
./buildDockerImage.sh -v 126.96.36.199 -s 3.68s user 8.15s system 1% cpu 10:57.49 total
Perhaps I’m being overly dramatic; however, the Docker Ecosystem has lots of high expectations and one of those is rapid development and deployment through small, composable artifacts. Granted, building and deploying a new version of database is not a common occurrence; however, the process it not conducive to DevOps. That said, this is their first foray into this, so I’m still excited to see the change.
dockerfiles git:(master) docker images
oracle/database 188.8.131.52-se2 f788cd5b4b9d 4 minutes ago 14.8 GB
oraclelinux 7-slim 442ebf722584 6 days ago 114 MB
fedora latest 15895ef0b3b2 7 days ago 231 MB
microsoft/mssql-server-linux latest 7b1c26822d97 7 days ago 1.35 GB
nginx latest 5766334bdaa0 3 weeks ago 183 MB
ubuntu latest 0ef2e08ed3fa 8 weeks ago 130 MB
14GB? I take that back.
Start the container
Let’s get the party started…
dockerfiles git:(master) docker run --name oracledb -p 1521:1521 -p 5500:5500 oracle/database:184.108.40.206-se2
ORACLE PASSWORD FOR SYS, SYSTEM AND PDBADMIN:
LSNRCTL for Linux: Version 220.127.116.11.0 - Production on 28-APR-2017 03:21:48
Copyright (c) 1991, 2016, Oracle. All rights reserved.
TNSLSNR for Linux: Version 18.104.22.168.0 - Production
System parameter file is /opt/oracle/product/22.214.171.124/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/91c68ac2b2bf/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Copying database files
Huzzah! After about 9 minutes, it’s finally started! Let’s test it!
It is at this point that I realize I’ve already gone through 2 drams of Aberlour and I should probably stop for the night. Provided there is enough interest (and whiskey), I’ll write-up Step 2 of getting this running on Kubernetes in ACS. As for now, I should stop while the world is only mildly spinning.
NOTE 1: If the database auto-generates a password with a “/” in it, I’ve found it doesn’t work. You can change that by running: docker exec ./setPassword.sh
NOTE 2: If you run this multiple times, make sure to run “docker system prune” as it fills up your disk fast. On my 3rd try, I hit the following error, even with lots of space on my disk. [FATAL] [DBT-06604] The location specified for 'Fast Recovery Area Location' has insufficient free space.
CAUSE: Only (9,793MB) free space is available on the location (/opt/oracle/oradata/fast_recovery_area/ORCLCDB/).
ACTION: Choose a 'Fast Recovery Area Location' that has enough space (minimum of (12,780MB)) or free up space on the specified location.
After hearing about it for years, I was fortunate enough to attend DockerCon this time around. Since joining Microsoft as a Open Source Technical Evangelist, 80% of my job is either learning or teaching. This was my first OSS conference since joining Microsoft, and I was eager to share with others my experiences.
I was even more excited to find out that a Drew Erny (my Godmother’s grandson) was not only attending, but presenting! It was also a change for me to hobnob with some of the Docker elite and some of the other Microsoft movers and shakers.
Docker Multi-Stage Build – TL;DR – Specify multiple FROM’s separate build env from deploy artifact. For more details
MobyProject – Open Source project to help developers create their own Docker-like container platform. This one was unclear at first, until I read a few more articles on it.
LinuxKit – A toolkit for building secure, portable and lean operating systems for containers was open sourced live on stage!
Topics ranged from enterprise deployments to enterprise scaling to enterprise security and “how to convince your enterprise boss” and “Docker Enterprise. Look at how Enterprisey we are and how Dockery other enterprises are”.
Day 1’s keynote felt more developer centric, and Day 2’s felt more enterprise centric. Afterwards, I also noticed the undertone of “Look how Enterprise Docker is” in not just the keynotes, but many of the presentations. Docker is definitely positioning itself to be more respected in the Enterprise world. I get it and completely understand it, but the message was tilted every so slightly towards that slant.
NOTE: There used to be rumors of Microsoft buying Docker. If Microsoft had, and then Docker made the same Enterprise slant, there would be a HUGE backlash. Docker has worked hard to be beloved and it shows.
Since I registered late, I missed a number of the critical emails including an FYI to RSVP to a party that was waitlisted by the time I discovered it. Thankfully, by then I had found my own crew to dine and drink with.
The DockerCon app was helpful for detailing the tracks and available sessions and adding them to the DockerCon app’s calendar. Would be helpful if it exported to a personal calendar for reminders as I got caught up in the Expo hall many times.
As a coordinator of 1000+ people events, I understand exactly how difficult this is. Your best hope is that no one really notices the blood, sweat and tears that go into setting it up. And it’s now that everything is done that I appreciate how good of a job they did.
The was more than adequate signage and information for what is happening and where.
This is the first convention I’ve been to that included a swing set, which was awesome. Lots of break-out areas, separated by pallets and bean-bag private spaces.
Microsoft and IBM were the platinum sponsors and it showed as they were the first two you saw when walking in. Outside of that, there were plenty of vendors eager to talk and lots of great swag. Drones were the most popular prize, but sadly the luck of the Coonass wasn’t with me.
Lots of great vendors. I got to pick the brains of talented teams at AWS, Rancher, Yippie.io, Redhat, Docker, Aqua, RedisLabs, 1&1, Citrix, Cloud Native Compute Foundation, Oracle (yes, that Oracle. They provide Oracle server on containers now!)
Lots of great presentations and speakers.
“Creating Effective Images” was the top rated and thankfully repeated since I missed it the first time. I highly recommend watching when it becomes available online.
Docker Swarm Deep Dive – Drew Erny did a great job of headlining this talk with demos from some of his compatriots. I saw how Docker bakes security into everything they do which will make all of our lives easier. I have been focused on Kubernetes, but the new announcements for Docker Swarm have gotten me really excited, especially how they handle Secrets and image security, software supply chain lifecycle and desktop deployments.
Here’s some great quotes I overheard:
“I only use microservices to effectively hide the root cause of any problem I create”
“Whatever layer you’re at, the layer below you is just magic”
“To quote WuTang: Cache rules everything around me.”
Prior to DockerCon, I really hoped to attend and meet a few more Microsoft’ers and some Docker’ers(?) but got swept up into the community and the common goal it has for deploying software better, faster, stronger. I can’t wait till next year.
P.S. If you are interested in toying around with Docker, check out: http://training.play-with-docker.com/ It’s a great walkthrough without the need to install anything (browser based development!)
The great Larry Wall claims that these are the three great virtues of a great programmer. And I whole heartedly agree. However, If I were to propose three virtues, they’d be: Inquisitiveness, Acceptance, and Stubbornness.
My name is Tommy Falgout, I’m a new employee at Microsoft and I have no idea what I’m doing.
I don’t know .NET. Or Azure. I don’t own a Windows Phone. Heck, the last Windows OS I “owned” was XP.
What I do have is ~20 years of experience in *nix and Open Source software development. I helped develop the original SMS implementation for GSM, back when phones were only meant for voice. I wrote telecom automation systems in Perl, PHP and MySQL 3.x. I then worked at Yahoo for 9 years where I expanded my brain to build their live events engine to broadcast Obama’s Inauguration, the Royal Wedding and the NFL games (Yahoo used to own NFL streaming rights in the 2000’s) I migrated to Yahoo’s Infrastructure database which was the duct tape keeping everything together and integrated.
As a Technical Evangelist, I’m building upon all that experience as a foundation for this new opportunity. To take Azure to the next level.
The thing is…I don’t know how to Azure yet. But that’s the point of this blog. To detail my findings and explore the union of Open Source and Azure. This will be a dumping ground and lesson’s learned. I’m a big fan of transparency and learning from other people’s mistakes. My hope is that you can learn from mine.