How do you run inspec.io on Solaris without killing yourself ??

We are using inspec.io for our post deployment verification / checking strategy. When I evaluated inspec, I really like a few things about it:

• It had a small toolchain and good support for various platforms - Windows, Redhat, Ubuntu, Mac etc. Easy to install, experiment and implement.
• It's supermarket checks is like collection of checks done and outsourced by other teams which are open sourced which we can leverage and quickly get working on our own script to do code as governance. 
• You do your scripts in Ruby. Period.
• The ability to run locally as well using ssh which is something that I really liked as well. 

We were planning to use inspec as the post deployment verification after deployment as part of our Continuous Delivery strategy. Everything was going along fine, until today. A few things hit us. This target server that we wanted to run the scripts on was a Solaris platform. First warning bell tolls. Sola-whut-again ? Really ? No packages were available for Solaris for inspec on their site. 

So that rules out calling our post deployment checks locally on the server. I mean of course this is not the end. We can try to install inspec on the server using source. Have not tried this, don't know what kinds of misery awaits to take huge chunks of meat from my arm when I reach out to open this door once, but I have a feeling we will have an easy time with this one. At this point too I was thinking ... I might not know enough here but who in their right minds runs a production system on an OS whose very existence instantly puts the whole project into a deep tech dept ?? Put another way, I was at one point in my life, keyword here definitely being "was" impressed with SunOS way back in 2005 back in the days of IpVx watching in awe as my Senior banged out long strings of commands without the help of the tab key and could somehow magically get the commands right every single time while he looked straight ahead and banged them out in blinding speed. Fast forward to 2019, let's just say after all that's happened I have lost equal measure of respect for that senior as well the OS on which he was banging out those commands on. Come on .... who uses SunOS in this day and age ? I don't really like using RedHat when managing servers but RedHat is manna from heaven compared to this rusty old doornail. Thanos ... hand me that glove while I snap this tired old OS out of it's existence in my workplace! The day was getting along, tired and a trying to avoid pushing my blood pressure up any further, I soldiered on for other solutions that we could use.

Installing inspec on the Jenkins server ?

So, then we thought about calling inspec from the build server which is a Jenkins server. So, the plan: Install inspec on the Jenkins server, call the scripts from the Jenkins server and use the "-t ssh:// ..." argument to target the deployment server. Sounds good. Not ideal but it's workable. I don't really like this idea as in order to run it in this fashion, ssh keys has to be maintained in the Jenkins server or some kind of script wizardy has to be imbued in the scripts to make it run correctly. At this stage my first paddle did not look so good, so I decided what is the harm in trying the second paddle ... turns out the second paddle has it's problem as well. It's running an almost ancient version of Windows. I know, I know who in their right minds would run Jenkins on Windows right ? Sigh, for the sake of argument and the readability of this post, believe me, we did question this. Let's just say the server OS choice and installation is out of our control and our control extends as far as being able to configure stuff using Jenkins only. Again we hit the same problem here: There are not packages available for this OS as well. Yes another day in the land of "Dev" on this side of the mountain while "Ops" being on the other continent! So yayy for our second paddle sinking to the murky depths of the river. 

So what now ?

The way I look at it, we are left with a few choices now. All unknowns ... great! 

1. Try to bash my skull in and install inspec on Solaris. There was one measly comment on SO when someone asked this question pointing to some Chef scripts. That's all ! Confidence level plummeted to the core of the earth after seeing this. The usually boisterous SO was strangely quiet this time. 
2. Try to install all of our post deployment scripts on an external RedHat server and call them from there. The question here then becomes how the heck to you capture the status of the checks ? Is this even possible ?
3. Spin up a docker instance in Jenkins and call the scripts from within the docker from the Jenkins server ? I really don't know this one and I have not tried it. 

I am reaching out to all those 48 people who read this blog if you guys have ever tried or better yet succeeded in using inspec on your Solaris boxes before, please find it in the kindness of your hearts to share in the comments section on how you did it to benefit the tired writer of this blog as well as the others who might one day stumble upon this blog post. Cheers.

Some of the things I did not anticipate ...

Here are some of the things that I did not think to anticipate when tasked to implement a DevOps pipeline:

1. The humongous sized enterprise grade software such as RTC (Rationale Team Concert from IBM) and UDevelop. They are used mainly for the development side for the development workflow. These guys are so big and bulky that you can't even download and study them on your own laptop as you would while you were investigating the functionality of an OSS software or library. They are a real road block to coming up with good viable replacement. You cannot replace what you cannot study. Most of the developers hate these pieces of software for development they are clunky and hard to use but these guy are real juggernaut when it comes to ops side features. Basically we came to the conclusion that this software was chosen mainly to fulfil some ops side requirements not so much the whole flow or pipeline. 
2. These #1 pieces of software have a huge list of feature set that are not easily replaceable using OpenSource software without either: i. String a boat load of existing software together or ii. Accepting some limitations with the new replacement.
3. Only jump servers being able to access key servers thus our strategies using Ansible as provisioning strategies crumble to dust. These jump servers are Windows 7 usually and the best that we have figured so far to get Ansible there would be to upgrade them to Windows 10. 
4. After figuring out 3, it seems that UDevelop is this magical piece of software that can do all the things that Ansible can do and more, so in the end we were thinking should we use it or should we go out to replace it. We don't like how it's hard to test and learn but at the same time never re-invent the wheel right ?
5. Number 5 I sort of anticipated but was still taken quite aback after realising the degree of the apathy people in the departments corporate have of the outcome of their initiatives or efforts. It seems that most of the departments are so far removed from the outcome that they do not even care if what they do (or don't) or if their projects even have a positive net effect on the outcome or the final result of deliverables. Not all are like that but the ones that do care have gained enough wisdom to only speak out when the timing is right or to just not speak up at all. This is sad to me. 
6. The ops side of the equation is in another company all together which complicates change and proposals to change as you not only have to consider all the pros and cons in your own company but also if the changes will effect the other side. Bottom line is you have twice the redtape and twice the detractors. 

I breathe a silent prayer to the gods of Continuous Improvement that our efforts sees meets more smooth and happy days ahead and we do not loose site of our goals ahead. 

The Human in Devops

What was significant this week ?

This week a mild epiphany came to me right after a somewhat heated and tense meeting with a team of developers plus project owner of a web project. They were angry and they were not afraid to show it. They were somewhat miffed about the fact that the head wrote them an email pretty much forcing them to participate to make our DevOps initiative a success. All kinds of expletive words were running through my head in relation to describing this team of flabby, tired looking individuals in front of me, which belied the cool demeanour and composure that I was trying so hard to maintain.

It happened. In the spur of the moment I too got engulfed in a sea of negativity and for a few minutes lost site of what is the most important component or pillar in a successful DevOps initiative. The people. 

"What a bunch of mule heads !" I thought. It's as plain as day, once this initiative is a success everybody can go home earlier and everything will be more predictable and we can do much much more than we could before. "Why are you fighting this ?!" I was ready to throw my hands up in defeat when it finally dawned on me.

"Codes that power DevOps projects don't write themselves. People write those code" 
"Without people powering our initiative now, we are just a few guys with a bunch of code and tools that are irrelevant"

Boom! These thoughts hit me like lightning and in that moment I felt and equal measure of wisdom brought by this realisation as well as disgust at my stupidity of forgetting one of the main tenants and requirements to make the dream of a successful DevOps project a success.

It was then I realised 2 very important mistakes I had made so far:

1. I was reaching out horizontally to push our agenda across. Developers loved what we proposed and that was pretty much it. It's cool and it's cutting edge. It stopped there. "Hey thanks for sharing that cool tool ! I will try it in my project when I get the chance!" is pretty much the maximum you can expect to get from such an exchange. For you to gain any traction, you have got to sell your proposed solution or improvement to the stakeholders or the decision makers. Efforts that usually require people to do the right thing or go out of their way to do some unplanned kindness or rightness usually results in zilch. 
2. I did not try to see the tool that I was proposing from the eyes of the beholders. It was too much of a leap. Much like how Abraham it's impossible for you to frog leap from sadness to happiness, so it was how the developers felt. They knew it was good for them, they can see it was good for them, they felt it could have the potential to improve their lives but alas they did not internalise it. The proverbial light bulb did not turn on inside of them, more correctly said, I did not do enough to turn that light on. I could see some people opening up, but when this realisation hit me, I just ended the meeting. I have not done enough of understanding where these people that I hoped to implement DevOps were. I had to do that first. 

Do I miss coding ? Do I miss hunkering down and prototyping my way to showcase a tool or to get something to work ? Of course! Who wouldn't but main thing I keep on going back to is ... what is the main goal and expectation of the people who hired me to lead their DevOps push ? Is it to wire together some tools and configure something so they can use it ? At small enough scale probably that is enough of value, but when you want the horses you lead to the puddle to drink you need to give them a reason and just because you are drinking, you can't expect them to follow suit. 

I am going to reach out more, I am going to understand more and I am going to engage more. All the people pieces needs to be in place before the pieces start falling automatically. Stay tuned if this is interesting ...