How do you run inspec.io on Solaris without killing yourself ??

We are using inspec.io for our post deployment verification / checking strategy. When I evaluated inspec, I really like a few things about it:

• It had a small toolchain and good support for various platforms - Windows, Redhat, Ubuntu, Mac etc. Easy to install, experiment and implement.
• It's supermarket checks is like collection of checks done and outsourced by other teams which are open sourced which we can leverage and quickly get working on our own script to do code as governance. 
• You do your scripts in Ruby. Period.
• The ability to run locally as well using ssh which is something that I really liked as well. 

We were planning to use inspec as the post deployment verification after deployment as part of our Continuous Delivery strategy. Everything was going along fine, until today. A few things hit us. This target server that we wanted to run the scripts on was a Solaris platform. First warning bell tolls. Sola-whut-again ? Really ? No packages were available for Solaris for inspec on their site. 

So that rules out calling our post deployment checks locally on the server. I mean of course this is not the end. We can try to install inspec on the server using source. Have not tried this, don't know what kinds of misery awaits to take huge chunks of meat from my arm when I reach out to open this door once, but I have a feeling we will have an easy time with this one. At this point too I was thinking ... I might not know enough here but who in their right minds runs a production system on an OS whose very existence instantly puts the whole project into a deep tech dept ?? Put another way, I was at one point in my life, keyword here definitely being "was" impressed with SunOS way back in 2005 back in the days of IpVx watching in awe as my Senior banged out long strings of commands without the help of the tab key and could somehow magically get the commands right every single time while he looked straight ahead and banged them out in blinding speed. Fast forward to 2019, let's just say after all that's happened I have lost equal measure of respect for that senior as well the OS on which he was banging out those commands on. Come on .... who uses SunOS in this day and age ? I don't really like using RedHat when managing servers but RedHat is manna from heaven compared to this rusty old doornail. Thanos ... hand me that glove while I snap this tired old OS out of it's existence in my workplace! The day was getting along, tired and a trying to avoid pushing my blood pressure up any further, I soldiered on for other solutions that we could use.

Installing inspec on the Jenkins server ?

So, then we thought about calling inspec from the build server which is a Jenkins server. So, the plan: Install inspec on the Jenkins server, call the scripts from the Jenkins server and use the "-t ssh:// ..." argument to target the deployment server. Sounds good. Not ideal but it's workable. I don't really like this idea as in order to run it in this fashion, ssh keys has to be maintained in the Jenkins server or some kind of script wizardy has to be imbued in the scripts to make it run correctly. At this stage my first paddle did not look so good, so I decided what is the harm in trying the second paddle ... turns out the second paddle has it's problem as well. It's running an almost ancient version of Windows. I know, I know who in their right minds would run Jenkins on Windows right ? Sigh, for the sake of argument and the readability of this post, believe me, we did question this. Let's just say the server OS choice and installation is out of our control and our control extends as far as being able to configure stuff using Jenkins only. Again we hit the same problem here: There are not packages available for this OS as well. Yes another day in the land of "Dev" on this side of the mountain while "Ops" being on the other continent! So yayy for our second paddle sinking to the murky depths of the river. 

So what now ?

The way I look at it, we are left with a few choices now. All unknowns ... great! 

1. Try to bash my skull in and install inspec on Solaris. There was one measly comment on SO when someone asked this question pointing to some Chef scripts. That's all ! Confidence level plummeted to the core of the earth after seeing this. The usually boisterous SO was strangely quiet this time. 
2. Try to install all of our post deployment scripts on an external RedHat server and call them from there. The question here then becomes how the heck to you capture the status of the checks ? Is this even possible ?
3. Spin up a docker instance in Jenkins and call the scripts from within the docker from the Jenkins server ? I really don't know this one and I have not tried it. 

I am reaching out to all those 48 people who read this blog if you guys have ever tried or better yet succeeded in using inspec on your Solaris boxes before, please find it in the kindness of your hearts to share in the comments section on how you did it to benefit the tired writer of this blog as well as the others who might one day stumble upon this blog post. Cheers.

Some of the things I did not anticipate ...

Here are some of the things that I did not think to anticipate when tasked to implement a DevOps pipeline:

1. The humongous sized enterprise grade software such as RTC (Rationale Team Concert from IBM) and UDevelop. They are used mainly for the development side for the development workflow. These guys are so big and bulky that you can't even download and study them on your own laptop as you would while you were investigating the functionality of an OSS software or library. They are a real road block to coming up with good viable replacement. You cannot replace what you cannot study. Most of the developers hate these pieces of software for development they are clunky and hard to use but these guy are real juggernaut when it comes to ops side features. Basically we came to the conclusion that this software was chosen mainly to fulfil some ops side requirements not so much the whole flow or pipeline. 
2. These #1 pieces of software have a huge list of feature set that are not easily replaceable using OpenSource software without either: i. String a boat load of existing software together or ii. Accepting some limitations with the new replacement.
3. Only jump servers being able to access key servers thus our strategies using Ansible as provisioning strategies crumble to dust. These jump servers are Windows 7 usually and the best that we have figured so far to get Ansible there would be to upgrade them to Windows 10. 
4. After figuring out 3, it seems that UDevelop is this magical piece of software that can do all the things that Ansible can do and more, so in the end we were thinking should we use it or should we go out to replace it. We don't like how it's hard to test and learn but at the same time never re-invent the wheel right ?
5. Number 5 I sort of anticipated but was still taken quite aback after realising the degree of the apathy people in the departments corporate have of the outcome of their initiatives or efforts. It seems that most of the departments are so far removed from the outcome that they do not even care if what they do (or don't) or if their projects even have a positive net effect on the outcome or the final result of deliverables. Not all are like that but the ones that do care have gained enough wisdom to only speak out when the timing is right or to just not speak up at all. This is sad to me. 
6. The ops side of the equation is in another company all together which complicates change and proposals to change as you not only have to consider all the pros and cons in your own company but also if the changes will effect the other side. Bottom line is you have twice the redtape and twice the detractors. 

I breathe a silent prayer to the gods of Continuous Improvement that our efforts sees meets more smooth and happy days ahead and we do not loose site of our goals ahead. 

The Human in Devops

What was significant this week ?

This week a mild epiphany came to me right after a somewhat heated and tense meeting with a team of developers plus project owner of a web project. They were angry and they were not afraid to show it. They were somewhat miffed about the fact that the head wrote them an email pretty much forcing them to participate to make our DevOps initiative a success. All kinds of expletive words were running through my head in relation to describing this team of flabby, tired looking individuals in front of me, which belied the cool demeanour and composure that I was trying so hard to maintain.

It happened. In the spur of the moment I too got engulfed in a sea of negativity and for a few minutes lost site of what is the most important component or pillar in a successful DevOps initiative. The people. 

"What a bunch of mule heads !" I thought. It's as plain as day, once this initiative is a success everybody can go home earlier and everything will be more predictable and we can do much much more than we could before. "Why are you fighting this ?!" I was ready to throw my hands up in defeat when it finally dawned on me.

"Codes that power DevOps projects don't write themselves. People write those code" 
"Without people powering our initiative now, we are just a few guys with a bunch of code and tools that are irrelevant"

Boom! These thoughts hit me like lightning and in that moment I felt and equal measure of wisdom brought by this realisation as well as disgust at my stupidity of forgetting one of the main tenants and requirements to make the dream of a successful DevOps project a success.

It was then I realised 2 very important mistakes I had made so far:

1. I was reaching out horizontally to push our agenda across. Developers loved what we proposed and that was pretty much it. It's cool and it's cutting edge. It stopped there. "Hey thanks for sharing that cool tool ! I will try it in my project when I get the chance!" is pretty much the maximum you can expect to get from such an exchange. For you to gain any traction, you have got to sell your proposed solution or improvement to the stakeholders or the decision makers. Efforts that usually require people to do the right thing or go out of their way to do some unplanned kindness or rightness usually results in zilch. 
2. I did not try to see the tool that I was proposing from the eyes of the beholders. It was too much of a leap. Much like how Abraham it's impossible for you to frog leap from sadness to happiness, so it was how the developers felt. They knew it was good for them, they can see it was good for them, they felt it could have the potential to improve their lives but alas they did not internalise it. The proverbial light bulb did not turn on inside of them, more correctly said, I did not do enough to turn that light on. I could see some people opening up, but when this realisation hit me, I just ended the meeting. I have not done enough of understanding where these people that I hoped to implement DevOps were. I had to do that first. 

Do I miss coding ? Do I miss hunkering down and prototyping my way to showcase a tool or to get something to work ? Of course! Who wouldn't but main thing I keep on going back to is ... what is the main goal and expectation of the people who hired me to lead their DevOps push ? Is it to wire together some tools and configure something so they can use it ? At small enough scale probably that is enough of value, but when you want the horses you lead to the puddle to drink you need to give them a reason and just because you are drinking, you can't expect them to follow suit. 

I am going to reach out more, I am going to understand more and I am going to engage more. All the people pieces needs to be in place before the pieces start falling automatically. Stay tuned if this is interesting ... 

DevOps for corporate ?

Implementing a DevOps strategy with your army of one team is most of the the time easy. It's just dependent on the sweat on your brow. A matter of experimentation, glue the pieces together push it out and you are pretty much good to go.

Put in a corporate environment into the mix and add in about a few hundred more people into the mix and you get a pretty interesting mix. Strangely this time around I don't find the exercise exhausting or that much frustrating. I have been struggling and examining my own feelings and comparing to find the difference in the situation this and why don't I feel so restless or frustrated. I think it's a function of growing up and the promise I see in the people that I am helping now. The difference I see with the people now is that they want to change and they feel a need to do so. No question about that.

However add in that dash of fear of the unknown of the new methods and practices and you get a a jumble of wishful thinking and half implemented ideas and visions. Main thing is try not to judge. That is very important. The book "Factfulness" says it best. Try to believe that people are not stupid. Practices or things are a certain way for a reason. Sticking with the belief that people or practices in an organisation is a certain way because people are stupid or stubborn just makes the task or changing or overhauling something in that same organisation so much insurmountable and frustrating for me.

Believing that people inherently want to do the right thing given the chance puts you on a mode of thinking or view that have a much higher chance for success. Then the only thing left are practices or governance that were erected to protect a collective belief or state not because people want to make other people's life harder but usually because they did not know better and were of the belief that the governance or checks will help their current situation.

Now I see a lot of opportunities for improvement and change that can be unlocked via many levels governance unlocked by convincing committees and gatekeepers. I am hopeful and see many opportunities to help and make things better enabling the organisation to move faster than it had before in the past.

Currently reading ...

A new year and new challenges

It's a new year already. I have this uncharacteristically made some promises to myself. I have set a goal this year to read more, 20 books more to be exact. The promise was solemnised in the goodreads app. So far I am 8 books in and off to a good start, hopefully I keep it up midway and finish strong come December of 2019.

The new year brought about too challenges in looking at some of the things I do daily from a different perspective. It was about at that time too, that I came across this great book called Grit written by Angela Duckworth. Some contents of that book seems to be speaking to me directly and forced me to look back at my own history in some areas with the harsh light of truth.  Sometimes or most often than not stuck in the throes of your own self righteousness of what things you feel that you have been wronged or those who have wronged you, your blinders does not allow you to see the whole truth, or I guess self preservation distorts the truth for you so that you can still get out of bed each day and champion another day of tilting at your windmills.

Some chapters of this book is like a balm or the soothing syrup to the hoarse voice that cries out in my head that I should be constantly jumping around and looking for the next best thing. The voices have died down to a degree until you can say I am only "semi looking" for a good thing now, as in only if something really good comes along I will try out for it. Thinking ahead I would rather leave behind regrets for things I have really gave a good shot to that I failed rather than all half tries which have still lingering hopes in there that I could have triumphed or prevailed if I had stuck to it longer of tried that last great idea I had before I got struck down by the powers that be. Talking about the "powers that be" I think too that I might have been hiding too long behind being mistreated in my jobs or stuck in a job where I was helpless to make a change. I bailed just before things got bad or more accurately before my ideas had time to get bad.

I am going to steer and commandeer this one until I get some results be good or bad to see where it takes me this time. I don't want to leave a legacy of half tried ideas behind. For those of you that want to give this great book a try, you can find the book here, full disclosure it's an affiliate link:

Grit: The Power of Passion and Perseverance

Will tell you more about this book in my upcoming posts.