Splunk2go - an iPhone application for accessing splunk data

Januar 6th, 2010

For those not familiar with it (why are you reading this?? :-) ) splunk (www.splunk.com) is an IT search engine. It indexes anything from log files to performance metrics in your infrastructure, and allows you to work with the results in a very slick user interface. Think of it (warning: highly simplified statement) as Google for IT data. In this regard it is way beyond anything the traditional IT management vendors offer. I personally really like it and the company behind it (they’ve been very responsive and helpful when I talked to them a few months back).

Launching Splunk2go

Launching Splunk2go


Like any IT management application you normally use splunk from your desktop for administrative, troubleshooting and management purposes. However, every now and then you do run into situations where you are on the road, in a meeting, buried somewhere under a raised server room floor, or (admit it!) in the privacy of your bathroom and really need to access some of the data that splunk manages for your.

This is where splunk2go comes in. It is basically a simple UI for accessing splunk data on your iPhone, allowing for free form search and access to saved searches.

As you launch splunk2go you are presented with a simple tab-based user interface. If you have used splunk before, you’ll feel right at home:

Default Application Screen

Default Application Screen




The “Free splunk” tab allows you to freely enter splunk search commands. So let’s say you want to look for error messages from any of your systems. Simply enter “error” in the search bar, and the application will return them to you. By default it will display events from the last hour, but you can configure this to your liking.

Search Results

Search Results

Browse Search Results

Browse Search Results


As you can see in the screenshots, splunk2go gives you the timeline of the number of event, allows you to filter the results by tapping on the timeline or by host, event source and event type, and the gives you access to the raw event results.
By tapping on any of the events you can view the event details.
Event Details

Event Details

More Event Details

More Event Details


If you switch over to the Saved Searches tab, you have access to the saved searches on the splunk server, and can run them directly. This is helpful for complex queries that might be tedious to type in on the iPhone. You can still refine the search queries, as they are available from the search bar for you.
Saved Searches

Saved Searches

Saved Searches 2

Saved Searches 2

Search Results 3

Search Results 3

This is the current state of the application. I created it a while back when I was teaching myself iPhone programming. Let me know if there are additional functionalities that you might be interested in.

Enterprise 2.0 meets ITIL: Building a collaborative IT Service Management Practice, Part 1

Oktober 9th, 2009

This is the first part of a set of blog posts where I’ll be introducing a pet project of mine. Combining IT operations and service management with Web 2.0/Enterprise 2.0 concepts.
Service Space

In the ITIL books and on some vendor slides the Service Knowledge Management System (SKMS) sounds like the dream of every IT manager. A system giving a complete and up to date overview and background information of an IT service.

The mistake the ITIL authors and most vendors make in my opinion is treating knowledge management as a process. This is a very traditional approach, very reminiscent of some tools of the previous generation of knowledge management systems.
The problem of this approach is: knowledge management as a process driven discipline barely works in heavily regulated industries (healthcare, defense, aerospace come to mind), because in these industries it is a set requirement, and people don’t get paid if it’s not done!
Lufthansa is not going to accept the delivery of their shiny new Airbus 380 without the proper and mindblowingly detailed documentation of every mundane detail of the plane.
Now compare this to the level of effort the typical IT organization puts into documenting their IT services. In most cases this is a loose combination of a set of word documents, visio diagrams and excel sheets. Some organizations will claim to have their systems documented in a CMDB, maybe even in one of the CMDB products from the HPs or IBMs of this world.
My point is nobody will get fired if the documentation of a service is not up to date and everybody will get their paycheck even though crucial details of the IT services mode of operation are stored in their heads only.
In an environment like this it is a unrealistic assumption that you can force the documentation of this type of information into a formal process. People are always just going to do the bare minimum, because they don’t want to do more and they don’t have to do more. Appealing to common sense (“if you document this, it will make your life easier”) is a moot undertaking here.
The only realistic way of approaching this is to make sharing of information
- easy to do
- convenient (and not only from a long term perspective)
- maybe even *gasp* rewarding and fun

Now, incidentally enough haven’t we just seen this happen on the internet over the last years? Ten or even 5 years ago sharing any type of content on the internet was the domain of a select few. Corporate websites, few personal homepages and dispersed discussion forums. Communication happened in e-mail.

Today we see a multitude of easily accessible ways for sharing content on the public internet: blogs, twitter, Facebook, LinkedIn, etc. Of course there is a certain amount of hype associated with this movement, but the shift from just consuming information to a two way communication environment has undeniably happened.

Now, if I take a look the state of IT operations and especially the tool support that most of the established (but sadly also the up and coming) players in this market offer, it seems to me that they are still stuck in the internet of yore.

In most cases I am looking at a set of different tools for the different disciplines (incident management, operations, monitoring, etc). Some vendors offer portal solutions that tied their product suite together, such as the Tivoli Enterprise Portal. The very consistent thing in all of these offerings (at least the ones that I have seen) is that they are basically just usable for consuming information.

I admit I am a products person. So please forgive me if I explain these concepts with a product example.
I started experimenting with enriching traditional IT operations tools with social media elements in the spring of 2008. My beef with the CMDB product I was working with at that time was that I was
- pretty difficult to use for the regular user
- not easy to share anecdotal or “soft” data in the product

I’ve said it before, I’ll say it again: IT is a people discipline. A lot of it is based on personal experience, anecdotal information, “tribal” data. This is especially true in IT operations.

The result of this work was a prototype called CMDWiki. A frontend to a CMDB based on the semantic mediawiki software. The main benefits I saw here, were that it combined the process based elements of creating and maintaining a CMDB (as it did not allow people to change CI data that was managed by any of the ITIL processes) with the ease of use of a wiki for finding information and sharing of “soft” information related to a CI.

This year I set out to build a more refined version of this concept.

For a lack of a better term, I am calling this a collaborative SKMS for now. I am well aware that this is a big name. Suggestions for something more humble are welcome.

So what is the use case for the Collaborative SKMS?
As in many other cases when it comes to managing IT operations, the cobblers children have no shoes. While it provides sophisticated services to other departments like sales and HR that allow for highly integrated, automated processes, IT itself is in most cases managed via email and Word documents. Of course, most organizations have implemented one or more ITIL processes (or similar processes from competing frameworks). However, as stated before information technology is about more than processes. When it comes to collaborating around aspects of an IT infrastructure, most organizations are stuck in the 20th century and use email, fileshares, etc. Progressive organizations might have implemented wikis or Notes databases for collaboration.
Still, all the negative aspects of this, information buried in emails, difficulties in finding information in a multitude of wikis, Sharepoint sites and fileshares, no communication across departs, geographies and hierarchies prevail in most IT organizations.

The collaborative SKMS is an example of bringing Enterprise 2.0 functionalities and concepts to the IT organization and merging them with existing solutions already in place in many organizations, such as helpdesk, operations monitoring, change management, even CMDBs.
For this prototype I have selected the HP BTO portfolio. This is not meant to be a specific endorsement, its just the portfolio I am most familiar with.

So let’s take a look at the involved elements
HP ServiceManager: Helpdesk, Incident Management, Problem Management, Change Management
HP Release Control: Change Management/Change Impact
HP Business Availability Center: Monitoring, Service Level Reporting
HP Universal CMDB: CMDB

Again, feel free to replace these with other vendors products, they are only examples in this case.

The system has multiple goals:
- Bring together information from multiple solutions used in IT
- Organize the IT environment into a structure that is consistent and relevant for collaboration
- Allow for communication and collaboration in a consolidated environment

Especially the last point is important to mention. Of course communication and collaboration are happening in IT operations today. The problem in my opinion is how they are happening. In every environment I have visited this happens via email in 80% of all cases. The rest usually sits in Office documents on fileservers.

As an entry point into the system the user will be presented with two choices:

A centrally defined homepage

Social SKMS Homepage

Social SKMS Homepage

or their personal homepage, which they can configure to show the information that is relevant to them. In this case we see the default one.

Personalized Dashboard

Personalized Dashboard

So how is the system structured?
There are two distinct structural elements in the system: Spaces and Social Groups.

Spaces represent a centrally defined hierarchy. In the case of the Collaborative SKMS a space would represent things like services (and underlying services if there are dependencies), departments, etc. Spaces are used for information that is relatively static in it’s hierarchy.

Social Groups are used for all information that is not bound to a hierarchy. I.e. whereas there might be spaces representing SAP system P01 and SAP system P33 (IT services in this case), there will be a social group called “SAP Basis” where the people running these systems will come together to communicate and collaborate around all things SAP, no matter where they reside in the organizational hierarchy or in which geographic location they work.

Especially the social groups are addressing an issue most larger IT organizations have today: They are not only functional silos (the DBAs, the networking guys, the Windows admins), there are also geographic silos. I’ve seen way too many customers where the networking team in Frankfurt did not even know which tools the team in Munich was using.
Social groups allow these teams to come together around their common area of interest.

Let’s take a look at an example of a space representing an IT service. In this case the service is an online banking application that is run by the IT department.

Entry point into a service sapce

Entry point into a service sapce

The entry point into this service space includes a real time display of the current status of the service (in this case coming from Business Availability Center). Also displayed are the statuses of projects related to this service, and an activity stream of recent additions and changes to the service space. In addition the available actions and the most active people in relation to this service (remember this is about people) are shown.
Note the different tabs for the content types available for this service. This includes configuration items, changes and incidents in this case (federated in from the appropriate source systems), but also more generic elements such as discussions (happening directly in the system or in email), documentation and blog posts. We’ll look at these in some more detail. The all content tab consolidates all activities into a single activity stream, which is available via a users dashboard, RSS feeds, email notification, even via their iPhones.

Let’s take a step back: A view like this consolidates all activity around a service in a consolidated, archived and searchable view. In addition to this, the content is also actionable, as we will see in a minute.
While this might sound trivial, something like this is actually incredibly difficult to do in most enterprise systems management suites. In most product suites I am familiar with, this would really be impossible, without a major development undertaking.

This is the end of the first part of the introduction to Collaborative ITSM. The second and third installments will follow shortly. I’m very interested in feedback, so feel free to share your thoughts.

When corporations twitter

September 10th, 2009

This is a translation of a hilarious, but true blogpost, that made the rounds in the German scene recently. http://www.trendopfer.de/wahrheit/2009/08/wenn-unternehmen-twittern/

Subject: Tweet #1
Friday, 10:01 AM
Dear F. In response to our meeting on monday of last week, I’m sending you our first tweet. Could you please publish it? We stuck to the 140 character limit. Should you have further questions feel free to get back to me.
Best regards, H.


Re: Tweet #1
Friday, 10:15 AM
Hi H. Thanks for your email. It seems you have forgotten the Tweet?!
Cheers
F.


Re Re: Tweet #1
Friday, 10:17 AM
Dear F. Sorry. I’ve attached the word document to this email.
Best regards, H.


FW Re Re: Tweet#1
Friday, 10:30 AM
Hello K. Department X wants to twitter. Can you take a look at this? I’ll forward to to M. for approval afterwards.
Lunch?
Later F.


Re: FW Re Re: Tweet#1
Friday, 11:30 AM
Hi F. the tweet is ok, IMHO
Lunch sounds good. 1 o’clock downstairs!
Cheers K.


FW: Re: FW: Re: Re: Tweet#1
Friday, 12:30 PM
Hello H. I’ve checked back with our corporate spokesperson and there are no objections from their side. I’m going to send it to M. for approval. The tweet should be online by EOB today.
Cheers F.


Twitter Department X Approval, Tweet #1
Friday, 12:45 PM
Dear Mr. M.
In conjunction with the new social media strategy of our organization and as a direct result of our workshop in may, department X has created a twitter-account for them. They want to start twittering immediately. I have attached their first tweet (please refer to the attached word document) for your approval. Mr. K from corporate communication has no objections.
Yours sincerely F.


Re: FW: Re: FW: Re: Re: Tweet#1
Friday, 2:30 PM
Dear F. Any feedback from management?
Cheers H.

Re: Re: FW: Re: FW: Re: Re: Tweet#1
Friday, 2:55 PM
Hello H, They’ve been in a meeting since lunch. Seems it’s going to take some more time.
Greets F.

Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Friday, 2:58 PM
Dear F., ok, its a few more hours until EOB.
L8r H.

Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Friday, 6:05 PM
Dear F. have you heard anything from M. I need wrap up for today…
Cheers H.

Re: Twitter Department X Approval, Tweet #1
Friday, 6:45 PM
Who or what is this Twitter?
M.
This message is confidential and may be privileged. If you are not the intended recipient, we kindly ask you to please inform the sender. Any unauthorised dissemination or copying hereof is prohibited. This message serves for information purposes only and shall not have any legally binding effect. Given that e-mails can easily be subject to manipulation, we can not accept any liability for the content provided.


Re: Re: Twitter Department X Approval, Tweet #1
Friday, 6:47 PM
Dear Mr M.,
twitter is a so called microblogging service for the fast exchange of information. We decided during our workshop in may to use this service as an additional communications channel. Department X is the first user.
I am wishing you a wonderful weekend.
Best regards F.

Sent from my iPhone.

Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Friday, 7:00 PM
Dear F. I am leaving the office now. I can check my email from home, tho’ ;-)
Enjoy your weekend
Cheers H.

Re: Re: Re: Twitter Department X Approval, Tweet #1
Monday 07:15 AM
Approved. You’ll have to explain this twitter thing to me again sometime
M.

This message is confidential and may be privileged. If you are not the intended recipient, we kindly ask you to please inform the sender. Any unauthorised dissemination or copying hereof is prohibited. This message serves for information purposes only and shall not have any legally binding effect. Given that e-mails can easily be subject to manipulation, we can not accept any liability for the content provided.

Re:Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Monday, 10:05 AM
Hello H. , the tweet is now approved. You had some spelling mistakes in there (Upper/lowercase) which I corrected for you. The corrected version is attached.
Best F.

Re: Re:Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Monday, 12:30 PM
Hello F., sorry, I was in a meeting. Thanks for the approval. The spelling mistake is not a mistake. We chose to write everything lowercase. Our intern said this is common on the internet.
Best H.

Re: Re: Re:Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Monday 1:00 PM
CC: K., M.
Hello H., I am not interested in what’s “common” or not on the internet. When your department is publishing messages on twitter you are doing this in the name of our company. This includes correct spelling. Please stick to the version approved by Mr. M and K. (see email from Monday 10:05). In addition it would be helpful if you could create a communication plan for the tweets planned for the next 2 weeks. With that we have better visibility into this.

Best wishes F.

Re:Re: Re: Re:Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Monday 1:40 PM
CC: K, M., P., D., L.
Hello F., N. will work with our new intern on a few slides (powerpoint) to explain our communication strategy for twitter. I’ll schedule a meeting for the end of next week. I think it would make sense if the decision makers from all involved departments would attend as well. We’ll publish an excel sheet of the tweets for the next two weeks on the intranet. Hopefully that will speed up the approval process..
Best regards H.

Re:Re:Re: Re: Re:Re: Re: Re: Re: Re: FW: Re: FW: Re: Re: Tweet#1
Monday 1:49 PM
CC: K, M., P., L.
Hello H. what’s this about?
Cheers D.
….
tweet#1
Monday 2:56 PM
CC: _department_x_all_employees
Dear colleagues
Our first tweet just got published. Sorry, but the approval took a little bit longer than expected. I’ve attached a screenshot for you.
Thanks to everyone working on this
Best regards,
F.

(TGIF. Have a good time, everyone)