“Disk Failure”, watch your S.M.A.R.T!

Yes, for those in any sort of "ops" or Sysadmin position, the very phrase makes your blood run cold causing cranial dermis spasms, usually around the eyes, along with that sinking feeling that makes you feel like you are exiting this reality at warp speed.

Well, its that taboo topic that needs talking about.  But you may say/think that “I have backups, I am fine” or “I use Various RAID levels with redundancy, I’ll be fine.”  but are you really?  If this is you, READ ON!  :)

Recently I had a NAS Failure.  Well, not so much as a complete failure, but rather, a NAS that was seemingly not happy.  Causing significant service disruptions and delays.  Big deal right?  Deal with it and move on.  Well recently, after fully integrating a storage area network in my workplace, I was thinking this is great.  I can spin up a VM, move it from node to node, add disks when I need more space, and all seemed hunky dory.  

Many months into its service, services seemed to sporadically crash, or have significant delays.   As I was diagnosing issues, I was thinking there was an issue with my compute node setup in my ESXi cloud.  My virtual machines would simply freeze.  They would show many disk errors.  I thought there was an interruption in the transport layer (or on the wire if you will) because once I rebooted the compute node all machines seemed to work again.  Slowly these issues began to bleed through to other servers connected to the SAN.  

I thought, and I admit perhaps naively, that well, the particular NAS (or Network Attached Storage) unit that was serving these servers had RAID5.  If there was a disk issue, it would fail the disk to the hot spare and given the redundant nature of RAID5, it would continue to operate and alert me of the failed disk.  WRONG!   

For those who are not familiar with RAID5, in a nut shell RAID is the way you configure multiple hard disks to operate together.  RAID-5 for example, what I was using, is the practice of using 3 or more disks with fault tolerance such that if one hard disk failed the other 2 would function and there would be no data loss.  

So, in my case, I was using 7 2TB disks, effectively having 6 2TB drives or  12TB of storage  (less formatting) available.  RAID5 complexity allows any one disk to fail and there would be no service impacting issues.  Especially with the “hot spare” waiting.  (A hot spare is an installed disk on standby in case another fails.)

Now back to the point, I began to suspect that a particular NAS server was showing signs of complete failure.  After making sure all my backups were up to date, I began to take a look at the system.  When I looked at the console, It was violently spewing out disk error and retry messages.  Every time there was a service stall, the console would begin to projectile puke these massages that make you freeze in fear.  Could my NAS be completely failing???  Could RAID5 have deceived me?

With poise and professionalism, I got my composure and exited my server room to face an office at 9am slowly growing full of people asking questions like; “why is FogBugz not working?” and “why cant I access the shared files on the server?” … and on and on.  I was looking at a systems failure of about 50% of all the services I operate.

On looking at the web client of the NAS, I began to check the health of all the disks.  What was actually happening was a SMART drive abnormality.  (SMART is the function that is built into all modern hard drives that can detect pending failure).  A disk in the RAID5 array had not completely failed, rather, it was about to.  It was not quite timing out to a failure, but as the drive was franticly trying to reallocate bad sectors, it stalled out the whole disk array.  But it would restore operations before it times out completely to failure but this was service impacting.  

So what I am trying to say here is that you cannot rely on RAID levels as a redundant measure.  You MUST also monitor the SMART status of all the attached drives.  A drive with an emanate failure can take out an entire array and play havoc with your services.  All I needed to do was remove the bad disk and the system failed over to the hot spare.  After the RAID rebuild completed, I added a new hot spare and all was good in the land of NAS once again.

Moral of my story here is that make sure all your monitoring systems are in check.  Setup regular SMART tests on all production systems.  In the case of this drive the SMART system was telling me that there was an abnormally high rate of bad sector reallocation.  It was a bad disk.  

As a side note here, we have the incorrect thought that modern hard drives are more stable than older hard drives.  But in actuality, the have MUCH more bad sector counts( more sectors, more failures).  They just reallocate these bad sectors and mark them as bad more frequently.  We just never know about it now unless you run disk utilities to view this internal data.

Also, if you reply on a NAS unit for as many services as I do here, I recommend that you have a redundant NAS unit that is mirroring so that you can fail over to in  cases like this.  Fix the issues and keep services running.  Sadly, this was not in my budget this year.

Until Next time,

//Ian\\

PS: If you have any topic you are interested in, please let me know, and feel free to leave comments and start discussion.  It would be nice to see an actual comment rather than spam :)  

Digital Signage Expo 2013

Well folks, better late than never.  This year I have the opportunity once again to attend Digital Signage Expo, or DSE.   This year marked the 10^th Anniversary of the Digital Signage Expo.  I will avoid talking about typical displays.  But rather, I would like to highlight the unique installations and technologies.  LCD displays are  simply displays.  With various sizes and aspects.  There is not much to be said (well with the exception of 4K displays!).  The teaser here is uniquely shaped displays, unique applications/installations, LED assisted bezels, and players sub $100 that are capable of playing HTML5 content. 

Lets start with the show floor and talk about the technologies I have seen. Planar had a rather unique mosaic installation (above).  Using their software to deliver your content.  As digital advertising becomes more of the 'norm' being creative and unique with your displays is going to be more important.  Although this install was not ground breaking, it was a rather nice and unique installation.  Although it may be costly to install, it can deliver a memorable experience.

Touch revolution also had some new products.  This year, they debuted their commercial Android tablets!  Seen here they are rated for 24/7 operation, and run PoE (Power over Ethernet) so no extra cabling is required other than Ethernet and of course they do run wireless.  These are made for fixed installation (shelf side or other applications).  These also come with a software toolkit to lock the devices down at a software level.  Onboard they have an ethernet, MicroSD card slot, usb ports and power port.  I really like these and can see many applications for use in retail.  Unlike using consumer tablets in an commercial application, these are designed for 24/7 operation and are front side, NON-branded.  I cant stress this enough.  Commercial devices should be non branded.  You are using their products to advertise OTHER products.  So why would once pay prime dollars for branded commercial devices.  This was refteshing to see.

Among other innovative products, were e-paper based  wifi enabled shelf price tags from Opticon.  These  were disposable products, however, based on 5 price changes a day, would run for about 5 years.  From various companies were other e-ink/paper based signs which were quite effective for things like room signage.  These are great for B/W NON INTRUSIVE, low energy usage signage.  On initial glance you would not realize they were digital signs.

One product did grab my attention.  As we know, many manufacturers are trying to eliminate the screen bezel all together.  However we know this is not w/o limitations.  Lxi International did come up with a very interesting way to work around it.  Their displays are simply displays WITH a bezel.  However the bezels are strips of LED lights which change colour to match what’s happening on the edges.  Although you can see the edges...   they do look better than black lines in my opinion.  Have a look!

  

Another very interesting product that would be a serious asset to actually gaining quantitative data for installations was a self contained customer profiling box.  We have had technology that would asses a viewers dwell time, approximate age and sex for a while, but they have been very complex installations and required some high end hardware.  Now Aware Live Technologies has come up with an appliance to do just this.  Its a small box with Ethernet, power and an Asus (Kinect like) camera attached.  This can be hidden within an installation to log and graph quantitative data.  

Seen above is a picture of me on its monitoring screen.  It has assigned me a unique visitor number, sex, age and added a count to the amount of people in front of it.  You can aggregate this data as to the direction the person has moved off the screen, and other valuable data for the client.  

Another interesting product was a video delivery device made by Matrox which would scale output to match the screen.  This would be a very useful device when sending video to may different screens.   So long as the aspect is the same resolution, it would not matter as it would scale the content for us.

There were also many other oddly shaped displays such as long aspects and rounded displays. 

Among other things were a plethora of signage players.  Many too complex and useful for very specific installations.  However the last item I would like to profile is Broadsign's $99 android player.  This small device is very light weight, and by small, I mean very small.  Sadly I did not get a picture of the device I was just so excited about this product and got talking to the rep for a while about it and walked away and did not take a snap.  Lets just say it will fit in the palm of your hand.  Also, since the device is Android, Dual Core 1.5Ghz it should be able to deliver signage through our stack.

This years show was much larger than 2012, and full of energy you see at the likes of InfoComm.  If you are into signage or displays of any sort, a visit to DSE will be in order.  Dont forget all, that next years show will be at the Sans Feb 11-13.  Not the LasVegas Convention Centre as it has been in the past.  Mark your calendars now.

//Ian\\

Computing Power Paradigm Shift

So I have noticed that I am now part of a paradigm shift.  I am not talking about a major shift in assumptions here nor the dreaded marketing speak, but a shift in how I consume my computing power.   I personally see computers as part of our lives now.  By far we are not the Star Trek characters known as Binars...  yet...  But I noticed having grown up as a small child entering the computer revolution and the loving geekdom that my life became, that it was about how bad ass your 286, 386 and 486 rigs could be!   Ok that dates me.  The Pentium entered the scene when I was in my OAC or grade 13 year in high school.

But I digress, back to the point.  I feel that unless you are a serious gamer or somebody who requires graphics, or scientific computing power, that its no longer about how powerful your PC is.  I realized this (at least for myself) in the last few days as I was getting increasingly frustrated with my Samsung Galaxy Note.

This is a very fine phone.  In fact, for most people its more than they need.  I however want the best from my mobile device.  I use it more than I do my own laptop now.  It keeps me posted on new messages, I take calls on it, I listen to music on it.  a LOT of music on it.  While I do that, I am usually playing a game, watching a movie, surfing,or reading and replying to emails before I arrive at the office.

Since I used my phone so much, I began to get frustrated at the slowness it would have to switch between tasks.  Since I used so many apps frequently, it had issues keeping up with my scatter minded multitasking.  I found that it was becoming too slow.


So even though my laptop is a MacBook Pro Core 2 Duo with 8GB Ram, and I run flight simulators and musical editing suites, it was primary for me to have a better, faster phone.  Lets face it, the Laptop I have does its job.  This is where I realized as I walked out of the local wireless store with my Samsung Galaxy Note 2 in hand, in a mild state of cognitive dissonance,  that I was more willing to drop my hard earned money on a new phone rather than a new Laptop or an upgrade.

I have entered the computing power paradigm shift.  Its all about the portable device now.  I never thought in my increasingly luddite mind that this would ever happen, but hey, perhaps I am not going to become a luddite after all!  I am ahead of the curve?

Thoughts?

Digital Signage Expo 2013

Ok folks!   Its that time if year again and at the end of the month, I will be headed down to Sin City once again to take in the latest in Digital Signage technology at DSE2013.  As usual, I plan to post while I am there.

So stay Tuned.  Also to my few followers, Sorry I have not posted in some time.  Busy time of year, but I am back at it once again!

Cheers Folks!
//Ian\\

GrooveShark.com - My most used service!

The Best Streaming Music Service Ever!

Those of you who know me in person, will have heard the words GrooveShark so many times! Yes, I am a self admitted GrooveShark whore! I LOVE this service. If you like music and you have lived under a rock, time to wake up, put the CD's and MP3's away. Enter GrooveShark. Most anything you want to hear, when you want to hear it.

Whats even better, is I have discovered artists that I never would have heard thanks to the Explore function, and the 'radio' function. I also find it fascinating to see what inspires other artists by listening to their play lists!!!

now I have also used many software applications that “try” to build play sets based on your listening, however, Grooveshark has been the best at this. So long as you don't have the likes of my partner and I selecting very different tracks (Various Show Tunes, to Electric Presidents, to Linken Park), it functions quite well. Even then, I experience some different and new things when we build a combined eclectic play list. Lol!

Another very cool feature is the ability to tweet/facebook what you are listening to, moreover, you can get a direct URL to share in any other fashion.  << BONUS!  VERY COOL!   I also hear new tracks from what my friends share.  You can also "friend" and follow other GrooveShark users!   This is a GREAT MUSIC DISCOVERY TOOL!

All of this is FREE! Now, if you are complete music buff, then I would HIGLY recommend that you pay for an account which gives you GrooveShark Mobile! Yes, that's right, a very functional app that allows you to OFFLINE MUSIC! You can Offline songs while you are on WiFi so as to not suck up your data plan. But you are free to use your LTE/4G network to download and use the Radio function as well. It also gives you the option to prefer more compressed streams to conserve..... nice, but, not for me.

GrooveShark has been around for many years now, and I have been a user since the early days. They have grown up! They are now a world class Streaming Service. You can even beta their new interface if you which, which I personally find more functional. To see the new interface, please check out www.grooveshark.com/new

Cheers!

Samsung, In the box... In a good way!

The Samsung UD22.  Finally, manufacturers are thinking outside  the box.    Paradoxical thought brought Samsung back into the box.  Finally a decent 1:1 aspect ration screen!

Some of you may be thinking ok, why?  But if you are thinking of building a video array using many screens, a 1:1 aspect is an essential building block!  Can you imagine Lego with out the square blocks?  Moreover the UD22 1:1 aspect ratio screen is in the pro series.  Meaning it has all the pro features like a robust PSU, DVI Daisy chaining, MagicInfo and a slim bezel (.3mm). 

Although I had the pleasure of visiting Samsung HQ Canada and got to see the device, it was the day after public launch, so I am unable to find the details of the device. 

Since I am planning to use them in a new installation, I will receive details soon.  What I can tell you is, the resolution is 960x960.  The contrast ratio and viewing angle, I cannot give you numbers, but I can say its brilliant,  and the and - GREAT!  Only when looking at it side on do you loose image quality.

What I can share here is an image taken while at the Samsung Offices of the display.  There are 2 of them side by side.  I cant wait to get in there for some scale testing and the actual installation of them!

While I was there I also saw their new Transparent Display Case.  Not unlike the SamilCTS Screen I talked about from Digital Signage Expo earlier this year.

Until Next Time!

//Ian\\

Experiment Over

Firstly, pardon my gap in the experiment.

Day 3 and 4 I did nothing except work (so I was using my traditional linux desktop) so there was really nothing to report.  However moving forward, using my tablet and keyboard only served to slow me down by a large degree.  So I had to abandon because it was more of a hindrance   

But lets look at the functionality, speed of typing aside.

The Good:

• I was able to accomplish everything I started out to do.  This included working on spreadsheets, complex Word documents, SSHing to Unix servers, FTPing and making changed to my web site, and standard media consumption.
• Accomplishing what I set out to do was quite easy and only required downloading a few apps, that I otherwise would not have needed under a Linux environment.
• The Nexus 7 is a very fast device!   In many cases typing aside, i was able to accomplish some things faster than using a traditional desktop.
• I can go a long way without interfacing with the grand grid!  AKA Electricity!   Battery in a tablet is WAY better than a traditional laptop.
• I was fine without a mouse and did not miss it (see the bad)

The Bad

• Typing was very very slow!!!!  This was the main thing that made me stop the experiment.  Although I could 'get by'.  My error rate was way way higher than a traditional keyboard.  This was due to the size of the keyboard.
• Bluetooth wow's.  I am not sure if this is just a bad quality keyboard, however after not typing for about 2 min, the keyboard would sleep.  So when I tapped/clicked on a text feild, the on screen keyboard would pop up until the bluetooth connection was re-established (About 10-20 seconds) at which point the on screen keyboard would go away.
• Using a Bluetooth Mouse and Keyboard at the same time caused communication issues.  (stuck and repeated letters, and no letters)  I would love to know if this is just my combination of hardware.  If anyone out there has a Nexus 7 and a bluetooth keyboard and mouse, pair and use them together and let me know if you see this issue!   Thanks!

In Conclusion, I think this could have been a great success if the following factors were to change.  Most importantly, a full size keyboard that was not prone to the above noted issues.  Secondly, if working to permanently replacing a laptop or a desktop, a 10"+ screen would be a serious asset.  I did get tired of using the 7" screen when working for more than half hour.

Questions Comments?  Lets talk!

//Ian\\