SUMIFS Just Changed My Life

I realize I’m years, decades even, behind on this one, but I just discovered the Excel function SUMIFS and my life will never be the same.

I’m a huge data nerd and so of course I catalog all of my electric bills and electricity usage going back about 6 years now, plus some weather data. I finally got fed up doing my own aggregation columns and searched around to find the solution. Of course Google had the answer immediately.

What’s even more amazing to me is that all of these functions are available in Google Sheets, online, for free.

I realize that with SUMIFS I’m only barely scratching the surface here. The rabbit hole is deep with DSUM, COUNTIF and so many friends.

Someday I should post a view-only link to my data collection sheets.

September 15, 2014

The History of English Podcast

I was recently made aware of the great History of English Podcast.

The author, Kevin Stroud, while not a professional linguist, is a wonderful story teller. Each episode covers a combination of the etymologies of English as well as the history, that is the people, who have spoken this language in its various forms from it’s roots thousands of years ago.

I’m still playing catchup, but even after less than 10 episodes in I’m finding that it has firmly drawn my attention to words, explaining how various words came into English, and why even words with similar meanings sounds so different from one another, i.e. why horse as well as equine.

If you have any interest in history and even a passing curiosity about the English language I would highly recommend that you check it out.

August 28, 2014

Handling Motion JPEG Streams on iOS

I have several Foscam Cameras around the outside of my house. They’re very easy to setup, tolerate the outdoor conditions admirably, and are incredibly affordable for what they offer.

As with everything else around my house, I like to build software that customizes my view into my home (or in this case, outside my home). To that end I’ve build an app I call Argos that lets me monitor all sorts of sensors on my property.

Once I installed the first set of cameras I wanted to be able to implement some views that would display the current video stream from each camera. After looking into the documentation I discovered that the cameras I have offer two types of video streams: windows streaming video (asf) and motion JPEG.

I don’t have a lot of experience writing software to handle video streams. But as I read the basic description it seemed that a motion JPEG stream is just an http stream that continually pushes out a series of jpeg images.

Oh, well that’s easy. Right?

Well, not so fast.

What Is Motion JPEG?

It also turns out that there is no such thing as a true motion jpeg standard. However, there are two typical implementations, Motion JPEG-A and Motion JPEG-B. Motion JPEG-A supports the concept of markers, while Motion JPEG-B does not. This difference is important. For the rest of this discussion however all we need to know is that the Foscam camera stream is Motion JPEG-A.

A Motion JPEG-A stream looks (to me) a lot like a multipart email message. There are several sections, each separated by a long string of semi-random characters. Within each section is some encoded (or not) binary data that represents the object in that section. In our case, each section is a JPEG image.

We can see what this looks like by using the curl command:

{% highlight Shell %} › curl -D - “http://192.168.300.301/videostream.cgi?user=admin&pwd=SECRETS” HTTP/1.1 200 OK Server: Netwave IP Camera Date: Wed, 02 Jul 2014 22:28:03 GMT Accept-Ranges: bytes Connection: close Content-Type: multipart/x-mixed-replace;boundary=ipcamera

–ipcamera Content-Type: image/jpeg Content-Length: 43996

????JFIF???!???

[a lot of binary data]

–ipcamera Content-Type: image/jpeg Content-Length: 44176

????JFIF???!???

[a lot more binary data] {% endhighlight %}

It just goes on and on like this.

We can see in the header of the response that the boundary text will be ipcamera, and the two lines following each boundary include the content type and the content length.

So How Do We Parse This?

This is the basic approach to parsing a data stream like this:

  1. Read in the first chunk of data
  2. Does the chunk contain a boundary marker?
  3. If so, is that boundary marker the first boundary marker?
  4. If it is the first one, then skip it.
  5. Is there another marker? If so, then we have a complete image.
  6. If we have a complete image, find the start and end of the image, remove those from our buffer, and process the image.
  7. If we do not yet have a complete image, append the data to the buffer, and wait for the next chunk of data.

The key here is that we never know how many chunks it will take to make one image. In an ideal world we’d just get one chunk per image and we could throw that right into an NSData object and convert it to a UIImage.

Here’s the code I have so far for parsing the Motion JPEG stream:

{% gist jeffmcfadden/4970e952466d50eaf533 %}

The heart of the code is in func URLSession(session: NSURLSession!, dataTask: NSURLSessionDataTask!, didReceiveData: NSData!) . That’s where we attempt to see if we’ve hit the end of an image, and if so, extract it from the buffer.

Bugs

So far the code works fairly well, except that from time to time when I attempt to make a UIImage out of this I get a failure. I’m not sure if my data out of my camera is bad (unlikely) or if I’m just messing up the process of extracting the data (much more likely).

Improvements

What I’m currently not doing, but probabaly should be doing, is using the Content-Length header to verify the length of the image data before passing it off. I do wonder if that wouldn’t be a far more reliable way to extract the data from the buffer.

Future

Beyond cleaning up the code a bit and trying to make it more reliable, I would love for this view to include some other nice features down the road, like the gesture recognizers to allow me to implement panning/tilt via gesture. Several apps dedicated to IP Camera viewing do this, and it wouldn’t be very difficult at all to get it right.

July 2, 2014

AR.Drone 2.0 Initial Impressions

Update 2014 I would no longer recommend the AR.Drone. I plan to post a followup article explaining why at a later time.

For my birthday this year I decided to get myself a UAV. I had looked at a lot of options, both fixed-wing and various quad/hex/octo copters. In the end I settled on the Parrot AR.Drone 2.0.

Why an AR.Drone?

The AR.Drone has a few things going for it as an entry-level UAV. First, it’s wildly popular, so finding information is easy. Second, the mod community is extensive, active, and so well developed you can buy mod kits rather than having to home-brew each and every part yourself. Last, it’s very easy to repair.

What I Bought

I bought the AR.Drone 2.0, which comes with the quadricopter chassis, 2 hulls, 1 battery and 1 charger. I added a second battery and a propeller repair kit. So far I haven’t broken any props, but I did manage to crack the indoor hull on one of my first flights. A bit of tape or hobby glue would be a fine fix, or you can just buy a new hull.

Battery Life

The biggest disappointment to me so far has been battery life. I had read up on the AR.Drone in advance, so I wasn’t surprised when I got the 10-15 minutes of battery that everyone else was getting with the stock batteries. But, it’s still a bummer to have to swap out a battery every 15 minutes or less, and it takes about an hour to charge (per battery), leaving you with less than stellar fly/charge ratios.

One of the next modifications I intend to do is move to a third-party battery pack, hopefully in the 1500+mAh range (See Future Modifications, below), but for now I’m just learning to live with 30 minutes of flight time with the 2 batteries I have.

Ease of Use

The AR.Drone is extremely easy to fly. Once you get your phone or tablet app open and into pilot mode, you press “take off” and the drone pops up and hovers about 1m above the ground. Using accelerometers and the downward-facing camera, the AR.Drone attempts to hover in place any time you let go of the controls (or if communication is lost).

Control is done on the device with a joypad-esque interface. One thumb handles forward/backward and left/right and the other thumb handles up/down and rotation. The preferences allow you to set a maximum rate of change for all the possible controls, as well as a maximum altitude. By turning down the levels you can get a very nice and smooth reaction out of the drone even with the shakiest of fingers.

When you first get started you definitely want to fly somewhere with a lot of space. It takes a while to get used to the right rate of response, and if you’re dealing with range issues (you will be) then there’s yet another factor to deal with. I made the mistake of flying indoors first. Unless you have a pretty big room (with no furniture on the ground) I’d highly recommend using a park or other open space first.

One thing that took me a while to figure out is that the hover-in-place feature of the AR.Drone relies very heavily on the downward-facing camera. If the terrain below the drone does not have much definition (for example, carpet) then the drone will slowly move away from its position, typically in the downwind direction. To prevent this, fly over train with some verifiable patterns (like rocks, etc.) or put your own objects down on the ground under the drone (anything from a frisbee or a piece of paper will work). Also, walking under the drone (or having pets or small children doing the same) will confuse the drone’s position-holding software, and it will think it’s moving when it’s not.

If you’ve been reading around about the AR.Drone you might have read about “Absolute Control” mode. In this mode, no matter what orientation the drone is in, you can move it relative to you as opposed to relative to the drone. Sadly, this mode no longer exists in the app due to patent litigation. If you have a jailbroken device you might be able to find an old .ipa that has this feature. I’m not sure what the situation is on Android.

Range

Range was the second most disappointing feature for me. It definitely performs as advertised right around 50m of range, but I wanted a lot more than 50m. This led to my first modification, a wifi range extender. Using the range extender I have been able to get more like 100-150m of range. Once I upgrade the built-in antenna with the wheel antenna modification, things should improve to 300m+.

If you spend some time poking around on YouTube you’ll find videos like this one where people are getting 1km+ out of their drones. That’s pretty fantastic.

If Parrot comes out with the GPS module that they’ve promised then the range issue might subside because you’ll be able to use QGroundControl to setup a flight plan and send the drone off on its own without requiring constant communication.

Video Quality

Video quailty with the downward facing camera is a joke. It’s useful to see what the drone is seeing, and occasionally you get interesting results from high altitudes, but you’d never use it for much of anything.

Now, the front-facing HD camera, on the other hand, is great. The quailty is excellent. It adjusts quickly to changing light levels and has great daylight performance. Low-light performance is extremely grainy, but night flying is itself extremely challenging; I’d recommend using it during the day only.

The Jelly Effect can be fairly strong depending on conditions, which is frustrating when you think you’re getting a smooth shot and it ends up smushy. Generally it’s been plenty smooth for me, with jelly happening during major maneuvers while my steady shots have been steady (windy conditions being an exception sometimes).

If you want to see what typical quality is like, checkout the videos on youtube.

The manual recommends that if you want to store your videos, use a USB stick instead of recording them over wifi to your device. After trying both methods I would definitely recommend using the USB stick. You don’t have to compete with the video recording for resources, improving your control connection during flight. And at $13 for a USB stick that’s just barely big enough to fit into the USB slot, it’s hard to argue with this approach.

Future Modifications

With flight times being so short with the stock batteries, adding third party batteries is a must. I intend to look into this upgrade next.

After adding the range extender it quickly became obvious that the drone itself needs an antenna upgrade. I intend to use the advanced antenna mod from dronemods.

If Parrot ever comes out with the GPS dongle I will definitley be upgrading. A fully autonomous UAV is the ultimate goal here, of course.

Failing the GPS module ever coming out, I might attempt to integrate ArduPilot. That would be a lot more work, but also probably pretty fun.

A lot of people online have done bearings/bushings/gears upgrades, with various claims of battery life extension, flight smoothness, etc. I’m fairly skeptical of most of these claims, but may try at least the bearings out at some point.

Tips

Here are a few tips for anyone getting started with their new AR.Drone:

  1. Get a USB stick and use that to record your videos. It will save you WiFi bandwidth and improve your connection throughout your flight.
  2. If you’re going to use a USB stick, wait to connect it until after the drone boots up and the lights are green. You’ll know the drone has recognized the USB stick when the USB icon shows up in white in the toolbar in the pilot app.
  3. The drone will not hover in place very well without well-defined terrain beneath it. Keep this in mind when you choose a place to hover, especially when you’re first getting started.
  4. A WiFi Range Extender really makes the drone more fun. Get one if you live in an area with a lot of 2.4 GHz signals.
  5. Buy more batteries. The right play here is probably to buy a handful of low-cost 3rd party batteries with a solid Li-Poly charger. I intend to do this at some point and will write about my findings.

Final Thoughts

If you’re looking to get into RC flying and you’d like something that’s easy to fly with full UAV potential, the AR.Drone is a great place to start. It’s not perfect, and you can’t yet mount anti-tank missles on it, but for the price and moddability it just can’t be beat.

★★★★☆ Highly recommended.

April 27, 2013

SSH Home via iCloud

Note: As of 2013, this does not work any more

  1. Run this on your home computer: $ mDNS -E
  2. Note the domain at the bottom
  3. Note your home computer hostname $ uname -n And remove the ‘.local’
  4. Now you can ssh to [hostname].[domain] from anywhere if both computers are connected to iCloud.
  5. Profit!
June 5, 2012

Build Your Own High Top Tables

We recently hosted a backyard party, and we wanted some high top tables for people to be able to mingle around. We looked around online but nothing was as simple as we wanted (we were going to cover them with table clothes so they didn’t need to be fancy), and the ones we were able to find at all were pricier than we wanted. So, we did what you do when you want something to exactly match your vision: we built them.

The Materials

  1. Three 1”x4” boards, cut to 40” in length.
  2. Three 3” hinges, with hardware.
  3. One raw round table top, 24” diameter. I found these at Lowes.

The total cost of materials ended up being about $35 per table. The table top was by far the priciest piece, at about $18 each.

The Build

  1. Cut your boards to length (40”), if you haven’t already.
  2. Use a square or protractor and mark the exact center of the underside of your table top.
  3. Position your hinges in a triangle around the center mark, with about 1/8” space between the hinges. See the picture below for an example.
  4. Mark the holes on the table top for the hinges with a pencil or pen.
  5. Remove the hinges and drill pilot holes for your screws. Here’s a guide for choosing the right size pilot hole. If you skip this step, you will likely crack your table top.
  6. For each board, lay a hinge on the end, making sure that you have the orientation correct. Compare with the photos below if you’re unsure.
  7. Mark the holes on the boards for the hinge with a pencil or pen.
  8. Remove the hinges and drill pilot holes for your screws. Here’s a guide for choosing the right size pilot hole. If you skip this step, you will likely crack your table leg.
  9. Get out your screws and attach the hinges to the boards first, screwing them down snug but not so tight that you crack the wood or strip the hole. Be sure you have the hinge oriented properly (when attatched to the table top the hinges should swing away from the table freely, see the photos below to compare).
  10. One by one, screw the hinges to the table top, again double checking your orientation. You may need an extra hand to hold up the legs as you finish if you dont have enough space to lay out the legs flat.
  11. Now gather the legs carefully together and flip the table over so that it is upright. Spread out the legs so that they are even with the edge of the table top. Adjust the legs a bit as necessary to create a level table top. You’re done! 1

The Result

  • The Final Product
  • Top Down View
  • Underside Of The Table
  • The Table Leg
  1. We left ours unfinished, but if you plan to use them without tableclothes or leave them outside you should paint or stain them. Also, if you plan to use them on a very smooth surface, you’ll need to find some rubber feet or build a strap to prevent the legs from slipping out.

May 20, 2012

ESV API Gem

This weekend someone tweeted me asking if esv.jeffmcfadden.com was down temporarily or permanently.

The site had actually been down ever since I moved my website off of PHP (a long time ago). I hadn’t bothered to bring the site back, mostly out of laziness/busyness.

Having a couple hours of free time I figured now was as good a time as any to bring the site back. From doing some research a while ago I knew I’d end up writing my own ESV API gem because what was already out there wasn’t really what I was looking for.

I wanted a gem I could easily drop into my rails app that wrapped all the methods in the API in a dead-simple and straightforward way. It needed to work with rails 3.1 on ruby 1.9. I wanted to be able to call the api and get html back that I could just drop into a view.

After a couple of hours I came up with esv_api. To test it out I threw together a quick rails app for the esv site and dropped it onto Heroku.

If you use the gem or find it helpful I’d love to hear from you.

January 23, 2012

Inexpensive Live Streaming For Your Church

Updated April 2013, including new options like an All-In-One solution from LiveStream

Last winter (2010-2011), from January through March, our church attendance was down nearly 30%. Between a few vacations and an intense RSV and Flu season the congregation was knocked down pretty hard. Knocked down and out (of the service), but not completely out of participation thanks to our live stream.

We’ve been livestreaming video of our service since we started 2.5 years ago. When we planted, I couldn’t find any helpful information about livestreaming that wasn’t aimed at large churches with much larger budgets than we were working with. Looking around recently for some equipment I still couldn’t find a good reference. So, here’s a bit of a history of what we’ve done at Grace Church. Hopefully it can be of help!

Getting Started

Diving into livestreaming isn’t free, but even the smallest of churches (we started with 30 people) can participate.

To get started you’ll need someone dedicated to the task of acquiring equipment, learning how it all works together, and training others to help out.

Equipment

At a minimum you’ll need a computer, a camera, possibly a video adapter, an audio feed from your soundboard, an internet connection, and an account with a video streaming service.

Computer

To stream your service you’re going to need a computer. The computer takes the video from the camera and encodes it into a format ready for streaming via the streaming service. To be able to encode video at the quality you probably want, you’ll need a relatively powerful computer. Look for something with at least an Intel Core 2 Duo processor. If you’re already using a computer for projecting words for songs, you might be able to use that computer to do double duty.

We’re a portable church, and we use mostly Apple equipment, so we’re using a 15” Macbook Pro. Our pastor uses it as his computer during the week.

If you’re going to use a firewire video converter like the one below you’ll need a computer with a Firewire 400 or 800 input.

Cost: $0 (If the computer you own will work, or you can borrow one) - $1199

Video Camera

When we got started we just used a cheap Logitech webcam as our video camera. The quality was low. You could barely see the stage. But it worked! It was better than nothing for the family stuck at home with sick kids.

Today you can buy a Canon Vixia Camcorder for under $300. If you’re going to convert your video for uploading later (to a service like vimeo) then you’ll want to get an HD camcorder.

The only requirement of your camcorder is that it has live composite or S-Video output. What this means is that the camera will output a low-def video feed at the same time that it is recording.

If you really have no plans to record your video on the camera, you can ditch the live-output requirement, and maybe even pick up an older still camera with video output.

Also keep in mind that if your camera is going to be in the back of a room you’ll want a high optical zoom (10x+). You do not care at all about the term “digital zoom”. “Digital Zoom” is a hoax. You only care about optical zoom.

Cost: $79-$400+

While you’re at it don’t forget to get a big memory card so you don’t have to deal with running out of space. 64GB SD Cards are getting cheaper every day.

Video Adapter

If you aren’t using a basic webcam, then you’ll need a way to get the video stream from the camera to the computer. This is where the video adapter comes in. If you have a computer with a firewire port of some kind, then go buy one of these.

Updated: If you have USB3 on the computer, then one of these encoders from black magic would make a great solution. Here’s a thunderbolt version if that’s what you have to work with.

Cost: $150+

Internet Connection

To stream live you’ll need an internet connection. For the best quality you’ll need something with at least a 1 Megabit Per Second upload speed. You can get away with a bit slower if you lower the quailty of the video you’re uploading.

Being a portable church, we don’t have any internet available to us in our building. We’re now using a Verizon Jetpack.

Cost: $0 (You already have it)-$60/month

Streaming Service Account

Don’t bother looking around. Just go to livestream.com and sign up now. It’s free ($350/m if you don’t want ads to appear in your stream) and it Just Works. They have native software for both Windows and Mac.

Download the free Procaster software.

Cost: Free

Audio Feed

The last thing you’ll need is an audio feed from your soundboard. Talk to your sound operator about what you’ll need to get a feed to your computer. Usually you’ll just need a simple audio cable and maybe an adapter.

If you don’t have a sound board then you’ll need somekind of mic setup to get the feed for your stream.

If you’re really stuck you could try just pulling in the ambient room audio from a microphone at the computer. This might get you by, but you’ll be dealing with some pretty bad audio, and you’ll pick up voices of anyone whispering nearby (potentially embarrassing!)

Putting It All Together

So you’ve spend somewhere between $79 and $3000 getting the equipment you need to be able to livestream your service. What now?

  1. Take the video (and maybe audio depending on how you want to set things up) out of your video camera and feed it into your video converter (and thus into your computer).

  2. Take the audio feed and either feed it into the camcorder (if you’re going to record live on the camera) or into the computer doing the streaming.

  3. Fireup the Livestream Procaster software and login to your account. Configure any settings under the preference tabs, then click “Go Live”! You’re streaming!

Upgrades

At Grace Church, in the time since we started streaming, we’ve upgraded equipment and updated workflow, but the basics are all the same, and we still don’t have a huge budget.

We upgraded our camera from a webcam to a camcorder + video adapter. We upgraded our tripod recently.

Here are some other options available to you as you upgrade your setup:

  • Get A Better camera
  • Get Multiple cameras
  • Remove the ads from your streaming service
  • Embed the livestream on your website
  • Use the high+mobile quality streaming to give everyone a chance to view your stream wherever they are
  • Promote your stream on Twitter and Facebook

Keep Streaming

This isn’t the post for it, but there are a lot of great reasons to keep streaming your services for your congregation. From the sick or elderly to the mothers’ cry room, there are members of your church that can’t otherwise participate in your Sunday Service. A livestream helps keep them connected and ultimately builds the church. So keep on streaming.

New: All-In-One Solution

If you are looking for a true all-in-one solution, you should checkout the $500 Livestream Broadcaster. You’ll still need an internet connection (until Summer of 2013 when the LTE version comes out) to get out of the building, but if you’re starting from scratch this may greatly simplify things for you.

Questions?

If you have any questions about live streaming feel free to ping me on twitter.

December 31, 2011

How To rsync Your Jekyll Site

So you’ve got your website building statically with jekyll, and now you need to get these files to your webserver, and do it quickly.

First, setup your ssh config file (~/.ssh/config) with an entry for your server (you are connecting with keys and not passwords, right?):

Host 10.9.34.55
  User john doe
  PreferredAuthentications publickey,keyboard-interactive,password
  IdentityFile ~/.ssh/id_rsa
  IdentitiesOnly yes

From there, rsyncing your site is a piece of cake:

cd _site
rsync --delete -ave ssh ./* mydoimain.com:/var/www/www.mydomain.com/

If you’re using this in a cron job then you probably want to do something like this:

rsync --delete -aqe ssh ./* mydoimain.com:/var/www/www.mydomain.com/

so that you aren’t getting any non-error data output.

April 13, 2011

How To Get Free Topo Maps

I really like a good map. A GPS + Laptop is great, but nothing works like a piece of paper when the batteries go dead or your laptop falls into the lake. Unfortunately, finding high quality topo maps for free is a chore.

When I first started searching for topo maps all I was able to find were those provided by Jared Benedict over at redjar.org. Then he started the map ransom project, which finally freed the maps. Now you can download any of the US Quads you want over at libre map. The drawback here is that the topos are all 100dpi scans (for most of the quads). Depending on how you print them, it can be difficult to make out some of the details.

Well, after some more searching I was directed to the USDA Forest Service geospatial data clearinghouse. Here you can download high quality (300dpi mostly) raster TIFs of the same quads, produced by the NFS. One great thing about the NFS maps is that the forest roads are all numbered.

The downside is that some of the NFS maps don’t have roads the USGS topos have, and vice versa. In practice it’s definitely best to have as many maps as you can of an area if you want to have the best coverage of an area.

February 17, 2009

How My Wallet Destroyed 4 Hard Drives

(And how yours could too) <div class="quickJump" style="margin-top: 15px;padding: 5px; background-color: #DDD;border: 1px solid #CCC"> Quick Jump: The Experiment | The Test Video | Conclusion </div>

Hard disk drives are a marvel of modern technology. With the ability to store the data of millions of books on a single hard drive, we’ve never been able to store more data at a lower price than we are now, and storage capacity is only going up while prices are only going down.

Even more amazing is the sheer complexity at work in a modern drive. The electrical, chemical, mechanical, computer, and magnetic engineering that must take place to develop a drive boggles the mind. A hard disk drive is truly a monument to modern technological achievement, and at only a couple hundred dollars a piece, most people have at least a handful in the various electrical devices around the house. From the TiVo to the family laptop to a camcorder to an iPod, hard drives are in more and more of devices we own.

With access to such cheap and powerful storage, it’s easy to overlook the fragility of modern disk storage — a lesson I learned the hard way.

In April of 2007 I received my first MacBook Pro from Apple. I was upgrading from a Powerbook G4 I had owned for 2 years, which had served me with only a couple of part failures (a bad battery and a hard drive that failed after a year). I took the new laptop home and started copying my data over. After two days I had it all setup. It was working wonderfully.

The Story

About 7 days after I brought it home I was checking my email with the computer on my lap when I heard a loud PING! cry out from behind the case. It certainly didn’t sound like a good noise, and the grinding that followed was even worse. My programs started to lock up and my computer would not restart. Dejected, I took the laptop to the Apple store, where a new hard drive was installed. Two days later I took the laptop back home and spent days reinstalling software. I was partially surprised that there was a failure so soon in the life of the laptop, but hard drives do die, and some die quickly.

Back up and running things were going well until 4 days later when it happened again. I was opening up the laptop when PING! went the computer. The repair kept me computerless for 5 days.

With the second failure I was skeptical that this computer would ever work right, and my doubts were validated when the THIRD drive failed after only another 10 days. Same as the first two, pinging, clicking, and grinding signaled the death of my laptop.

This time the technicians at the Apple store heard my cries and replaced the unit. I walked home with a brand new sealed in box laptop, glad that my hard drive problems were over.

At least for 5 days anyway.

That’s when it happened for the FOURTH time, on a completely new computer. I was surfing the web on a Saturday when I heard the ping of death. My heart stopped and I wondered if I was cursed.

When I took the laptop in they interrogated me for a while and eventually I left with a third brand new laptop (thank you Apple). As I was walking out of the store one of the questions a tech asked me rang again in my mind: ‘Did you expose the laptop to any strange magnetic fields?’ Of course my answer was ‘No.’ I couldn’t think of one. But then as I was leaving it occurred to me that my money clip wallet, which sits in my front pocket, does have a magnet. ‘But surely the laptop is shielded enough that couldn’t matter, could it?’ I wondered.

As I pondered more and more it seemed plausible. Every failure followed a session using the notebook on my lap. And as I read up on hard drive operation it seemed evident that if you could get a magnetic field to reach into the drive, you could definitely destroy it.

That’s when I realized that we had to do an experiment. What we discovered is terrifying: modern hard drives are much more fragile than we often consider, and exposure to even weak magnetic fields from close range could render a working drive absolutely useless.

The Experiment

While it was suggested to me to purchase a notebook hard drive, expose it to a magnetic field, and see if we could break it, then return it to the store of purchase, I thought that a bit risky, unethical, and potentially pricey. Instead, I opted to do some simple testing under the premise that if we can get a magnetic field into the drive operating area, then we can conclude that it is plausible that said magnetic field could ruin the drive. In other words, if the magnet can affect the drive at all, consider the drive dead.

Enter my old Hitachi 2.5′ notebook drive. It failed after one year of service, and I opened it up to see how it looked inside. Since the drive was already dead, it seemed like the best test subject.

Here is the plan:
1. Place iron filings onto disk surface.
2. Expose disk to magnets from varying distances.
3. Look for fluctuations in filings when exposed to fields.

First, we get some filings (filed from a piece of steel lying around) and dump them onto the drive. Now, we get out the money clip. First we’ll test with the magnet that I think might have killed my computers, then we’ll test with other sources if that yields no results.

The first test is startling. Not only does the field easily penetrate the bottom of the drive, it’s incredibly strong at the point of the top of the platter.

Moving the magnet away continues to yield disturbing results. We were able to make the pieces dance from a range of upwards of 5cm away!

And lest anyone ask, the top of the case has no obvious shielding properties.

Test Video

So What?

In plain english, any moderate magnetic field, from a speaker to a fridge magnet, could potentially disrupt the magnetic field of a hard drive, rendering it unreadable. Keeping all magnets of any kind far away from drives is vital.

The Conclusion

Hard disk drives are remarkable pieces of electronics. They incorporate a variety of technologies to store the valuable information that we create each day. But for all their amazing qualities, we find them to be startlingly fragile. If ever you were looking for evidence to reinforce the need to spread data around as you back it up, look no further than the above.

As for my laptops, most likely the money clip’s magnet interfered with the operation of my hard drives (all four of them), causing the destructive effects that I witnessed. While perhaps not perfectly confirmed, this myth is definitely plausible.

Appendix: Obvious Questions

Since undoubtedly someone will ask these questions, or argue that this test is unreasonable, here are my thoughts on the most likely rebuttals.

1. You didn’t prove the drive would actually die. You just moved metal filings.

True. But if a magnetic field can get into the platter region it can interfere with the fields already in place, and since both movement and data storage/retrieval rely on magnetic actuation, it is perfectly reasonable to conclude that the drive would be damaged in some way.

2. You didn’t take the laptop case into account.

Also true. However, in the case of my powerbook, the aluminum case provides little to zero effective shielding. The hard drive is located on the bottom of the case, in the front left corner. The effective distance in a real world scenario would be approximately 1cm. We were able to move filings at least 5cm away from the magnet.

3. Not all laptops have the drive in a similar location.

Also true. And that does matter. Had the drives been on the other side of the case they probably never would have been affected by the magnet in my pocket. But since the magnet could easily penetrate an entire plastic case from bottom to top, no modern unshielded laptop seems secure.

Additional Information

Flickr Photo Set
The Flickr set with all the photos taken for the experiment.

February 17, 2009

Geodesic Distance Function in PHP

I was working on some code to process some GPS track logs and I wanted to be able to accurately calculate the distance between 2 points in the log. I did some searches, but there were no obvious solutions written in PHP. However, I did run across Chris Veness’ version in javascript. I worked through his function and mashed it up as PHP. The result seems to work, but for me with slightly less accuracy than the javascript version. At this point the problems seems to be the limit on the number of significant digits that PHP is using vs. JavaScript. If anyone finds any other problems, definitely let me know.

Why is this so great? Not only is this distance based on a curved surface, but you can vary the ellipsoid model being used (see vars $a, $b, and $f) to get a result with even greater accuracy. Just to clarify, the HowFar function does require the points to be in radians, not degrees, so be sure to use the deg2rad function to convert your lat/long into a usable format.

https://gist.github.com/jeffmcfadden/a6c789424cd02ef3ad4a

February 17, 2009

Build Yourself An iPod Shuffle Charger

Summary

The iPod Suffle is a really cool piece of hardware. One of the drawbacks, however, is that it has an internal battery that is recharged via USB power when it is plugged into your computer. This is all good and well except that it requires a good 4 hours to charge the Shuffle from completely dead to completely full. If you’re on the road this can be rather impractical. Thus, there was a need for a batterypack/charger device that can extend the usefulness of the Shuffle. Apple sells such a device but it is rather expensive for what it does. Granted it looks very nice, but I knew I could make the same product for less. And who can give up a chance to solder?

Alternatives

Hack-A-Day has a similar project. The trick is that if you want the shuffle to function properly you have to tie both data pins (green and white wires) to ground. Pick you poison, this is the only trick you need to be aware of when making a Shuffle-specific charger.

Ingredients

To make a Shuffle Charger you will need the following:

  • 1 +5VDC Voltage Regulator (RadioShack #276-1770) - $1.59
  • 1 9v Battery Snap Connector (RadioShack #270-325) - $1.99 for a 5 pack
  • 1 Project enclosure (I used a RadioShack #270-1802) - $2.69
  • 1 General purpose PC Board (*optional) (RadioShack #276-149) - $1.79
  • 1 9v battery - $2.99 or less
  • A female type A usb connector. You can find this off of a USB extension cable, or maybe some old junk hardware somewhere. - $0.00 - $6.99

Total for parts: About $12 - $15.

Tools

Additionally you will need the following tools:

  • Soldering Iron
  • Solder
  • Possibly some very high gauge wire (26 - 28 gauge)
  • Screwdriver
  • Volt Meter

Disclaimer

Look, I hardly know what I’m doing. This worked for me, but it might not work for you. If you solder together your own charger and something goes wrong and you fry your Shuffle don’t come crying to me. You take full responsibility for messing up your own Shuffle if you try this. If you are worried, buy apple’s battery pack and don’t take the chance. You have been warned.

Assembly

Note: Read all directions before beginning!

First, lay all the parts out on the table, and make sure that you have everything accessible.

Cut your usb cable in half and peel back the insulation so that you have all the wires out.

Strip the ends of the wires on the usb cable. Take the green, white, and black wires and twist them together.

Heat up the soldering iron and get it ready. Remember to always use saftey glasses when soldering!

Solder overview

The first solder you’re going to make is from the 9v battery clip to the voltage regulator. Solder the RED (+) wire to the INPUT on the voltage regulator. Check the back of the box to make sure, but this is usually the left-most pole.

Next, solder the white, black, and green usb twisted wires, AND the ground (-) from the 9v clip to the ground on the voltage regulator. This is probably the middle pole on the voltage regulator.

Finally, solder the (+) RED wire on the USB cable to the OUTPUT of the voltage regulator. This is probably the rightmost pole.

Solder overview

Next, double check your solder connections to make sure they are correct and there is no solder bleed-over.

Connector pinoutNow connect the + probe on your volt meter to pin 1 on the usb connector, and the - (black) probe to the other pins. On each pin the volt meter should read +5v. It is critical that you verify the output voltage for each pin. If you mess this up you could break your Shuffle!

If you are satisfied (and brave), connect the Shuffle to the usb connector. It should blink green, then yellow and green, and finally blink yellow steadily. If it does this, then your charger works. Congratulations! If not, check your solder joints and get your volt meter out again. And pray you didn’t fry your shuffle!

Case being packed

Finally, mount the parts inside the project case. I cut a notch on one end just wide enough for my usb connector, and filled the empty space with foam I had lying around from something I had shipped to myself. There’s no perfect way to do this. Just do what you think works. Heck you don’t have to mount it at all if you don’t want to!

Okay, now grab a beer and watch as your yellow light blinks on-off-on-off-on-off-on-off…

All connected

All connected

February 17, 2009