SUMIFS Just Changed My Life

I realize I’m years, decades even, behind on this one, but I just discovered the Excel function SUMIFS and my life will never be the same.

I’m a huge data nerd and so of course I catalog all of my electric bills and electricity usage going back about 6 years now, plus some weather data. I finally got fed up doing my own aggregation columns and searched around to find the solution. Of course Google had the answer immediately.

What’s even more amazing to me is that all of these functions are available in Google Sheets, online, for free.

I realize that with SUMIFS I’m only barely scratching the surface here. The rabbit hole is deep with DSUM, COUNTIF and so many friends.

Someday I should post a view-only link to my data collection sheets.

September 15, 2014

The History of English Podcast

I was recently made aware of the great History of English Podcast.

The author, Kevin Stroud, while not a professional linguist, is a wonderful story teller. Each episode covers a combination of the etymologies of English as well as the history, that is the people, who have spoken this language in its various forms from it’s roots thousands of years ago.

I’m still playing catchup, but even after less than 10 episodes in I’m finding that it has firmly drawn my attention to words, explaining how various words came into English, and why even words with similar meanings sounds so different from one another, i.e. why horse as well as equine.

If you have any interest in history and even a passing curiosity about the English language I would highly recommend that you check it out.

August 28, 2014

Handling Motion JPEG Streams on iOS

I have several Foscam Cameras around the outside of my house. They’re very easy to setup, tolerate the outdoor conditions admirably, and are incredibly affordable for what they offer.

As with everything else around my house, I like to build software that customizes my view into my home (or in this case, outside my home). To that end I’ve build an app I call Argos that lets me monitor all sorts of sensors on my property.

Once I installed the first set of cameras I wanted to be able to implement some views that would display the current video stream from each camera. After looking into the documentation I discovered that the cameras I have offer two types of video streams: windows streaming video (asf) and motion JPEG.

I don’t have a lot of experience writing software to handle video streams. But as I read the basic description it seemed that a motion JPEG stream is just an http stream that continually pushes out a series of jpeg images.

Oh, well that’s easy. Right?

Well, not so fast.

What Is Motion JPEG?

It also turns out that there is no such thing as a true motion jpeg standard. However, there are two typical implementations, Motion JPEG-A and Motion JPEG-B. Motion JPEG-A supports the concept of markers, while Motion JPEG-B does not. This difference is important. For the rest of this discussion however all we need to know is that the Foscam camera stream is Motion JPEG-A.

A Motion JPEG-A stream looks (to me) a lot like a multipart email message. There are several sections, each separated by a long string of semi-random characters. Within each section is some encoded (or not) binary data that represents the object in that section. In our case, each section is a JPEG image.

We can see what this looks like by using the curl command:

{% highlight Shell %} › curl -D - “http://192.168.300.301/videostream.cgi?user=admin&pwd=SECRETS” HTTP/1.1 200 OK Server: Netwave IP Camera Date: Wed, 02 Jul 2014 22:28:03 GMT Accept-Ranges: bytes Connection: close Content-Type: multipart/x-mixed-replace;boundary=ipcamera

–ipcamera Content-Type: image/jpeg Content-Length: 43996

????JFIF???!???

[a lot of binary data]

–ipcamera Content-Type: image/jpeg Content-Length: 44176

????JFIF???!???

[a lot more binary data] {% endhighlight %}

It just goes on and on like this.

We can see in the header of the response that the boundary text will be ipcamera, and the two lines following each boundary include the content type and the content length.

So How Do We Parse This?

This is the basic approach to parsing a data stream like this:

  1. Read in the first chunk of data
  2. Does the chunk contain a boundary marker?
  3. If so, is that boundary marker the first boundary marker?
  4. If it is the first one, then skip it.
  5. Is there another marker? If so, then we have a complete image.
  6. If we have a complete image, find the start and end of the image, remove those from our buffer, and process the image.
  7. If we do not yet have a complete image, append the data to the buffer, and wait for the next chunk of data.

The key here is that we never know how many chunks it will take to make one image. In an ideal world we’d just get one chunk per image and we could throw that right into an NSData object and convert it to a UIImage.

Here’s the code I have so far for parsing the Motion JPEG stream:

{% gist jeffmcfadden/4970e952466d50eaf533 %}

The heart of the code is in func URLSession(session: NSURLSession!, dataTask: NSURLSessionDataTask!, didReceiveData: NSData!) . That’s where we attempt to see if we’ve hit the end of an image, and if so, extract it from the buffer.

Bugs

So far the code works fairly well, except that from time to time when I attempt to make a UIImage out of this I get a failure. I’m not sure if my data out of my camera is bad (unlikely) or if I’m just messing up the process of extracting the data (much more likely).

Improvements

What I’m currently not doing, but probabaly should be doing, is using the Content-Length header to verify the length of the image data before passing it off. I do wonder if that wouldn’t be a far more reliable way to extract the data from the buffer.

Future

Beyond cleaning up the code a bit and trying to make it more reliable, I would love for this view to include some other nice features down the road, like the gesture recognizers to allow me to implement panning/tilt via gesture. Several apps dedicated to IP Camera viewing do this, and it wouldn’t be very difficult at all to get it right.

July 2, 2014

AR.Drone 2.0 Initial Impressions

Update 2014 I would no longer recommend the AR.Drone. I plan to post a followup article explaining why at a later time.

For my birthday this year I decided to get myself a UAV. I had looked at a lot of options, both fixed-wing and various quad/hex/octo copters. In the end I settled on the Parrot AR.Drone 2.0.

Continue Reading...
April 27, 2013

SSH Home via iCloud

Note: As of 2013, this does not work any more

  1. Run this on your home computer: $ mDNS -E
  2. Note the domain at the bottom
  3. Note your home computer hostname $ uname -n And remove the ‘.local’
  4. Now you can ssh to [hostname].[domain] from anywhere if both computers are connected to iCloud.
  5. Profit!
June 5, 2012

Build Your Own High Top Tables

We recently hosted a backyard party, and we wanted some high top tables for people to be able to mingle around. We looked around online but nothing was as simple as we wanted (we were going to cover them with table clothes so they didn’t need to be fancy), and the ones we were able to find at all were pricier than we wanted. So, we did what you do when you want something to exactly match your vision: we built them.

The Materials

  1. Three 1”x4” boards, cut to 40” in length.
  2. Three 3” hinges, with hardware.
  3. One raw round table top, 24” diameter. I found these at Lowes.

The total cost of materials ended up being about $35 per table. The table top was by far the priciest piece, at about $18 each.

Continue Reading...
May 20, 2012

ESV API Gem

This weekend someone tweeted me asking if esv.jeffmcfadden.com was down temporarily or permanently.

The site had actually been down ever since I moved my website off of PHP (a long time ago). I hadn’t bothered to bring the site back, mostly out of laziness/busyness.

Having a couple hours of free time I figured now was as good a time as any to bring the site back. From doing some research a while ago I knew I’d end up writing my own ESV API gem because what was already out there wasn’t really what I was looking for.

I wanted a gem I could easily drop into my rails app that wrapped all the methods in the API in a dead-simple and straightforward way. It needed to work with rails 3.1 on ruby 1.9. I wanted to be able to call the api and get html back that I could just drop into a view.

After a couple of hours I came up with esv_api. To test it out I threw together a quick rails app for the esv site and dropped it onto Heroku.

If you use the gem or find it helpful I’d love to hear from you.

January 23, 2012

Inexpensive Live Streaming For Your Church

Updated April 2013, including new options like an All-In-One solution from LiveStream

Last winter (2010-2011), from January through March, our church attendance was down nearly 30%. Between a few vacations and an intense RSV and Flu season the congregation was knocked down pretty hard. Knocked down and out (of the service), but not completely out of participation thanks to our live stream.

We’ve been livestreaming video of our service since we started 2.5 years ago. When we planted, I couldn’t find any helpful information about livestreaming that wasn’t aimed at large churches with much larger budgets than we were working with. Looking around recently for some equipment I still couldn’t find a good reference. So, here’s a bit of a history of what we’ve done at Grace Church. Hopefully it can be of help!

Getting Started

Diving into livestreaming isn’t free, but even the smallest of churches (we started with 30 people) can participate.

To get started you’ll need someone dedicated to the task of acquiring equipment, learning how it all works together, and training others to help out.

Equipment

At a minimum you’ll need a computer, a camera, possibly a video adapter, an audio feed from your soundboard, an internet connection, and an account with a video streaming service.

Computer

To stream your service you’re going to need a computer. The computer takes the video from the camera and encodes it into a format ready for streaming via the streaming service. To be able to encode video at the quality you probably want, you’ll need a relatively powerful computer. Look for something with at least an Intel Core 2 Duo processor. If you’re already using a computer for projecting words for songs, you might be able to use that computer to do double duty.

We’re a portable church, and we use mostly Apple equipment, so we’re using a 15” Macbook Pro. Our pastor uses it as his computer during the week.

If you’re going to use a firewire video converter like the one below you’ll need a computer with a Firewire 400 or 800 input.

Cost: $0 (If the computer you own will work, or you can borrow one) - $1199

Video Camera

When we got started we just used a cheap Logitech webcam as our video camera. The quality was low. You could barely see the stage. But it worked! It was better than nothing for the family stuck at home with sick kids.

Today you can buy a Canon Vixia Camcorder for under $300. If you’re going to convert your video for uploading later (to a service like vimeo) then you’ll want to get an HD camcorder.

The only requirement of your camcorder is that it has live composite or S-Video output. What this means is that the camera will output a low-def video feed at the same time that it is recording.

If you really have no plans to record your video on the camera, you can ditch the live-output requirement, and maybe even pick up an older still camera with video output.

Also keep in mind that if your camera is going to be in the back of a room you’ll want a high optical zoom (10x+). You do not care at all about the term “digital zoom”. “Digital Zoom” is a hoax. You only care about optical zoom.

Cost: $79-$400+

While you’re at it don’t forget to get a big memory card so you don’t have to deal with running out of space. 64GB SD Cards are getting cheaper every day.

Video Adapter

If you aren’t using a basic webcam, then you’ll need a way to get the video stream from the camera to the computer. This is where the video adapter comes in. If you have a computer with a firewire port of some kind, then go buy one of these.

Updated: If you have USB3 on the computer, then one of these encoders from black magic would make a great solution. Here’s a thunderbolt version if that’s what you have to work with.

Cost: $150+

Internet Connection

To stream live you’ll need an internet connection. For the best quality you’ll need something with at least a 1 Megabit Per Second upload speed. You can get away with a bit slower if you lower the quailty of the video you’re uploading.

Being a portable church, we don’t have any internet available to us in our building. We’re now using a Verizon Jetpack.

Cost: $0 (You already have it)-$60/month

Streaming Service Account

Don’t bother looking around. Just go to livestream.com and sign up now. It’s free ($350/m if you don’t want ads to appear in your stream) and it Just Works. They have native software for both Windows and Mac.

Download the free Procaster software.

Cost: Free

Audio Feed

The last thing you’ll need is an audio feed from your soundboard. Talk to your sound operator about what you’ll need to get a feed to your computer. Usually you’ll just need a simple audio cable and maybe an adapter.

If you don’t have a sound board then you’ll need somekind of mic setup to get the feed for your stream.

If you’re really stuck you could try just pulling in the ambient room audio from a microphone at the computer. This might get you by, but you’ll be dealing with some pretty bad audio, and you’ll pick up voices of anyone whispering nearby (potentially embarrassing!)

Putting It All Together

So you’ve spend somewhere between $79 and $3000 getting the equipment you need to be able to livestream your service. What now?

  1. Take the video (and maybe audio depending on how you want to set things up) out of your video camera and feed it into your video converter (and thus into your computer).

  2. Take the audio feed and either feed it into the camcorder (if you’re going to record live on the camera) or into the computer doing the streaming.

  3. Fireup the Livestream Procaster software and login to your account. Configure any settings under the preference tabs, then click “Go Live”! You’re streaming!

Upgrades

At Grace Church, in the time since we started streaming, we’ve upgraded equipment and updated workflow, but the basics are all the same, and we still don’t have a huge budget.

We upgraded our camera from a webcam to a camcorder + video adapter. We upgraded our tripod recently.

Here are some other options available to you as you upgrade your setup:

  • Get A Better camera
  • Get Multiple cameras
  • Remove the ads from your streaming service
  • Embed the livestream on your website
  • Use the high+mobile quality streaming to give everyone a chance to view your stream wherever they are
  • Promote your stream on Twitter and Facebook

Keep Streaming

This isn’t the post for it, but there are a lot of great reasons to keep streaming your services for your congregation. From the sick or elderly to the mothers’ cry room, there are members of your church that can’t otherwise participate in your Sunday Service. A livestream helps keep them connected and ultimately builds the church. So keep on streaming.

New: All-In-One Solution

If you are looking for a true all-in-one solution, you should checkout the $500 Livestream Broadcaster. You’ll still need an internet connection (until Summer of 2013 when the LTE version comes out) to get out of the building, but if you’re starting from scratch this may greatly simplify things for you.

Questions?

If you have any questions about live streaming feel free to ping me on twitter.

December 31, 2011

How To rsync Your Jekyll Site

So you’ve got your website building statically with jekyll, and now you need to get these files to your webserver, and do it quickly.

First, setup your ssh config file (~/.ssh/config) with an entry for your server (you are connecting with keys and not passwords, right?):

Host 10.9.34.55
  User john doe
  PreferredAuthentications publickey,keyboard-interactive,password
  IdentityFile ~/.ssh/id_rsa
  IdentitiesOnly yes

From there, rsyncing your site is a piece of cake:

cd _site
rsync --delete -ave ssh ./* mydoimain.com:/var/www/www.mydomain.com/

If you’re using this in a cron job then you probably want to do something like this:

rsync --delete -aqe ssh ./* mydoimain.com:/var/www/www.mydomain.com/

so that you aren’t getting any non-error data output.

April 13, 2011

How To Get Free Topo Maps

I really like a good map. A GPS + Laptop is great, but nothing works like a piece of paper when the batteries go dead or your laptop falls into the lake. Unfortunately, finding high quality topo maps for free is a chore.

When I first started searching for topo maps all I was able to find were those provided by Jared Benedict over at redjar.org. Then he started the map ransom project, which finally freed the maps. Now you can download any of the US Quads you want over at libre map. The drawback here is that the topos are all 100dpi scans (for most of the quads). Depending on how you print them, it can be difficult to make out some of the details.

Well, after some more searching I was directed to the USDA Forest Service geospatial data clearinghouse. Here you can download high quality (300dpi mostly) raster TIFs of the same quads, produced by the NFS. One great thing about the NFS maps is that the forest roads are all numbered.

The downside is that some of the NFS maps don’t have roads the USGS topos have, and vice versa. In practice it’s definitely best to have as many maps as you can of an area if you want to have the best coverage of an area.

February 17, 2009

How My Wallet Destroyed 4 Hard Drives

(And how yours could too)

Continue Reading...
February 17, 2009

Geodesic Distance Function in PHP

I was working on some code to process some GPS track logs and I wanted to be able to accurately calculate the distance between 2 points in the log. I did some searches, but there were no obvious solutions written in PHP. However, I did run across Chris Veness’ version in javascript. I worked through his function and mashed it up as PHP. The result seems to work, but for me with slightly less accuracy than the javascript version. At this point the problems seems to be the limit on the number of significant digits that PHP is using vs. JavaScript. If anyone finds any other problems, definitely let me know.

Why is this so great? Not only is this distance based on a curved surface, but you can vary the ellipsoid model being used (see vars $a, $b, and $f) to get a result with even greater accuracy. Just to clarify, the HowFar function does require the points to be in radians, not degrees, so be sure to use the deg2rad function to convert your lat/long into a usable format.

https://gist.github.com/jeffmcfadden/a6c789424cd02ef3ad4a

February 17, 2009

Build Yourself An iPod Shuffle Charger

Summary

The iPod Suffle is a really cool piece of hardware. One of the drawbacks, however, is that it has an internal battery that is recharged via USB power when it is plugged into your computer. This is all good and well except that it requires a good 4 hours to charge the Shuffle from completely dead to completely full. If you’re on the road this can be rather impractical. Thus, there was a need for a batterypack/charger device that can extend the usefulness of the Shuffle. Apple sells such a device but it is rather expensive for what it does. Granted it looks very nice, but I knew I could make the same product for less. And who can give up a chance to solder?

Continue Reading...
February 17, 2009