Category Archives: Coding

Coding, Java, Ruby

FFI for Ruby and an mp4v2 example

1 Comment

Previously on my blog: In my TiVo2Podcast stuff I automated the process of putting chapters around commercials, but had to call out to a small C++ app I wrote to put the chapters in using libmp4v2.

A few weeks ago I was looking at some ruby gems for a project I was working on and stumbled across ffi, a foreign function interface gem for ruby, or as its docs put it: “a ruby extension for programmatically loading dynamic libraries, binding functions within them, and calling those functions from Ruby code.”  As long as you know the function signatures that you need, its pretty trivial to make the calls from Ruby. You do need to be aware of memory management stuff sometimes, but overall its pretty easy, especially for basic use. If you’re only going to be working in Ruby and need access to a C library, this is much easier than mucking with swig, that’s for sure.

The mind-blowing part for me is that the authors of the gem have made it smart enough to know what flavor of ruby vm and platform the code is running in and it does the right thing, no matter if its JRuby or on Windows or whatever. While I haven’t had a chance to use it yet, I suspect this property will be useful with JRuby at work in the future.
Continue reading FFI for Ruby and an mp4v2 example →

Coding, Geek

Google Authenticator (and implementing it on Linux)

10 Comments

A few weeks ago Google brought their two-factor auth product, Google Authenticator, to the iPhone and Android devices.  (There may be other implementations they released, but those are all I’ve actually touched.)  Their immediate use for it was with your google accounts, specifically Google mail seemed to be their target. The day after it was released, I was lucky enough to have it turned on for my accounts and I’ve been using it since then.

Not that I’m an internet security expert, but it’s seems a pretty straight-forward software token implementation. On Google’s side, to seed the software on your smartphone, it uses a QR code. (I point this out as its one of the few uses of QR codes in the wild that I’ve seen that doesn’t make me want to vomit everywhere. For the record, the only other use of QR codes that I have condoned is @tcar using them to rickroll people.) They also give you a hand full of one time password codes for you to print out to keep in your wallet in case you are without your smartphone.

In practice with Google’s accounts, when you log in, you’re additionally asked for the current code.  At that time, you can choose to have google assume the machine you’re on is good for 30 days.  For a home or work machine that isn’t going anywhere, that’s probably safe, but I find myself not checking that box a lot.

The only downside is that any application you have that interacts with a Google service but can’t do a secondary form of authentication, such as an mail/IMAP client, you’ll need to set up “application-specific passwords.” These are passwords that application alone uses to get to your account that you can revoke at any time. Its not difficult, its just tedious as I ended up needing to create 10 different passwords due to the variety of applications I use that interact with Google.  However, now that they are set up, I don’t have to touch them again unless one of the passwords get compromised.

In any case, if you depend on some google services, and you have a smartphone, I highly recommend looking into this with your account.

After using Google Authenticator for a few days with google, I became aware of their project on google code. Besides having the code for the Android and Blackberry applications, it contains code for a PAM module. That really peaked my interest as I’ve always toyed with the idea of implementing two-factor auth at home and on the server I share with my friends, but there hasn’t really been a conveniently deployable way to do it.

Here’s where the linux part starts

There’s no code release for the PAM module yet, so you’ll need to check out a copy using Mercurial.  Once that’s downloaded, you want to make sure you have your PAM development libraries installed.  I also suggest (as the Google wiki’s page says) having libqrencode installed so the google-authenticator command line setup tool can spit a QR code out at you to more easily activate it in your smartphone.  Once those are there, its a pretty easy compile since the Google folks seem to be developing it on Linux. If you’re on another platform, your milage may vary.  I also see via a comment on the wiki page that someone made a Ubuntu ppa of it, so that might simplify matters as well.

By default, the PAM module is all or nothing, so either all your users need to be set up for two-factor auth or no one can be.  There is a patch that allows you to modify this behavior to ignore two-factor auth if it hasn’t been set up or not, which would work for a good transitory period. I haven’t implemented this yet, as its from a recent comment on the wiki page, but when I go to implement this on my shared server I’ll be making use of it.

On my ubuntu server, once the PAM module was installed, I just added the following line to /etc/pam.d/sshd after the existing “@include common-auth“:

auth required pam_google_authenticator.so

Once that was added, I ran the google-authenticator command line tool to create the shared secret and control file for the two-factor authentication.  Once you run it you should see something like this:

Once you say yes to that question and capture the QR code with Google Authenticator on your smartphone, you’re bleepin’ golden.  (Yes, I dummied up an account to generate that, took the screenshot, and then erased that account.  I’m not completely dumb.)

One thing you might need to do is edit your sshd configuration to make sure that ChallengeResponseAuthentication is turned on.  This allows ssh to interactively do extra challenges as required by PAM.  By default this is off in Fedora and Ubuntu.

Coding, Geek, TiVo2Podcast

TiVo2Podcast update

2 Comments

It’s been a long time since I’ve put a new version of TiVo2Podcast out there for people to play with.  I’ve made a lot of changes and tightened things up alot.  There’s still a lot of things I want to do but haven’t gotten to, but I did finally put up a public git repository in case others wanted to join in the fun.

The major changes in this release:

  • Commercials are detected and chapters are added around them.
  • Added “clean up” functionality so that you can deleted unneeded files and the database and rss feeds can reflect that.
  • Increased the wait time for locating the tivo from 2 seconds to 5.
  • Fixed problem created by quotes in the show description.
  • Attempt to avoid re-encoding dupes by checking the program id.
  • Lost of behind the scenes refactoring.

The first item I consider a major enhancement, so I put it in italics.  To make that work, you’ll need to get wine, comskip, and build a helper app.  The documentation is in the hastily wrote README. (A reminder, this is intended for PERSONAL USE ONLY, do not set up podcast feeds and violate the ethics (and also the laws) of copyright left and right.)

As always, this has run daily for months and months on linux. It should work on other UNIXes just fine. On Windows, I have no idea.

Download: tivoscripts-20110123.tar.gz

Coding, Geek, TiVo2Podcast

Addings chapters to an existing mp4/m4v file

1 Comment

In the process of working on my TiVo to Video Podcast stuff I finally got to the point where I wanted to do something about commercials.  The ideal solution would be callable from ruby or at least the command line so that it can be used in an automated manner.  After a few weeks or research and playing around I found a workable solution that required me writing some code.

Early on I made the design decision to put chapter markers in around the commercials rather than cut them out.  The main reason for this is “What if the detection is wrong?”  For a 30 minute show, you’d end up missing more than half it in a worse case scenario.  Since its easy to jump ahead to the next chapter on my iPhone and iPad this seemed like a good decision.

Here’s the three most important facts I learned in my research…

  1. Commercial detection: No question, comskip is the right tool to use here.  The downside is that by default its a windows-only command line tool.  It works perfectly under wine, so that mitigates it.  Right now I use it via wine, but the source is available, so in the long run it would be good to have a native linux binary to call.  Comskip creates a variety of output formats, so I picked one that seemed to be the easiest to work with.
  2. There is no good command line tool to add chapters to an existing mp4/m4v file: I dug around and found a lot of potential solutions, but the all were either not on linux, couldn’t take the resultant files that comskip spit out, or just not a good fit for what I was doing.
  3. The MP4v2 library had primitives for adding the chapters: From this point forward, it was just writing some code that did exactly what I wanted.

The code below expects three arguments: 1) The video file to work on, 2) The chapter file output from comskip in ZoomPlayer chapter format, 3) and the total length in seconds of the video file.  The last one I might be able to remove once I have more brain time to devote to this.

AddChapterInfo.cpp
raw download
#include <fstream>
#include <string>
#include <boost/lexical_cast.hpp>
#include <boost/regex.hpp>
#include <mp4v2/mp4v2.h>
 
// Compile with something like: g++  AddChapterInfo.cpp -o AddChapterInfo -lmp4v2 -lboost_regex
int main(int argc, char *argv[])
{
    char *m4vfilename = argv[1];
    char *chapfilename = argv[2];
    uint32_t total_length = boost::lexical_cast<uint32_t>(argv[3]);
 
    std::ifstream chapfile(chapfilename);
 
    MP4FileHandle m4vfile = MP4Modify(m4vfilename);
 
    // Add the chapter track, have it reference the first track
    // (should be the video) and set the "clock ticks per second" to 1.
    // (We may want to set that to 1000 to go into milliseconds.)
    MP4TrackId chapter_track = MP4AddChapterTextTrack(m4vfile, 1, 1000);
 
    boost::regex chpre("^AddChapterBySecond\\((\\d+),");
    boost::smatch rem;
    std::string s;
    uint32_t last_time = 0;
    while (getline(chapfile, s))
    {
        if (boost::regex_search(s, rem, chpre))
        {
            uint32_t t = boost::lexical_cast<int>(rem[1]) * 1000;
            if (t > 0)
            {
                MP4AddChapter(m4vfile, chapter_track, t - last_time);
                last_time = t;
            }
        }
    }
 
    if (total_length - last_time > 0)
    {
        MP4AddChapter(m4vfile, chapter_track, total_length - last_time);
    }
 
    MP4Close(m4vfile);
    MP4Optimize(m4vfilename);
 
    return 0;
}

I’ve been using this code for over two weeks straight and has been operating perfectly, but obviously this code could be made a lot more robust, especially in the areas of error handling. I’ve only run into issues when comskip guesses commercials wrong, which is only payoff for putting chapters in instead of nuking the commercials all together.

In the long run, I should either write and release a generic tool that helps the next poor sap like me or work on using swig bindings to mp4v2 so I could just do the calls in ruby.

Coding, Entertainment, Geek, Linux, OS X, Podcasts, Ruby, TiVo2Podcast, Unix

Tivo2Podcast update

Leave a comment

I’ve made a few updates since my last release a few weeks ago. I thought I’d toss an updated version out there.  What’s new in this version:

  • Duration is no longer hard-coded to 32:00 and actually reflects the duration of the show
  • The script will attempt to find the TiVo via Bonjour/mDNS/ZeroConf/DNS-SD/whatever unless passed a -t flag with the TiVo’s IP address. If you have more than one TiVo, it will go with the first one it finds.
  • Moved the stuff in lib to lib/tivo so the package is more easier sucked in by something like encap or stow

Download: tivoscripts-20100314.tar.gz

When I get some motivation later in the week, I’ll put the git archive online, incase anyone wants to clone it and do some development on it.

Coding, Entertainment, Geek, iPhone, Linux, OS X, Podcasts, Ruby, TiVo2Podcast, Unix

TiVo -> Video Podcast

1 Comment

Previously on “You can imagine where it goes from here”: We released a script to download stuff from the tivo, and then made some improvements to it.

After two years of saying I was going to fully automate the process of downloading and transcoding shows for my iPhone, I finally got off my ass and did it.  The script is called TiVo2Podcast and it not only does the downloading and transcoding, but it stuffs the resultant video into a an RSS feed for easy consumption/playback by a podcatcher such as iTunes. I’m now automatically getting the shows off my TiVo and onto my iPhone for easy commute-time consumption. (I commute by train, I do not recommend commute-time consumption if you are driving.)

The ruby script wraps tivodecode, HandbrakeCLI, and AtomicParsley and is intended to be run from cron.  I’ve tested this on Linux, but it should run on any UNIX-alike, but it won’t run on windows since I make liberal use of the system() call. Also, this is intended for PERSONAL USE ONLY, do not set up podcast feeds and violate the ethics (and also the laws) of copyright left and right.

This is a very early version and can certainly use some tweaks and enhancements, primarily in configuring the shows you want to capture.  Right now, configuration is in the form of doing INSERT statements in SQLite.  Not very friendly, but it gets the job done until I can make a quick and dirty question based TUI. Here’s an example of setting up getting the best fucking news team on the planet:

INSERT INTO configs (config_name, show_name, rss_filename, rss_link,
                     rss_baseurl, rss_ownername, rss_owneremail, ep_to_keep, encode_decomb)
            VALUES ('tds', 'The Daily Show', 'tds.xml', 'http://www.thedailyshow.com/', 
                    'http://example.com/podcasts/', 'Keith T. Garner', 'kgarner@example.com', 4, 1);

Download tivoscripts-20100304.tar.gz and let me know what you think. Make sure you read the README!

[Update 3/5: Forgot to add that all the code I wrote is under the Simplified BSD License, so have at it.]

Coding, Geek, OS X

Build libcurl as a universal binary

Leave a comment

I release one of the projects from work as a universal binary on OS X.  Up until tomorrow that mean just i386 and ppc.  With snow leopard, it looks like it’ll be a good idea to support the 64-bit architectures as well, especially considering its an ODBC driver I’m working on and the native apps running at 64-bit will want to talk to it that way.

Since we used a lot of open source libraries to save us time, I need to have those built super-universal as well.  The first one I tackled was curl, which had some issues due to configure, so I had to write a shell script to do the hardwork for me.  It needs to run configure three times, and I got a lot of the information for it from http://curl.haxx.se/mail/lib-2009-05/0000.html.

#!/bin/sh
export CFLAGS="-O -g -isysroot /Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.4 -arch i386 -arch ppc"
./configure --prefix=/usr/local/encap/curl-7.19.6 --with-ssl=/usr --without-ca-bundle --disable-dependency-tracking

cp include/curl/curlbuild.h include/curl/curlbuild32.h

make distclean

export CFLAGS="-O -g -isysroot /Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.4 -arch x86_64 -arch ppc64"
./configure --prefix=/usr/local/encap/curl-7.19.6 --with-ssl=/usr --without-ca-bundle --disable-dependency-tracking

cp include/curl/curlbuild.h include/curl/curlbuild64.h

make distclean

export CFLAGS="-O -g -isysroot /Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.4 -arch i386 -arch ppc -arch x86_64 -arch ppc64"
./configure --prefix=/usr/local/encap/curl-7.19.6 --with-ssl=/usr --without-ca-bundle --disable-dependency-tracking

cat > include/curl/curlbuild.h <

There will be more of these as I build the other dependencies.
Coding, Geek, Internet

Direct plugin auto-update on WordPress 2.8

51 Comments

I get a lot of traffic on the site due to my post on getting direct (non-FTP) updates to work on WordPress 2.5.  This method worked up until 2.7.x.  With 2.8 out this week, I found during my svn switch a conflict was created due to a code change in file.php.  Looking at the changes, it looks like the wordpress developers created an easy way for one to short-circuit the update to use the method you want via a setting in wp-config.php.

So, in brief, the permissions and WP_TEMP_DIR settings from the older article still stand.  However, you no longer need to edit wp-admin/includes/file.php. Now you just need to edit your wp-config.php and add the following towards the bottom:

define('FS_METHOD', 'direct');
Coding, Geek, Unix

Fix/Tricks for plugin auto-update on WordPress 2.5

24 Comments

[READ THIS FIRST!!!  Update 6/13/2009: If you’ve come here looking to get plugin updates to work and you’re using WordPress 2.8, you really want to start with this more recent post on the topic and then come back for the permission information.]

One of the neat features of WordPress 2.5 is the click to install plugin upgrades, assuming the plugin is registered in the WordPress Plugin Directory. If certain conditions are correct on the server it can do it in place, otherwise it tries to do it via FTP.

To make it so wordpress could upgrade them on the server without FTP requires doing some permission changes. You should be aware, the changes I made allow the web server (Apache) to be able to write to the plugin directory. This creates some security exposure. Since I do nightly backups, for me this is an acceptable risk. You may make a different call.

The way I’ve done it also assumes you have admin rights on the unix box or you’re friendly with (s)he who does. Without admin rights to do the group ownership changes, you’re stuck having to make files writable by the world, and that’s not something I’d do. Luckily, I hold the power on the box(es) I care about. Continue reading Fix/Tricks for plugin auto-update on WordPress 2.5 →

Coding, Geek, Linux, OS X, Ruby

Place shifting action 3: Revenge of the Sith

2 Comments

tivo.jpgPreviously I put out an early version of my ruby based command line oriented tivo download script.

I’ve had a patch from MARK NOTARUS to make the menu have some more options and I’m using Console::ProgressBar from facets now.

It works well enough for my needs, but let me know if you hit any roadblocks. Just as a reminder, my target was to download stuff off my tivo on one of my headless Linux boxes for batch encoding for my iPhone and/or PSP.

Download tivo-ruby-0.2.tar.gz