May 11, 2007

Convert iTunes playlists to Slimserver m3u playlists

Cool gadgets | XSLT

I own few Slimdevices Squeezeboxes, which provide a great way to listen to your digital music. Through a regular remote control and the VCD screen you can choose the album, song or playlist that you want to listen to.

I rip all my music from CDs and organize and store it in iTunes. (By the way, the reason I never buy music from the iTunes Music Store is that I cannot play the DRM crippled songs on my squeezeboxes.) I then copy all the music to the hard disk of a small machine, which runs slimserver that serves the music to my squeezeboxes.

One thing that kind of bothered me for a long time was that iTunes playlist support for the slimserver software was somewhat broken. So I wrote a small script to convert the iTunes playlists into m3u playlists, the format used by slimserver. Below are the details.

Update (May 21st, 2008): Mike Hudson writes that the latest version of SqueezeCenter can now read iTunes XML, playlists included. However one still needs to change the location of the files within the XML file.

iTunes stores information about your music library into an XML file. Its format is the so-called plist, a rather unusual XML format. Its biggest flaw is that name value pairs are consecutive XML elements, rather than being associated. For example, this is an excerpt from the description of a song:

  <key>Track ID</key><integer>585</integer>
  <key>Name</key><string>Hells Bells</string>

The problem of is that you have to rely on the position of elements to figure out what they mean. An alternative would have been to represent the value as an attribute of the key element or as a child element, if it was a complex structure. Like this:

<key name="585">
    <key name="Track ID" integer="585"/>
    <key name="Name" string="Hells Bells"/>
    <key name="Artist" string="AC/DC"/>

In any case, the format is what it is, so I had to deal with it. I wrote a small XSLT program to extract the playlist information and generate the m3u playlists. It is called iTunes-to-m3u.xsl, and you can run it like this to generate the m3u playlists:

java -jar saxon8.jar ~/Music/iTunes/iTunes\ Music\ Library.xml iTunes-to-m3u.xsl

The XSLT engine used above is Saxon 8, an excellent XSLT 2.0 implementation. You might be able to use other engines too, including 1.0 ones. However if you do that you need to change the xsl:result-document to xsl:document, and make sure the engine support outputting to multiple files.

Depending on your setup, you might need to translate %20 sequences in file names to regular blanks. Since the location of the files on the slimserver box is different than that on my Mac, I also have to translate file paths. The following shell commands do it for me (I have everything packaged nicely in a shell script):

(cd /tmp
mkdir -p playlists
cd playlists
rm -f *.m3u *.new
~/bin/saxon ~/Music/iTunes/iTunes\ Music\ Library.xml ~/src/iTunes-to-m3u.xsl;
for f in *.m3u; do
  sed -e 's/%20/ /g' -e 's%/Volumes/BigDisk/Users%/home%g' <"$f" >"$f.new";
  mv "$f.new" "$f"
scp *.m3u music:~/Music/Playlists
Posted by ovidiu at 09:30 PM | Comments (0) |

October 28, 2003

XSLT whines

Java | XSLT

Bertrand writes:

According to Ugo Cei on the cocoon-users mailing list, XML isn't worth "a pile of fetid dingo kidney" (TM) as a scripting language. After writing lots of XSLT lately, I must agree.

I cannot agree more, although I very much like the data-driven execution model in XSLT. I also like the XPath data accessing model.

The top two things I don't like about XSLT are:

  • The XML syntax, which makes programs very verbose.
  • The extensibility mechanism is language agnostic, but it makes it difficult to write complex XSLT/Java programs that integrate nicely.

From this perspective, the SXML processing tools available for Scheme are a very nice integration of XML in a programming language. In SXML the XML processing is described as a real Scheme program that has access to all the library functions for traversing and accessing the XML tree. I wish Java had something of a similar simplicity, but instead it got all the DOM and SAX crap, and no portable XPath library implementation.

What is even scarier is that XSLT 2.0 is moving towards becoming a full programming language with an XML syntax.

One thing I started to pay more attention to is XQuery, and the excellent implementation done in Qexo by Per Bothner. XQuery promises to solve some of the XSLT verbosity I mentioned above. The processing model is closer to functional programming, although it does appear to loose the data-driven processing model I like in XSLT.

Posted by ovidiu at 02:38 PM |

May 12, 2003

More on advanced XSLT usage

Fun at Sand Dunes - Death Valley, California

I found this old article by Jason Diamond via Leigh Dodds. The article describes the same technique I was referring to in a previous entry. The article refers to an even older article by Eric van der Vlist describing the same technique.

The idea is to come up with your own set of XML elements that describe best your problem domain, annotate your HTML pages with these XML elements, and then use an XSLT stylesheet to transform the annotated HTML page into another XSLT stylesheet. This last XSLT stylesheet contains all the presentation information from the original HTML; it is applied on the real input data model to generate the output HTML.

A traditional user of XSLT would write this last stylesheet by hand, mixing the presentation with the actual code that traverses the data structure to be represented. To do this, the developer would first have to reverse engineer the structure of the HTML page, and embed that structure into the stylesheet. The process is time consuming, frustrating and error-prone; the result is ugly and hard to maintain.

Annotating an HTML page and generating the stylesheet automatically is a much easier solution to maintain in the long run. The HTML page designer doesn't have to know XSLT, they would follow exactly the same design process as before. The only thing they need to learn are the extension XML elements that describe where and how the dynamic data will be placed into the HTML page.

The challenge is on the developer of the extension XML elements to be able to cover all the corner cases, and make it possible for the designer to easily embed the data in the page templates.

Posted by ovidiu at 12:59 AM |

May 09, 2003

Processing XML configuration files using XSLT

Java | XSLT
Guardian at the Japanese Garden in Huntington Library, Pasadena CA

Charles Miller decided to use Scheme to describe configuration files instead of XML.

I have a different approach. Since most of the applications I write these days end up processing XML anyways, I prefer writing XML configurations file. Since I really like XSLT's power and to avoid the proliferation of tools like commons-digester and others, I write a simple XSLT stylesheet to process my configuration files. Such a stylesheet would need to receive as parameter a context Java object which needs to be initialized according to the configuration file.

The object initialization could be written in simple XSLT templates. Using Charles' example, lets say we have a hypothetic add-module XML element with a module attribute. We'd like to have a MyClass instance that gets invoked with a addModule method and passed the string value of module. The following XSLT stylesheet would do it:

<xsl:stylesheet version="1.0"

<!-- This is the MyClass instance to be initialized -->
<xsl:param name="context"/>

<xsl:template match="add-module">
<xsl:value-of select="MyClass:addModule($context, @module)"/>


An instance of MyClass would need to be passed as parameter to the XSLT stylesheet from the Java side of the world. This can be set using the Transformer.setParameter(String, Object) method.

XSLT comes for free in Java 1.4, so there is no memory overhead from using XSLT. Even so, I still prefer Saxon over Xalan 2.3.1 that is distributed in JDK 1.4.

Posted by ovidiu at 01:01 AM |

April 03, 2003

Advanced XSLT usage or using XSLT to generate XSLT

Lake Tahoe from Heavenly

XSLT is a very powerful language. Most people use it to convert XML to HTML: they have an XML document which needs to be translated to a rich HTML document.

The simplest way to do this is to identify how the input XML document maps to the final HTML page, and come up with the XSLT stylesheet that embeds in its templates the HTML elements of the end result. While this works fine for one HTML page, it becomes really tedious if the page needs to change often or if you need to format the input XML multiple times.

A much better approach is to take the resulting HTML page and annotate it with your own special elements. These special elements are invented for your specific problem to delimit portions of the HTML page that need to be replaced with dynamic content from the XML input file.

Once you've annotated the output file, you can write an XSLT program that takes the output page and generates an XSLT stylesheet. The generated XSLT stylesheet will take the input XML document and generate an HTML page with exactly the same structure of the page you want.

There are few things that you need to worry about when writing such an XSLT program. The output you generate is not a generic XML output, it is an XSLT program which needs to follow the XSLT namespace. To generate this namespace you need to use the xsl:namespace-alias element to alias a namespace in the current document to the XSLT namespace:


  <xsl:namespace-alias stylesheet-prefix="gen" result-prefix="xsl"/>

  <xsl:template match="/">
    <!-- Generate the structure of the XSL stylesheet -->
    <gen:stylesheet version="1.0">
      <gen:output method="html"/>

      <!-- put the logic for the generated XSLT here --!>
      <gen:template match="...">

Writing such XSLT programs is not trivial, but you'll appreciate how clean they are. You will not have to embed in your stylesheet any of the output HTML that you need to generate; the generate stylesheet will have all that.

Posted by ovidiu at 09:24 PM |

January 06, 2003

XSLT-process 2.2 for Emacs released

Emacs | Java | XSLT

I'm happy to announce a new release of the XSLT-process mode for Emacs. XSLT-process is an Emacs mode for applying and debugging XSLT stylesheets from within XEmacs or GNU Emacs. You can also use this mode to do processing of DocBook documents edited with Emacs.

Tony Addyman contributed most of the changes in this release. He's using Emacs and XSLT-process to teach XSLT to students in UK!

Here are the main changes since the 2.1 release:

  • XSLT-process now uses Saxon 6.5.2, Xalan 2.4.1 and FOP 0.20.4
  • Debugging using Xalan is now fully supported.
  • Both JDK 1.3 and 1.4 are supported.
  • During debugging sessions the output produced by executing the stylesheet is serialized immediately, then displayed and incrementally updated in an Emacs window.
  • Added parameter passing to the XSLT processor. These are parameters that are specified to all the stylesheets, in addition to any other parameters. They are specified through the customization menu.
  • Logging output, error messages, etc. generated by the FOP processor are displayed in the *Messages* buffer. The level of logging can be customized.
  • Improvements to the display of information during debugging.
  • MacOS X is now a supported platform, in addition to Linux and Windows 2000.
Posted by ovidiu at 10:52 AM |

October 16, 2002

Minor changes in XSLT-based RSS aggregator

Weblogs | XSLT

This evening I've changed the format of the subscriptions file in the XSLT-based RSS aggregator to include both the URL of the site and its RSS feed. I found several Weblogs which don't have RSS feeds.

Since I switched to NetNewsWire for reading the news, I no longer need to aggregate news on the main page. Instead I now generate the list of subscriptions at the right of each page using this XSLT stylesheet. Quite fun!

To solve the problem of the RSS2 variable namespace, I think a good solution is to have the main stylesheet automatically generate another stylesheet which uses the RSS feed's namespace as default namespace. This will avoid using ugly hacks like local-name() and other oddities in the XSLT aggregator stylesheet. More on this later.

Posted by ovidiu at 01:59 AM |

September 27, 2002

XML scripting and XSLT


Ugo Cei comments on my previous comparison of native XML scripting with XSLT:

Maybe the procedural model is more understood, but is this a reason to use an inferior solution? Using goto is easier to understand than structured programming, but Dijkstra showed us that goto is harmful. Structured programming is easier to understand than object oriented programming, but does anyone really advocate throwing objects out of the window? Procedural programming is easier to understand than functional programming, but would you choose a procedural solution over XSLT? I would almost never.

Then James Strachan responds:

XSLT is great for processing an XML document in a declarative way and outputting a new XML document result. If you wanna process the XML in a different way, like work with database, invoke web services, work with your business objects then using XPath with a real programming language (say a procedural OO one) is often a good idea.

I don't think there's one size fits all uses when it comes to processing XML, its quite a broad topic.

Many times I found myself as developer of various frameworks that work with XML in great need to access XML nodes. For that you definitely some ways to access and manipulate the document's data model, and integration between the programming language (Java here) and the XML representation is crucial. JXPath has been very successful in doing just that.

However I don't think a developer of an application should use such low level features. Instead, higher level frameworks should be used instead, since they hide all the low level complexities of the XML data model, and allow for nice ways to process the XML.

SOAP-RPC libraries are a good example of that, there's no need to understand the format of the messages that go back and forth. Cocoon is another example of framework which gives a wide range of options for processing various XML sources, without having to go into the details of manually parsing or explicitly manipulating XML nodes.

Posted by ovidiu at 02:51 AM |

September 24, 2002

RSS 2.0 support in my XSLT news aggregator

Weblogs | XSLT

I've implemented support for RSS 2.0 in my XSLT-based news aggregator. It turns out to support it I only had to copy the RSS 0.92 templates, and add the proper namespace. Really simple! Since I was here, I also added a fairly simple fallback mechanism, so if a new RSS is invented, the aggregator simply displays the information specified in the subscription.

I've placed all the files in a CVS repository, together with my MovableType templates.

Posted by ovidiu at 02:46 AM |

August 06, 2002

Xalan debugging support

Emacs | XSLT

Tony Addyman managed to add debugging support for Xalan 2.4D1 in XSLT-process for Emacs! Now XSLT-process supports both Saxon and Xalan as XSLT debuggers.

Xalan debugging support in XSLT-process has been problematic because of the lack of source location information. During late June 2001, I've modified Xalan to add support for file and line information in both the source XML and in the stylesheet.

For some reasons this code was not working in the 2.2 release, which made the XSLT-process support unusable. Because of the rapid changes in Xalan and the lack of time, I've decided to drop the debugging support for Xalan in XSLT-process. Tony took up the support for this and he's doing a great job at it!

Posted by ovidiu at 05:55 PM |

August 02, 2002

Blogroll using RSS feeds for MovableType

Weblogs | XSLT

I've finally managed to integrate my simple Blogroll engine into the MT system. The sytem is a combination of XML/XSLT, MT regeneration of HTML pages and crontab.

This is essentially composed of two components.

The first component is an XML file which describes what are the RSS feeds you're interested in. In this file you specify the location of the feed, and optionally you can give a simple name to it. This name is used in the generated HTML as the title for the news, in case the RSS feed doesn't specify a name in it (like Slashdot for example).

Here is a simple XML feed document:

  <user>Ovidiu Predescu</user> 
  <title>Ovidiu Predescu's news stories</title> 
    <rss name="Ovidiu Predescu" href="http://www.webweavertech.com/ovidiu/weblog/index.rdf" />
    <rss name="Slashdot" href="http://slashdot.org/slashdot.xml"/> 
    <rss name="James Strachan" href="http://james.weblogger.com/xml/rss.xml"/> 
    <rss name="Sam Ruby" href="http://radio.weblogs.com/0101679/rss.xml"/> 
    <rss name="Matthew Langham" href="http://radio.weblogs.com/0103021/rss.xml"/> 

The XSLT stylesheet then takes this document and translates into HTML. The stylesheet knows how to handle RSS 0.91 and RSS 0.92, which are the formats used by Radio, RSS 1.0 (preferred by Radio), and Backslash, which is used by Slashdot.

I defined the generated HTML file as template in MT, which I called RSSFeedNews. In my new templates for MT 2.21, I include this template in the appropriate place so it can appear in the generated HTML.

I want to automatically retrieve the news, create the HTML fragment to be inserted in each page, and then regenerate all the pages with the fresh news. For this task, I've dug into MT to see how to do it. The documentation was quite clear on what needs to be done, but I encountered some problems. Below is the script which seems to work fine now.

#!/usr/bin/perl -w 
use strict; 
use lib '/path/to/mt/lib'; 
use lib '/path/to/mt/extlib'; 
use MT; 
my $mt = MT->new( Config => '/path/to/mt/mt.cfg', 
                  Directory => '/path/to/mt'); 
my $result = $mt->rebuild( 
                          BlogID => 1, 
                          EntryCallback => sub { print " => ", $_[0]->title, "\n" } 
print "Done.\n"; 

The POD documentation for lib/MT.pm does not mention that the Directory parameter in new is required. I found that after few hours of poking around.

To run the translation you need to have an XSLT processor. I use Saxon, a compliant and very fast XSLT processor written in Java. With Saxon you run the translation like this:

java -jar saxon.jar subscriptions.xml rss2html-standalone.xsl

The rss2html-standalone.xsl XSLT stylesheet will generate a complete HTML, you'll need to customize it for inclusion in another HTML document. This is usually done by simply removing the unnecessary HTML tags from the rssfeed template.

I've put all the necessary code you need to run in this directory.

Posted by ovidiu at 01:18 AM |

July 31, 2002

RSS feed aggregator

Weblogs | XSLT

I've started working on a simple RSS feed aggregator. It is written entirely in XSLT, and so it has a very small footprint.

The idea is to describe the RSS feeds you're interested in using a simple XML file format. The XSLT stylesheet is then used to process this file, obtain the RSS feeds from the remote sites, and translate them to HTML.

This is a pull model, the program needs to be run at regular intervals to pull the RSS feeds and generate the HTML file, which could then be incorporated in the Weblog. A crontab entry should do the job.

I plan to integrate this with MovableType by having the generated file be a template file instead of a plain HTML file, which could then be imported in another template to assemble a larger page. I still need to figure out all the details.

I'll post the XSLT stylesheet and a sample description file as they become more stable.

Once this foundation is in place, a MovableType user interface can be easily added. The UI will simply generate the XML description file, which could be used to generate the rest of the bits. I'll probably need some help doing this portion, as I'm very new to MovableType and I don't think I'll have the time to build this.

Posted by ovidiu at 12:45 AM |
Ccol stuff
  Arduino TinyWebServer: part 3 and part 2

More from me
Picture gallery
Copyright © 2002-2011 Ovidiu Predescu.