Monday, 20th November, 2006

Rich-Media Delivery In Second Life — A HOWTO

secondlife quicktime SMIL howto

There are (amongst other things, of course) two things in relation to technology and code that get me excited: (1) using a set of restricted tools to achieve something previously thought impossible, and (2) getting tools to do things they weren't necessarily designed for.

Remember my "How to use Quark Xpress to make animation" tutorial?

Anyway. There's been a lot of hoo-ha-ing within Second Life educational circles recently (and elsewhere I imagine) about the possibilities of being able to place a web page as a texture on the side of a primitive object. A while back, we were promised "Mozilla on a Prim", but (to quote the Linden Scripting Language wiki)...

Web integration has been axed from 1.7 and won't be ready till further notice.

Shame huh? All those plans of plastering blogs and web pages all over the grid have had to be shelved.

Before I get into my little HOWTO, though, I wanted to briefly mention why this sort of thing might have been good.

With the population of Second Life rapily approaching two million, there's a lot of talk about things like "retention" and "user experience". For those of us who've been involved inworld from (quite) early on, the whole point of it seems obvious — in fact, I would bet that the vast majority of residents that have been inworld for over a year hardly ever think about the "what do I do here?" question that comes up every time you meet a newbie.

But for someone two weeks into SL, someone who's spent their dosh on prim-hair and been to a few laggy clubs, what is there to do?

Ages ago, when the web was just emerging, the point of that didn't seem to obvious either. Sure, I could stick pictures up of my cats (which I did ), and I could read the news and stuff, but for the main the web was a 'playful curiosity'.

These days though, pretty much all of us use it daily. We check our bank balances, we buy shit (lots of shit), we share photographs, we talk to each other ... it's ubiquitous to our lives.

But just right now, Second Life is nowhere near that. Granted, you can go in and chat to people, play around, that sort of thing — but (unless I'm much mistaken) there's no Killer App that you would think "Hmm, I need to do {X}, I'll just fire up Second Life..."

Within an education context, a lot of the 'uses' for SL have revolved around replicating existing teaching methods. So people build lecture theatres so students can experience the pleasure of reading someone type with super-lag. People drop notecard-givers on every corner, pumping out raw text¹ with very little creative possibilities.

Nobody's (AFAIK) really managed to find an application for SL that only SL can achieve.

Now, far be it from me to suggest I've found such a thing — but what I would suggest is that one of the main reasons for this is that Second Life is a very insular place. It doesn't integrate well with the rest of the web. The tools for displaying, sharing, talking (etc) are limited, and despite all the "Oooh! The immersion!" cries of a lot of educators, it's just plain hard to do useful things in there.

MoaP² is, I suspect, seen as an answer to this — if you can display web pages inworld, then you can integrate SL with your Virtual Learning Enviroment of choice, and provide a much better 'blended learning' (urgh) experience for your students.

Except, well, I kinda disagree I'll insist until I'm blue in the face that the best way of displaying a web page, is in a browser. It's what they're designed for

...

Anyway, enough negativity

The key, I think, is finding ways of letting information flow in and out of SL, and presenting that information in ways which are inherent to the medium.

AFAIK (again) there are only very limited ways of sending data between SL and the web — you can use email, you can use llHTTPRequest, you can stream audio, and you can plaster a QuickTime movie on the side of a prim.

Now, you might have seen some of my 'messings around' with using the QuickTime streaming capabilities of Second Life already. Ages ago, I started showing crappy-quality versions of some of my films inworld. But more recently, I've been using it to generate SL-web integration things — using the GD library of PHP to generate dynamic JPEG menus and bringing them inworld through the QT stream.

http://www.youtube.com/watch?v=e8afs9VIGfA

And the other day, you might have spotted me using the same technique to make a rough prototype of an SL version of our course website

But it was while I was Googling like crazy to find better ways of contructing the stream, that I came across an obscure part of QuickTime that I'd never heard of — SMIL.

Synchronized Multimedia Integration Language

SMIL (pronounced Smile ), or "Synchronized Multimedia Integration Language" is very similar in structure to HTML. In fact, a few idiosyncracies aside, it is pretty much HTML for QuickTime. A basic SMIL document looks like this...

<smil>
     <head>
          <layout>
               <!-- layout tags -->
          </layout>
     </head>

     <body>
          <!-- body tags -->
     </body>
</smil>

Basically, you describe the layout of your movie/page in the <layout></layout> part, and you fill that layout with content in the <body></body> part. Super easy

Here, download this example SMIL file to your hard disk, and open it up with QuickTime Player and see what I mean.

Here's one that has two images in it

If you open that second one up in a text editor, hopefully the structure should be easy to follow. First of all, I set the <root-layout /> of the movie, which determines its overall size:

<root-layout id="root" width="640" height="480" background-color="#333333" />

Then I define two regions — one for each image:

<region id="left" width="240" height="180" left="70" top="150" fit="meet" />
<region id="right" width="240" height="180" left="330" top="150" fit="meet" />

The fit="meet", BTW, is to make the image I'm going to put in the region stretch proportionally to fit.

Then, in the <body> section, I create a <par></par> group (so that the images display at the same time and not one after each other, load the images, and set which region they belong in:

<img src="http://static.flickr.com/120/295831047_6882df5ede_m.jpg" region="left" />
<img src="http://static.flickr.com/122/283412757_af372d055e_m.jpg" region="right" />

Simple huh?

Using this technique, I can place images anywhere I want within the canvas of my QuickTime movie (and therefore, on the side of a prim), and what's even better is that I can generate it all using very simple PHP.

Suppose I had an array with some Flickr images in it:

$images = array("299395093",
                "298824122",
                "295831047",
                "291672267"
                ) ;

With a few calls to the Flickr API, I could generate an SMIL document that placed them all on a grid that would display in SL on the side of a prim. In fact, if I did away with the <par></par> bit, I'd have a fully-functioning Flickr/SMIL-driven slideshow app — with a lot less code than the one AngryBeth and I made a while back.

What I've done with it though, is use it to build a composite image that I spread over several prims to create an inworld version of our website:

I'll come back to that in a bit...

QuickTime TeXML

"Yes Kisa. Very Pretty. But what about text?"

Glad you asked

Up until yesterday, I thought the only way to get rich-formatted text onto a prim, was to use the text functions in PHP. I've been jostling with cached JPEG images, imagefttext() — stuff like that.

It's OK, but it always seemed, well, inelegant

But while I was playing around with SMIL, I noticed that as well as <img /> and <video /> regions (yeah, it does video too — I should have mentioned that), you can also do <text /> regions.

Now, I've known about QuickTime text tracks for ages, but it never occurred to me to use them in this context. Partly because I'm a doofus, but also because — well, if you've ever opened a text file in QuickTime Player, you'll know what I mean here — the formatting is a little shite...

It essentially defaults to white, centred text on a black background, scrolling through each paragraph every 2 seconds.

See, I had no idea about QuickTime Text Descriptors

Simply put, Text Descriptors are like tags that you use to mark-up you text so that you can format it. You can set the font, the box-size, the point-size, the color, etc. It seemed to be what I was after.

(When I found it, it sparked off a good ten-minute "I'm a bloody genius" session)

But, well, I can't get it to work It would appear that all I have to do is type things like...

{plain} Plain text. {bold} Bold Text. {plain} Ta dah!

...to get yummy formatting, but every time I tried it would strip out everything up to the last descriptor, so all I'd get with the above would be:

"Ta dah!"

Not what I want, really. I want...

"Plain text. Bold Text. Ta dah!"

It's easy to play around with Text Descriptors — just open up a plain text file in QuickTime Player (Pro) and export it to text with descriptors...

...then dick-around with what it generates If you can work out where I'm going wrong, I'd much appreaciate it

I spent a good half of yesterday trying to work that out, until I spotted a different text option in the Export... dialogue box...

Now, I've only just scratched the surface of what QuickTime TeXML can do, but it appears to be an XML-based way of marking up text to display in a QuickTime movie

Here's an example...

<?xml version="1.0"?>
<?quicktime type="application/x-quicktime-tx3g"?>

<text3GTrack    trackWidth="640.0" 
                trackHeight="320.0" 
                timeScale="1000" 
                language="eng">
    <sample duration="2000" keyframe="true">
        <description    horizontalJustification="Left" 
                        verticalJustification="Top" 
                        backgroundColor="100%, 100%, 100%, 100%" 
                        format="tx3g">
            <defaultTextBox     x="16" 
                                y="16" 
                                width="592" 
                                height="268">
            </defaultTextBox>
            <fontTable>
                <font id="1" name="Arial"></font>
            </fontTable>
            <sharedStyles>
                <style id="1">  {font-table: 1}
                                {font-size: 18}
                                {font-style: normal}
                                {font-weight: normal}
                                {text-decoration: normal}
                                {color: 0%, 0%, 0%, 100%}
                </style>
            </sharedStyles>
        </description>
        <sampleData targetEncoding="utf8">
            <text styleID="1">Some text</text>
        </sampleData>
    </sample>
</text3GTrack>

It looks pretty complicated, but if I break it down it might be easier.

First, you have your usual XML stuff...

<?xml version="1.0"?>
<?quicktime type="application/x-quicktime-tx3g"?>

Then you set the dimensions and stuff of the track within the <text3GTrack> part...

<text3GTrack    trackWidth="640.0" 
                trackHeight="320.0" 
                timeScale="1000" 
                language="eng">

Next, you make a sample and describe certain properties of it — QuickTime will play each sample in turn...

    <sample duration="2000" keyframe="true">
        <description    horizontalJustification="Left" 
                        verticalJustification="Top" 
                        backgroundColor="100%, 100%, 100%, 100%" 
                        format="tx3g">

Within the sample, you first set the default text box...

            <defaultTextBox     x="16" 
                                y="16" 
                                width="592" 
                                height="268">
            </defaultTextBox>

...you load your fonts...

            <fontTable>
                <font id="1" name="Arial"></font>
            </fontTable>

...and you describe the styles of text you're going to use...

            <sharedStyles>
                <style id="1">  {font-table: 1}
                                {font-size: 18}
                                {font-style: normal}
                                {font-weight: normal}
                                {text-decoration: normal}
                                {color: 0%, 0%, 0%, 100%}
                </style>
            </sharedStyles>
        </description>

Next, you write the actual text itself within the <sampleData> part...

        <sampleData targetEncoding="utf8">
            <text styleID="1">Some text</text>
        </sampleData>

...and you close it all off

    </sample>
</text3GTrack>

Like I said, I've only just started playing with this. One thing I did last night, was to use a bit of regex to convert some basic HTML tags to the right format to use with this...

<?php
$text = strip_tags($text, '<b> <i>') ;
$text = wordwrap($text, 85, "\n", 1) ;
$text = str_replace('<b>', '</text><text styleID="bold">', $text) ;
$text = str_replace('</b>', '</text><text styleID="plain">', $text) ;
$text = str_replace('<i>', '</text><text styleID="italic">', $text) ;
$text = str_replace('</i>', '</text><text styleID="plain">', $text) ;
?>

...whilst having created the corresponding styles, natch

Anyway, there you go Fully-formatted text, along with flexible media-rich layouts, all ready to import onto the side of a prim.

Two caveats:

1. It's slow. Bloody slow.

2. You'll notice I have a wordwrap() in the code above. For the life of me I can't work out why the text isn't wrapping to the text box it's contained in. More research neaded...

A Practical Example

The object I've been working on — the SL-version of our course website (http://www.creativepractice.org) is currently hovering in the replica of our University that I built if you want to try it out.

The thing is, you see, I was watching some of the other tutors using the site during tutorials, and it occurred to me that this was a possible use of SL as an educational tool.

Granted, I can browse through all the students' work via the web, but to be able to have a conversation with a student — 'face to face' rather than via comments or emails — well, that's so much better

So basically, the object sends HTTP requests to a PHP script on the website that sends back the ids of the images to display. Then it sets the media URL to another script, with those ids in the request string, which builds the SMIL page and calls a further script to render the text using QuickTime TeXML.

With these scripts in place, the actual information passing between SL and the website is very minimal — I just send it an offset and a user id, and it works it all out for itself.

What's more, I could take the basic framework of the scripts, and just by changing the way I query our database to some Flickr API calls, I could make it into a fully-functioning Flickr browser.

...

Anyway. That's how I'm doing rich-media stuff in Second Life at the moment. Presumably, at some point in the future, other technologies will come along that render all this totally redundant. But, I like playing within the boundaries of what's available and trying to get it to do new stuff

Further Reading

Like I said, I've just started playing with all this. There's probably loads more stuff it can do, and loads more things I haven't found yet.

Have some links

SMIL Scripting for QuickTime — Apple's developer documentation for SMIL.

What's New in QuickTime 6.3 + 3GPP — an indepth documentation (again from Apple) of QuickTime TeXML (amongst other things).

W3C Synchronized Multimedia Home page — w3.org's SMIL page, with a ton more links.

Synchronized Multimedia Integration Language — Wikipedia, that starts talking about MMS as well.

TeXML — Now, I haven't really looked into this one yet, but (and I may be waaaay off the mark, so apologies if I am) I bet it would be possible to use some of this to translate various different formats — like HTML and PDFs — to LaTeX, and then to QuickTime TeXML. But like I said, I'm probably wrong

Have fun

¹ I love raw type. I don't read this weblog very often in its HTMLised form, I just read the raw source files. I'm also a big proponent of the ideas behind Jon Gruber's Markdown formatting. But sometimes, rich text is better.

² "Mozilla on a Prim"

Neil

You'll be wanting a permalink then, I guess

Kevin

Jane

Aw, thanks guys

Miss K

davew

Neil

You can never have too much Helvetica

Quick Quote

Sorry, just something I caught from a trailer at the end of QI that I recorded the other night...

"The Accidental Angler fishes for brown trout tomorrow at 8"

That's surely a euphamism for poo.

In the same vein, tonight at 7pm on Five US:

"Drag Racing"

...not what I thought it was going to be

Becky

Because I've been busy doing other, cooler things — that I'm not really supposed to talk about yet

Actually, shit. I can't believe I missed SL on the telly All this 'only watching stuff I've saved' business means I miss out on the random good stuff.

Zaddy

iconolith

Torley

Intolerable

Miss K

Hmm, someone's got too many MySQL connections open I hope Tom fixes it — things look a little 'default' around here this morning

@Zaddy

Isn't the text on the prim annoying to read?

As long as you use similar design strategies as you would when styling text for a TV screen, no it's not. So that's basically "Don't use small, serifed text"

I wouldn't, for example, try to replicate this weblog on the side of a prim — cos it just wouldn't fit. Or, at least, if I did, I'd find better ways of doing it rather than just squishing it all down to fit.

@iconolith

Thank heaven for your patience writing all that out.

Actually, that didn't take too long. I got on a bit of a 'roll' with it, and the only thing that really got in the way was this

What's funny ironic interesting is that I've got to write a similar thing for work — a "HOWTO start working with Photoshop", and for the life of me I can't summon up the same enthusiam

@Torley

I didn't even know about QuickTime text stylin' before

No, me neither. I can't believe no-one else thought of using it either For ages people have been going on about wanting text on a prim, and (as was pointed out to me in an email by a mutual contact ) this stuff has been available for two years

@Intolerable

Going to try this out with my wife's 'clickable regions'

I'm having to strain myself to not make a joke about that

But I love it That's bloody genius. Yeah, I guess it would be 'sluggish' — but I think we're only at an early stage in terms of what we can get SL to do, and I think it's really important that people play around with stuff like this.

I think sometimes people are scared to try and develop new things — because they're worried about new technologies coming along and superceding the work that they've done. I thikn that a lot of the time, the majority of engagement with a particular technology revolves around "learning what it can do" — rather than "learning what it can do that it wasn't designed for".

And I think that, perhaps, that's the one single thing that I hope will rub off on the students that I teach — that it's more important to keep pushing the boundaries of what's possible, because even if something comes along that makes everything you've done redundant, you'll be in a position to push that in more interesting places.

Or something. I need to write about that specifically one day...

Intolerable

It makes the marked text appear as a link, but the link does not actually work. Does this work for anyone?

'fraid not. There's no way (I know of) yet to tell what part of a prim has been clicked, so you can't pass that to the original script to be translated into clicking a "link"

Which is where your wife's clickable regions come in

Donnagh McDonnagh

    <par>   

        <audio src="themesong.mp3" />       

        <img src="poster.jpg" region="r1" dur="30 sec" />

        <text src="lyrics.txt" region="r2" dur="30 sec" />

    </par>

GreySheepDuo

Yes. That's what I'm using. I replace the "lyrics.txt" with a php script that generates QT TeXML, which is what gives me formatted text.

Junie

Diag

Enki Stardust

You realize, of course, that SMIL isn't specifically a quicktime thing, right?

Of course And the text stuff has been around for a few years — that page on it I link to is for QuickTime 6.