Building an Xdrive Data Management Toolkit

By Abel Lin
September 26, 2007

Content (or data) management systems are often associated with the Web. Tools such as wikis and blogs manage and publish data to Web sites. Content and data management, however, occurs at all levels of the software stack—any place where data must be stored, managed, and shared.

At SundayMorningRides.com, for example, we have to manage not only the current active Web (text, images, and videos) and GPS data; we also manage daily archives and backups, in addition to multiple development versions of our Web site and supporting software. In the case of development software, the data also must be managed so that multiple people can actively collaborate and participate.

Data management is acutely challenging for small and medium sized organizations that might not have the necessary local infrastructure (or in the case of backups and archives, the necessary off-site infrastructure) to provide an adequate data management system. Fortunately, tools like the AOL OpenXdrive (Xdrive) API exist to help you manage your data. More importantly, you can access Xdrive from the Xdrive Data Service Platform (XDSP), which lets developers access the storage, services, and functions of the XDSP programmatically through a set of Web services.

In this article I’ll provide an introduction to building Perl-based programs that access the Xdrive Web service API. This article will also serve as a nice introduction to developing Web service clients using Perl and the LWP (Library for WWW in Perl) library. These tools make up a toolkit that we use regularly at SundayMorningRides.com to interact with Xdrive to share, archive, and otherwise manage our data.

A Word About Web Services

Before we launch into the examples, I want to spend a bit of time covering Web services. These days the subject of Web services can invoke passionate debate between software architects. At the most basic level a Web service is a software system designed to provide a means of interaction between two resources over a network. Web Services are frequently simple Web-based APIs that are accessible over the Internet and are executed on a remote server. Much of the debate is over implementation.

Specifically, there are three common methods for implementing Web services: remote procedure call (RPC), service-orientated architecture (SOA), and representational state transfer (REST, or RESTful). Each has advantages and disadvantages. There are also hybrid models: RESTful SOA, POX/HTTP (Plain old XML over HTTP), and so on.

XDSP is a RESTful Web service that uses JSON (JavaScript Object Notation) as its data format. I have found that a useful way to think of RESTful Web services is the CRUD envelope, in which the four basic HTTP methods can be applied to a URL: GET, POST, PUT, and DELETE. The methods are detailed in the following table.

HTTP Method	CRUD Action	Description
POST	Create	Creates a new resource
GET	Retrieve	Retrieves the representation of a resource
PUT	Update	Updates a resource
DELETE	Delete	Deletes a resource

An Xdrive URL, for example, has the following format:


http://plus.xdrive.com/json/version/method;jsessionid=<sessionid>?data=<JSONencoded request>

/json/ is the servlet gateway; all JSON messages must go through the gateway to be processed
version is the version of the interface you want to use
method is the method name you want to call
jsessionid is the authenticated session ID
data is the JSON-encoded request (optional for some methods)

Software Prerequisites

In this tutorial you will create a toolkit of Perl-based programs to interact with Xdrive from the command line. The following examples assume that you have installed the following Perl modules on your system. You can find and download all of them, free, at CPAN (www.cpan.org).

LWP	The Library for WWW in Perl, used to issue POST and GET requests
JSON	Parses and converts JSON to and from Perl data structures
Data Dumper	“String-ifies” Perl data structures (suitable for both printing and evaluation); very useful for debugging

You will use the LWP library to perform GET and POST requests. Specifically, I want to demonstrate here how to use a “Web user agent” (LWP::UserAgent) to dispatch Web requests. As a side note, LWP::Simple is also a fantastic interface to the LWP library. If you are a fan of Perl and REST, you should take the time to become familiar with LWP.

You will use the JSON library to convert JSON to a Perl data structure. Data Dumper is a very useful library for viewing Perl data structures. This is a great tool for debugging or for internal developing, where the output doesn’t have to “look pretty.”

Authenticating to Xdrive

Currently, the JSESSION cookie that is created when you log in to your Xdrive account manages authentication to Xdrive. You will have to find the cookie from your browser and attach it to every URL. To avoid changing the URL in every program that makes up our toolkit, you can simply copy the cookie to a file that will be read by every program. In the following examples the cookie is placed in the “.jsessionid” file.

A Simple Example (a File Listing)

Following is a simple example that uses the file.getlisting method that returns the contents of a specific folder. If no parameters are given, as in this case, it automatically lists the contents in the root (“My Xdrive”) folder.

#!/usr/bin/perl

use LWP::UserAgent;
use JSON;
use Data::Dumper;

open(FILE, ".jsessionid"); @raw=<FILE>; close(FILE);
$JSESSIONID = $raw[0];

my $ua = LWP::UserAgent->new;
my $json = JSON->new(pretty => 1, delimiter => 0);

my $url = "http://plus.xdrive.com/json/v1.1/file.getlisting;jsessionid=$JSESSIONID?data=$json_req";

my $response = $ua->post($url);
$content = $response->content;

$obj = $json->jsonToObj($content);
print Dumper $obj;

This will return quite a long output, because Xdrive by default has several folders—for example, “My Music” and “My Documents.” For brevity, following is the output of one of those folders. You can examine the parameters that you will need for subsequent examples.

...
{
  'accessPerm' => bless( {
                           'value' => undef
                         }, 'JSON::NotString' ),
  'isMapped' => bless( {
                         'value' => 'false'
                       }, 'JSON::NotString' ),
  'size' => 0,
  'createTime' => '1184957838000',
  'filename' => 'My Videos',
  'parentId' => 'xdr:XFS-0123456789',
  'aliasTo' => 'xdr:null',
  'isShared' => bless( {
                         'value' => 'false'
                       }, 'JSON::NotString' ),
  'id' => 'xdr:XFS-9876543210',
  'type' => 'FileObject',
  'isDir' => bless( {
                      'value' => 'true'
                    }, 'JSON::NotString' ),
  'modifiedTime' => '1184957838000',
  'maxQuota' => -1
},
...

The parameter that you will want to notice is id, because that is how you will programmatically refer to any given folder. Also note parentId. This is the ID of the parent folder—in this case your home folder. Some Xdrive methods can be accessed using either POST or GET methods. Which method you choose depends on the design of your application, and your personal preference. To change the code from POST to GET, just make the following switch to the user agent.

my $response = $ua->get($url);

Creating a Folder

Next, you will create a folder to which you can upload and download files. You will also see how to create a JSON-encoded message to pass parameters to Xdrive methods. The method for creating a folder is called file.newfolder and requires two parameters :

destFolder: the ID of the folder in which to make the new subfolder
filename: the name of the folder

Both of these are the type FileObject. First you will encode the parameters into a Perl data structure ($req), and then you convert it to a JSON notation using objToJson. Then, send it to the server using the URL request parameter data.

#!/usr/bin/perl

use LWP::UserAgent;
use JSON;
use Data::Dumper;

open(FILE, ".jsessionid"); @raw=<FILE>; close(FILE);
$JSESSIONID = $raw[0];

my $json = JSON->new(pretty => 1, delimiter => 0);
my $ua = LWP::UserAgent->new;

$req = {
  "destFolder" =>
      { "type" => "FileObject",
      "id" =>  "xdr:XFS-9876543210" },
  "filename" =>
    { "type" => "FileObject",
      "filename" =>  "Tutorial" }
};
$json_req = $json->objToJson($req);

my $url = "http://plus.xdrive.com/json/v1.1/file.newfolder;jsessionid=$JSESSIONID?data=$json_req";
my $response = $ua->post($url);
$content = $response->content;

$obj = $json->jsonToObj($content);
print Dumper $obj;

As in the previous example, the Xdrive responses are JSON-encoded. Convert the JSON back to a Perl data structure using jsonToObj, and use Data Dumper as a convenient way to examine the results.

Uploading a File

Now you will upload a file into this directory using the method io.upload. This method has one required parameter and one optional parameter:

destFolder: the ID of the folder to upload the file
StatusToken (of data type StatusObject): a token to use with the method io.querystatus to check progress (optional)

In addition to these parameters, you have to pass the local file to the server. The file is sent in a multi-part message, just like a standard browser multi-part form post.


#!/usr/bin/perl

use LWP::UserAgent;
use JSON;
use Data::Dumper;

open(FILE, ".jsessionid"); @raw=<FILE>; close(FILE);
$JSESSIONID = $raw[0];

my $json = JSON->new(pretty => 1, delimiter => 0);
my $ua = LWP::UserAgent->new;

$req = {
  "destFolder" => {
      "type" => "FileObject",
      "id" => "xdr:XFS-1111111111" },
};
$json_req = $json->objToJson($req);

my $url = "http://plus.xdrive.com/json/v1.1/io.upload;jsessionid=$JSESSIONID?data=$json_req";
my $response = $ua->post($url,
       'Content_Type' => "multipart/form-data",
       'Content' => ['file' => ["/path/to/your/file"]],
);
$content = $response->content;

$obj = $json->jsonToObj($content);
print Dumper $obj;

Downloading a File

You probably are starting to get a handle on the process now. Like downloading, uploading is comprised of one required parameter and one optional parameter:

toDownload: the ID of the file to download
StatusToken (of data type StatusObject): a token to use with the method io.querystatus to check progress (optional)

StatusToken is a handy tool if your file is large. It lets you check the progress of the download.

#!/usr/bin/perl

use LWP::UserAgent;
use JSON;
use Data::Dumper;

open(FILE, ".jsessionid"); @raw=<FILE>; close(FILE);
$JSESSIONID = $raw[0];

my $json = JSON->new(pretty => 1, delimiter => 0);
my $ua = LWP::UserAgent->new;

$req = {
  "toDownload" => {
      "type" => "FileObject",
      "id" =>  "xdr:XFS-112233445” },
};
$json_req = $json->objToJson($req);

my $url = "http://plus.xdrive.com/json/v1.1/io.download;jsessionid=$JSESSIONID";
my $response = $ua->post($url,
   ['data' => "$json_req"]
);
$content = $response->content;

open(DAT,">/path/to/your/file");
print DAT $content;
close(DAT);

As you can see, in the case of downloading a file, there is no JSON response. You simply receive your data bytes, which you can write to a file. Also, note the alternative way to pass the data URL parameter. Which syntax you go with is personal preference.

Summary

Xdrive is a great resource for data management. In this tutorial you’ve started to develop a set of command-line tools to interact with Xdrive. At SundayMorningRides.com we’ve developed a set of Xdrive programs (we call them “Xcommands”) that are similar in syntax to their UNIX counterparts. For example, we can create “Xupload” from the upload example by simply adding command-line arguments for the local file and the target Xdrive folder.

% ./Xupload localfile Xdrive_folder_id

From the basic examples shown here, you might think that Xdrive is simply a data archival system. While it is indeed useful for data archival, Xdrive is also quite useful for data collaboration and sharing. Moreover, using the Xdrive Data Service Platform (XDSP) you can not only create tools and utilities, but you can also integrate Xdrive directly into your application. For example, with the io.view function, you can directly dump the contents of a file to a Web browser. I will cover these and other functions in future articles.

Topic Centers

Submit Post

More Articles