LoCI tools and services

Christopher Sellers and Harold Gonzales

April 21, 2008

Outline

LoDN
LibStdIO
LoDNFS
LEncoder
PyIBP
ScreenCasts

Architectural Stack

  LibSTDIO    LoDNFS

         LoDN

         LoRS

 Exnode LBone  End2End

         IBP

LoDN Purpose

• Exnode Repository
• Exnode Management Services
• Exnode “warming”

LoDN Usefulness

• User Configurable
• Web Interface
• File system-like approach for management
• Exnode Persistence

User Configurable

• Exnode accessible (published/unpublished)
• Distribution of exnode across depots

Web Interface

• LoDN service is provided over http[s] through a web server
• All data and actions are passed to LoDN through http[s].
• This makes it accessible and viewable via a browser.
• Provides a simple interface protocol for writing client code.

File system-like appearance

• It provides heirachical directory structure
• Traverse, create and remove directories
• Exnodes appear as files within each directory
• Each exnode has properties: sizes, accessibility

LoDN URI

• lodn://hostname[:port]/username/path-to-exnode

Exnode Persistence

• Warming
• Imported exnodes fall under the control of the warmer
• Users can control how exnodes are warmed and distributed

Warmer

• Periodically refreshes each allocation
• Replicates data across the user defined "sites"
• One allocation is replicated and maintained per site
• Controls sharing and distribution of data

Warmer Control

• The warmer operates based on user defined configuration files
• Users can specify their own sites for their exnodes
• The warmer operates on exnodes based on the "nearest" configuration file.
• Hierarchical arranged, per exnode and per directory and per account
• Warmer files can be managed through web interface.

<DEMO of LoDN and Warmer file>

LibStdIO

• Based on the stdio interface
• Serves as drop in replacement
• Offers the following functions:

      fopen, fdopen, fileno, ferror, setvbuf, fprintf, fread, putc, fputc, fputs, fwrite, 
      ftell, fseek, fclose, fflush, fgetc, fgets, feof, clearerr, fstat, ftruncate 

• The only change needed is to replace the stdio.h header file with libStdIO.h and to recompile with LoRS libraries.
• It does preprocessor replacement, to replace all of the above functions with their std_ equivalents in libStdIO.

• It works with the LoDN URI and is compatible with regular stdio.

• When a file is opened, libStdIO determines the type based on the URI scheme. For every file descriptor, there is an internal data structure of function pointers for the io functions to use. For standard files, the regular io functions are assigned. For LoDN URI, the LoRS (lors_*) functions are assigned in the data structure. When the std_* functions are called, this data structure is used to call the correction io function.

   Ex: Application --> std_fread() --> lors_fread() or fread() 

• When a LoDN URI is opened, libStdIO uses the http[s] protocol to export the exnode from LoDN and to the local file system for use.
• When a LoDN URI is closed, libStdIO uses the http[s] protocol to import the exnode to LoDN. - The logistical networking based io functions are implemented with the LoRS API with local buffering.

• In addition to the LoDN URI, environmental variables are used to pass information to libStdIO including username (LODN_USER), password (LODN_PASSWORD), ssl method (LODN_SSL_METHOD), and depots (LORS_DEPOTS).

• Example Applications:

    HDF5, grace, gnuplot and more

<LibStDIO Demo>

LoDNFS over Fuse

Purpose

• Many application expect lower Posix IO and filesystem interface if StdIO can be used.
• Every application has to be individual ported.

Fuse

• Filesystem in userspace or FUSE allows non-privileged users to create their own file systems without the need to write any kernel code.
• This is achieved by running the file system code in user space, while the FUSE module only provides a "bridge" to the actual kernel interfaces.
• In this case the FUSE interface handles the acquisition of the exnode and the collection of data from the depots, leaving the user free to use the mounted directory as they would a normal filesystem.
• The use of FUSE eliminates the need for applications to be ported, but also enforces filesystem semantics and necessitates the mounting of the desired lodn directories.
• A major advantage of fuse is that it allows us to not only use a command line interface but allows for the full range of applications of any mounted partition.

Implementation

• Fuse provides a set of functions to implement.
• The kernel performs filesystem requests through a pseudo device and fuse module that calls these functions.
• These functions work similar to the lors_* functions in libStdIO in that they use http[s] to communicate with LoDN and the LoRS API for operating on the exnode.

• Though still in development this tool allows for greater ease of use, a wider variety of applications, and a new level of convenience for the LoDN user community.

< LoDNFS Demo >

LEncoder: a logistical computing video transcoding service

• The NFU provides a mechanism for computation within the logisitcal network
• There exists a potential for computationally complex operations to be distributed and executed via many NFUs
• An example of such an operation is video transcoding ( the transformation of a video file by manipulating bitrate, codec, etc)
• LEncoder (based on Mplayer's MEncoder program) will introduce a facility for video transcoding within the logistical network
• encoding operations will read data from and write output to ibp capabilities
• A client program written using PyIBP will segment the data in the original file, distribute it across depots, invoke LEncoder on those segments, download the finished pieces, and coalesce the output file
• Current used planned for the transcoding of hdtv for the new ibpvoHD service

PyIBP

• Provides bindings to the C IBP client API in Python.
• In development
• Rapid prototyping for new tools and applications.
  • New ideas can be rapidly tried and tested in a high level language.
  • Adjustments and changes can be made quickly in Python before writing the production code in lower and more efficient languages like C.
• Python is heavily used by the scientific community for applications and gluing components together. This allows them to tie in another component (IBP) without rewriting their programs in C.
• For example, the client program for LEncoder is planned to be implemented with PyIBP.

Screen casts

• Provides a series of introductions to the various components of Logistical Networking.
• Targeted at different audiences (Users and Developers).
• The goal is to provide a template that explains to new users fundamentals of LN

• 3 main topics:

1. LN intro (10 min) - Everyone
2. LoDN (10 min) - Non tech users
3. LibStdIO/Fuse - Application Developers

• In development due in Mid Summer