Xgrid notes
This is just a place for notes and links on xgrid.
Info on the formatting of
Batch Files.
glossary
- agent : a deamon process running on a Mac that performs a calculation
- controller: a deamon process running on a Mac that hands pieces of a job to agents
- client : an application with a job that connects to controller
recipe
1)
Configure xgrid agent in
System Preferences > Sharing > Xgrid
Use a specific controller: thirty (computer URL or IP address)
Agent accepts tasks: always
Authentication method: password ***** (pick something)
This information is stored in
/etc/xgrid/agent/com.apple.xgrid.agent.plist.default
and
/etc/xgrid/agent/controller-password
Configure passwords for the xgrid controller by
copying that controller-password to /etc/xgrid/controller/
$ cd /etc/xgrid/
$ sudo cp agent/controller-password controller/agent-password
$ sudo cp agent/controller-password controller/client-password
and edit /etc/xgrid/controller/com.apple.xgrid.controller.plist.default
so that it contains
<key>AgentAuthentication</key>
<string>Password</string>
<key>ClientAuthentication</key>
<string>Password</string>
2) Click 'Start' to turn on the agent,
in Preferences / Sharing / Xgrid
3) Repeat for agents on as many Macs as you like
4) Turn on a controller, e.g. 'sudo xgridctl controller start'
5) Launch 'Xgrid Admin' (free download from apple; see below),
connect to a controller with the password,
and you should be able to see what agents it has.
6) For the rest, the command line client is 'xgrid'
which can query the controller and submit jobs.
Below are examples from Drew McCormack's
MacDevCenter Aug 2005 article; there's a link to it below.
# define what 'xg' means for all the following commands.
#
# Note that the PhyLab controller has IP address 10.1.4.133,
# and that the -h flag takes IP addresses. Of course, you'll
# to know the real password.
#
$ alias xg='xgrid -h thirty.local -auth Password -p ****'
$ xg -grid list
{gridList = (0); }
$ xg -grid attributes -gid 0
{gridAttributes = {gridMegahertz = 0; isDefault = YES; name = Xgrid; }; }
# Run a synchronous job
$ xg -job run /bin/hostname
mdhcp-19.marlboro.edu
# Run an asynchronous job
$ xg -job submit /bin/hostname
{jobIdentifier = 1;}
# Note that this job shows up in the 'Xgrid Admin' GUI
$ xg -job attributes -id 1
{
jobAttributes = {
activeCPUPower = 0;
applicationIdentifier = "com.apple.xgrid.cli";
dateNow = 2006-11-28 17:26:44 -0500;
dateStarted = 2006-11-28 17:25:48 -0500;
dateStopped = 2006-11-28 17:25:49 -0500;
dateSubmitted = 2006-11-28 17:25:47 -0500;
jobStatus = Finished;
name = "/bin/hostname";
percentDone = 100;
taskCount = 1;
undoneTaskCount = 0;
};
}
$ xg -job results -id 1
mdhcp5-19.marlboro.edu
You can also submit 'batch' jobs with more than one task;
then the controller automatically parses them out across
the available agents.
The article at http://www.macdevcenter.com/lpt/a/6159
describes how to set up a batch file that describes which
executables with which arguments make up each task.
There isn't, however, anything in the article that talks about
a way to 'automatically' divide up something into a parallel
job ... though if the job can be broken up into pieces, then
a script to do so might work something like this, assuming
you have an application which can do any piece of the task,
with command line arguments to specify which piece.
b) have xgrid ask the controller for the number of agents, n
c) create a batch control file that invokes the application
n times, with the right arguments to split the job up that way
d) use xgrid to submit the job,
e) periodically check to see its it done,
f) when finished, grab all the pieces (results), patch 'em together,
and deliver the output.
Here's a quote from http://unu.novajo.ca/simple/archives/000024.html
that confirms this:
"There is no magic with Xgrid: if you want every agent to do a slightly
different task than the other agents, then you need to provide a list of
slightly different arguments. What they are and how you generate them
depends on the problem at hand."
additions to the recipe
Submitting a batch job is similar to a 'submit' job, but the syntax is slightly different:
xg -job batch <batchFile>
where batchFile is an XML or plist file containing the specifications for the job. Check out the
Batch Files section for more info about the specifications file.
For batch jobs and submit jobs (I don't know if this works for run jobs), the results as printed to STDOUT can be gotten via the command
xg -job results -id xx
This will print those results to STDOUT on the calling computer. If, however, the job produced output files, the command
xg -job results -id xx -out ./results/
will drop all the output files from all the tasks into the folder ./results, creating it if necessary. Make sure to have each task name its output differently, I have no idea how it'd handle it otherwise.
It is possible to have xgrid direct STDOUT and STDERR to files, but I haven't figured that out yet.
mac application development
There are some skeleton Mac Xcode development projects in
the mac developer Xcode stuff. See
/Developer/Examples/Xgrid , including 'GridMandelbrot' and 'GridSample',
which includes the suggestively named 'MPI' (which is an acronym for 'message passing interface', I'd guess, a standard API for parallel applications).
resources
