Bundle development

This page describes bundle development process at Science Commons and is written mainly for the audience consisting of Alan R and Dan.

This is project internal documentation and only applies to the Science Commons bundle development setup.

Basics

See Package Makefile conventions for general information on setting up. In brief:

Do this on ashby or norbert
Let R = your subversion checkout (trunk)
Create a file in R/packages/default.mk per instructions (or just copy the one in ashby:~jar/nc/trunk/packages/, changing 'jar' to your user id and changing 'AUTHORITY_URI' per the instructions)
You don't want to create your own Medline cache - use mine instead. So set MEDLINE_CACHE to be /raid/jar/cache/medline

To update or modify an existing bundle B:

cd R/packages/B
make local changes to the code in the package if desired
Then:

make prepare
make cache
make -j 3 bundle

(The flag -j 3 means to use 3 CPUs.)

Then, after eyeballing the resulting and performing whatever unit tests you can (codify if possible using the 'check' target), put the new bundle version where RDFherd can see it (i.e. /raid/not-backed-up/development

make export

Then, deposit the bundle into the triple store. You have to do this on norbert:

sudo -u virtuoso rdfherd /wd/development bundle_update B

To rerun transitive closure you would need to do this:

rdfherd /wd/development bundle_update inferred-relations

or perhaps something a bit more systematic (TBD). (You may have to update the version number of that bundle... not sure what the procedure should be. Work in progress.)

Making a new bundle

When creating a new bundle (at Science Commons), you'll want to make a new subdirectory of trunk/packages/ (using 'svn mkdir'), with a Makefile, perhaps modeled on one of the other package Makefiles. Put any needed scripts in this directory. Do 'svn add' to put the Makefile and scripts into the svn repository.

'make prepare' should create the bundle directory and, if needed, the cache and work directories. 'make prepare' should also download any auxiliary software such as Java or python modules (work in progress: figure out where they should go), and create bundle file lists if these need to be computed, e.g. by listing a directory on an ftp site, loading a metadata file, or running a query. File lists can then be 'included' in the Makefile to guide the 'make' process. (this info belongs in 'makefile conventions')

Checkin

Create bundle directories initially with 'svn mkdir bundlename'.

Do 'svn add' to put the Makefile and scripts into the svn repository.

After you've done a satisfactory 'make' (= 'make bundle') there will be a new file Config-template.pl, which should also be checked in. This can be customized later if the bundle has any special needs such as running a script.

After exporting the bundle ('make export' or 'make snapshot') there will be an additional directory 'reference' that captures the association between the current bundle content and the version number assigned to that content. This should be kept in the svn repository so that if someone else re-creates the same content the same version number will be assigned. So 'svn add reference'.

Doing a new 'grep' of Medline abstracts

Choose a mnemonic name for your search, say 'hyla'

cd R/packages/medline
mkdir hyla
cd hyla

Create a file called 'hyla-pattern.txt' containing your regular expression. It must NOT contain a closing newline.

Create a Makefile in directory 'hyla' containing two lines:

RELATION=sc:uses_hyla_phrase
include ../alleles/Makefile

where in place of sc:uses_hyla_phrase, give a QName for the relation that relates the journal_article to the string that matches the regular expression.

The package should now be ready. Proceed as above, with the bundle name B being medline/hyla.

If you think this is a good 'grep' that others might want to include in their triple stores, you can check the directory and files into subversion. [Also add the bundle name to some master list somewhere... figure this out.]