PostgreSQL conversion continues

I’ve been writing the conversion script to pull the arachadmin database out of MySQL and into PostgreSQL.  Along the way I’ve fixed some problems I hadn’t noticed, so there will be at least one final update of the MySQL export on figshare.  I will commit the postgreSQL versions as a separate figshare document.  At the very least this will avoid confusion – one document if you want the legacy mysql and a different document for the maintained postgresql.  Getting this to work will probably close a number of the tickets in the arachadmin tracker.  Note that the associated wiki has a more-or-less complete coverage of the tables as of last August.  Expect an update once the database conversion has settled out.

Another advantage of moving to pyramid is the hope that the arachadmin editor will be in a state to distribute as a package – set up to run with Python 3.5.  Install postgres, the package, and the database from figshare and anyone should be able to browse a clone of the backend database.  A small step from ‘open source’ towards reproduciblity.

I plan call the new version of the editor app ‘arachcurator’.  I was planning to make the initial github commit this week, but since pyramid is promising that version 1.6 will come out of beta in a week or so, I’ll wait for that and build from that.  Changing the name is both more accurate (it does very little administration, but lots of curation) and means it won’t share a name with the database itself.   Perhaps I’ll change the name of the database when I commit it to figureshare (perhaps ‘backend’ term from spider anatomy).



Updating Sesame, starting the move to pyramid/postgresql

All technical stuff this week.  I’ve got the Spider-Behavior server code connected to travis-ci.  In the process of doing so, I ran into problem with Sesame, which turned out to be related to the maven setup.   It’s working and updated to the most recent 2.8 release.  This will allow some more changes, particularly some more formatting work on the narratives page.

While waiting for a response from the Sesame-users group, I started taking a serious look at moving arachadmin from web2py.  It looks like I’ll be able to move to a python3.5-pyramid-postgresql stack.  I have a first pass at the new table structure working with sqlalchemy, and will soon start the process of coding the transfer from mysql.  The experience is a lot closer to standard python development, rather than the somewhat cushioned ride offered by web2py.

Meanwhile, spending some time reviewing the opentree taxonomy against the world spider catalog.  So far nothing horrible, mostly groups added since 2009 and some unstable family splits.

No big news for Arachtober

Spider in leaf refuge

Unidentified spider constructing rolled leaf refuge.

Things weren’t really cleaned up enough for a big release announcement in October, though almost every page has had an update in the last month.  In addition to narrative ‘drill-down’ pages, all the other identifiers that correspond to something in the database now resolve to simple pages.  The narrative drill-down pages (such as this) generate a table of their component events.  They can generate both html and json, the later is (in many cases) simply the ‘tunneled’ json that Owlbuilder serializes into an rdfs:comment annotation (as discussed here). These still need some tweaking and there is still information for ordering the events in the narrative.  This, table pagination, and some javascript cleanup remain before I could consider a release.  Then it will be time for some more curation and thoughts of updating the curation tool (python 3.5 and pyramid and mysql -> postgres).

Owlbuilder and server backend code are gradually migrating to Java 1.8.  Meanwhile Oracle’s cloud-related noises are making me nervous about the future of Java on non-Oracle environments.   Events conspire to make me nervous about Java every couple of years, so maybe it’s not time to worry yet.