News from Sesame

Sesame is the triple store I’m currently using to back querying on the arachnolingua web page.  I just saw the message that Sesame has been adopted by the Eclipse Foundation as RDF4J.  This is probably a good thing, but it’s wait and see for now.   Nothing will break anytime soon, I’m still using the previous version (2.8).  Meanwhile fixing table permissions and starting on actual editing in arachcurator.

 

Advertisements

Exploring Alternatives

One of the nice parts about the switch to pyramid is that it supports a range of alternatives.  For example, at the moment I am trying out forms in WTForms after converting the templates to jinja2.  In some ways the arachcurator tool has become a hybrid of technologies from pylons and from flask.  I guess this means if pylons disappears tomorrow, I won’t be starting from scratch.  It was a gut-level instinct call that lead me to prefer pyramid over flask in the first place.

Meanwhile, all the browsing code is built, tested, and now back in a state of flux as I shift forms technology.  There are 3djs graphs for both individuals and their part as well as expanded display trees on the claim page.  I have added users (mostly for change tracking) and three levels of authorization with the hope to put an arachcurator server up on AWS at some point (sharing more than code and demonstrating replicability I hope).  I even went so far as to add password encoding with passlib.

Meanwhile, as discussed here, I’ve started building a T-box vocabulary for spider behavior.  This will extend what can be said in annotations, in a way that changes here will only simplify.

 

 

Progress with Pyramid

I’ve been pushing the conversion of arachadmin over to Pyramid pretty hard the past few weeks.   I started with the SQLAlchemy tutorial and then started with the project template, tried setting up the database (in Postgresql) in both Core and ORMS and chose the later (mostly for the more complete transaction support).  I rewrote the initdb script in the template to load and copy every (used) table in the current MySQL database and stuff it in the postgreSQL database defined by the schema.  Lots of problems with ordering of transactions due to primary keys that were also foreign keys points to a table of identifiers (goodbye mandatory integer indexes).  Just of note – I used a master table of URIs that were used as primary keys for records that persist objects (pubs, terms, properties, claims, individuals, taxa) as a straightforward way to guarantee uniqueness.  This is an issue because most of these classes of objects can have generated ids and this is an easy way to backup the id generation system.

Once I had the database, I picked up enough of Chameleon and Deform to implement (very) basic listing and browsing pages for publications, taxa, and claim (still in progress).  I haven’t done enough with Deform past the default display to have much to say, but the Chameleon templating system is working well – the basic templating and the macro system just work and I haven’t had any nasty surprises in each.  As long as I remember what belongs in the template and what needs to get passed in from the view function, everything is fine.

I’ve been trying to figure out why I feel more comfortable tinkering with the guts in this code than I was with web2py.  I think the difference is that a lot more is exposed in the process of getting started, especially moving beyond the basic ‘hello world’ app.  Web2py hides a lot more and it seems less transparent about the documentation examples are doing.  For example, specifying widths of strings in database columns vs. forms: because the packages for database and forms are separate in pyramid (and there are options) it is clearer what specifying a width in a particular place means.  In web2py it wasn’t clear whether a string length was specifying a database column width, a form display width, or a form validation constraint.  In summary, there is a lot of functionality packed into web2py, but I had trouble building a mental model of what was going on with a lot of the functionality (especially views).

The lack of serious support for unit-testing (beyond doc tests) didn’t encourage experimentation either.  I’m also happy that pylint, despite some complaints, seems a lot happier with the pyramid code I’ve feed it than when I tried web2py code.  With web2py I had to limit myself mostly to pep8 formatting if I didn’t want to drown in lists of problems due to the mismatch between pylint and web2py.  There is a pylint init file that claims to work with web2py code but I never saw any effect of including it (it said it loaded, then reported the same problems).

A new entry in the world of python code-checkers (or linters if you prefer) is the online tool from QuantifiedCode.  I haven’t tried this tool (or their new code ‘corrector’) on the new arachadmin code because I’m waiting for Pylons to officially release Pyramid 1.6 before I commit everything to github (just simpler that way).  The beta has been out for two weeks and no problems reported (I am actually using an older, alpha release of 1.6, no problems to report).

Enough rambling.  Wayne Maddison published a taxonomic revision of the Salticidae last month and at the same time released a large collection of his specimen photos.  So, since I have nothing new, have a northern Phidippus (not audex, which I didn’t find any pictures of).

 

na13-6898

 

 

PostgreSQL conversion continues

I’ve been writing the conversion script to pull the arachadmin database out of MySQL and into PostgreSQL.  Along the way I’ve fixed some problems I hadn’t noticed, so there will be at least one final update of the MySQL export on figshare.  I will commit the postgreSQL versions as a separate figshare document.  At the very least this will avoid confusion – one document if you want the legacy mysql and a different document for the maintained postgresql.  Getting this to work will probably close a number of the tickets in the arachadmin tracker.  Note that the associated wiki has a more-or-less complete coverage of the tables as of last August.  Expect an update once the database conversion has settled out.

Another advantage of moving to pyramid is the hope that the arachadmin editor will be in a state to distribute as a package – set up to run with Python 3.5.  Install postgres, the package, and the database from figshare and anyone should be able to browse a clone of the backend database.  A small step from ‘open source’ towards reproduciblity.

I plan call the new version of the editor app ‘arachcurator’.  I was planning to make the initial github commit this week, but since pyramid is promising that version 1.6 will come out of beta in a week or so, I’ll wait for that and build from that.  Changing the name is both more accurate (it does very little administration, but lots of curation) and means it won’t share a name with the database itself.   Perhaps I’ll change the name of the database when I commit it to figureshare (perhaps ‘backend’ term from spider anatomy).

 

Updating Sesame, starting the move to pyramid/postgresql

All technical stuff this week.  I’ve got the Spider-Behavior server code connected to travis-ci.  In the process of doing so, I ran into problem with Sesame, which turned out to be related to the maven setup.   It’s working and updated to the most recent 2.8 release.  This will allow some more changes, particularly some more formatting work on the narratives page.

While waiting for a response from the Sesame-users group, I started taking a serious look at moving arachadmin from web2py.  It looks like I’ll be able to move to a python3.5-pyramid-postgresql stack.  I have a first pass at the new table structure working with sqlalchemy, and will soon start the process of coding the transfer from mysql.  The experience is a lot closer to standard python development, rather than the somewhat cushioned ride offered by web2py.

Meanwhile, spending some time reviewing the opentree taxonomy against the world spider catalog.  So far nothing horrible, mostly groups added since 2009 and some unstable family splits.

No big news for Arachtober

Spider in leaf refuge

Unidentified spider constructing rolled leaf refuge.

Things weren’t really cleaned up enough for a big release announcement in October, though almost every page has had an update in the last month.  In addition to narrative ‘drill-down’ pages, all the other arachb.org/arachb/ARACHB_xxxxxxx identifiers that correspond to something in the database now resolve to simple pages.  The narrative drill-down pages (such as this) generate a table of their component events.  They can generate both html and json, the later is (in many cases) simply the ‘tunneled’ json that Owlbuilder serializes into an rdfs:comment annotation (as discussed here). These still need some tweaking and there is still information for ordering the events in the narrative.  This, table pagination, and some javascript cleanup remain before I could consider a release.  Then it will be time for some more curation and thoughts of updating the curation tool (python 3.5 and pyramid and mysql -> postgres).

Owlbuilder and server backend code are gradually migrating to Java 1.8.  Meanwhile Oracle’s cloud-related noises are making me nervous about the future of Java on non-Oracle environments.   Events conspire to make me nervous about Java every couple of years, so maybe it’s not time to worry yet.

Major changes, more to come

There are some major changes up at Arachnolingua.  Queries for behavior events that occur as part of narratives is working, though not the way I think, upon reflection, they should, so expect some more changes (hint: narratives have ARACHB uri’s and will be getting their own pages).  There are also 404 and server error pages that show more than the generic messages from Apache Tomcat.  I’ve also upgraded the version of Bootstrap, the javascript/css library that defines a lot of the look and feel.  The font and other changes are a small but noticeable improvement and if you look in the upper left, you can actually make out the spider on the dictionary.  Finally, Arachnolingua has a server upgrade (I’ve moved up to an aws t2-small instance and put the t1-micro out to pasture).

Lots more to change, and then a wider announcement before the end of Arachtober.