What’s changed

Well, I got the website back up in under a week.  I had to build from scratch since I wasn’t able to salvage anything from the AWS instance.  Of course the data and code are all sitting on my laptop, so it might have been just configuration and reloading.  However, time marches on, and since I was installing from scratch, I took the opportunity to update OS (Ubuntu 16.04), Java (1.8), Apache (2.4), Tomcat (minor update) and to upgrade from Sesame 2.8 to rdf4j, its replacement.  One thing that didn’t get upgraded was CORS-filter.  Although it was available, I discovered that Tomcat had equivalent functionality available, just waiting to be configured.  CORS-filter has served me well and it seems to still be maintained.

Once I had everything configured, it seems that queries are somewhat faster, which I assume is thanks to rdf4j and possibly upgrading java.

A few days later, I took advantage of some posts I had found while figuring out how to reconnect apache and tomcat to set things up for https, so now all transactions to arachb.org and arachnolingua should be redirected to an https connection.  Switching over to https wasn’t much more difficult that configuring Apache in the first place.   I went with a commercial certificate provider since I had paid for a certificate I never activated a few years ago.  Hopefully the renewal will be easy.

On the inside, Apache and Tomcat are speaking over the JServ protocol (ajp13) which may help with performance as well.

Work on the curation tool is continuing, and I think I have found the sweet spot mix of methods, url redirects, and jinja2 templates to get the claim editor to finally lay down.  The story is all about switching between individual and class expression participants, but I’ll save it for another time, as I do need to finish the post I’m righting on my other blog on the role of machine learning in the study of behavior.

 

Advertisements

Website down

The AWS server I host on died on Wednesday.  It did seem to be a slow death, it was completely unavailable within hours of Amazon notifying me.   I tried to capture images from the EBS volume but the image I created didn’t boot.  So I’m rebuilding from scratch.  I have all the data stored locally, but I am taking the opportunity to update some of the software (Linux to Ubuntu 16.04, Apache to 2.4, Java to 8, Sesame to rdf4j 2).  I’m sticking with Tomcat7 on the backend, though I should probably do some experimenting with Tomcat8 since it claims to be easier to manage.  I will definitely make an image once everything is in place.  There’s a first time for everything and this is my first catastrophic failure on AWS.  Live and learn.

I’m working toward having things back up in the coming week.

 

 

Note to self and others

Arachnolingua focuses its OWL expressions on the claim (= assertion, statement) of an individual behavior event or class expression and properties should start there and work out.  Thus: courtship_event –has_participant–>palp–part_of–>male–has_type–>Habronattus sp.  There may be consequences for this decision (especially for class level statements), but it is better to be consistent and document the design decision here for now.

This should eventually make it into the ‘primer’ documents for the curation tool and the database schema as well.  I wonder if there are any tools in Protege for looking at graphs of individuals – maybe lego?

Last few weeks

Writing lots of unit tests and associated refactoring.  Part of this was inspired by reading though Martin’s (2008) Clean Code,  which had been sitting on my shelf for a couple of years.  Most useful thing I found was Martin’s admission that even he writes big ugly functions on the first pass.  Definitely a lot of cleanup in the new arachcurator editor.  Also triggering some simplification in the database – I’ve removed a many-to-many mapping table (participant2claim) since the relation is really many-to-one.  I think there are a couple of other tables that will suffer the same fate.

I should get back to cleaning up my list of terms as well.

img_20160615_102730841_hdr-1

Exploring Alternatives

One of the nice parts about the switch to pyramid is that it supports a range of alternatives.  For example, at the moment I am trying out forms in WTForms after converting the templates to jinja2.  In some ways the arachcurator tool has become a hybrid of technologies from pylons and from flask.  I guess this means if pylons disappears tomorrow, I won’t be starting from scratch.  It was a gut-level instinct call that lead me to prefer pyramid over flask in the first place.

Meanwhile, all the browsing code is built, tested, and now back in a state of flux as I shift forms technology.  There are 3djs graphs for both individuals and their part as well as expanded display trees on the claim page.  I have added users (mostly for change tracking) and three levels of authorization with the hope to put an arachcurator server up on AWS at some point (sharing more than code and demonstrating replicability I hope).  I even went so far as to add password encoding with passlib.

Meanwhile, as discussed here, I’ve started building a T-box vocabulary for spider behavior.  This will extend what can be said in annotations, in a way that changes here will only simplify.

 

 

Updating Sesame, starting the move to pyramid/postgresql

All technical stuff this week.  I’ve got the Spider-Behavior server code connected to travis-ci.  In the process of doing so, I ran into problem with Sesame, which turned out to be related to the maven setup.   It’s working and updated to the most recent 2.8 release.  This will allow some more changes, particularly some more formatting work on the narratives page.

While waiting for a response from the Sesame-users group, I started taking a serious look at moving arachadmin from web2py.  It looks like I’ll be able to move to a python3.5-pyramid-postgresql stack.  I have a first pass at the new table structure working with sqlalchemy, and will soon start the process of coding the transfer from mysql.  The experience is a lot closer to standard python development, rather than the somewhat cushioned ride offered by web2py.

Meanwhile, spending some time reviewing the opentree taxonomy against the world spider catalog.  So far nothing horrible, mostly groups added since 2009 and some unstable family splits.

No big news for Arachtober

Spider in leaf refuge

Unidentified spider constructing rolled leaf refuge.

Things weren’t really cleaned up enough for a big release announcement in October, though almost every page has had an update in the last month.  In addition to narrative ‘drill-down’ pages, all the other arachb.org/arachb/ARACHB_xxxxxxx identifiers that correspond to something in the database now resolve to simple pages.  The narrative drill-down pages (such as this) generate a table of their component events.  They can generate both html and json, the later is (in many cases) simply the ‘tunneled’ json that Owlbuilder serializes into an rdfs:comment annotation (as discussed here). These still need some tweaking and there is still information for ordering the events in the narrative.  This, table pagination, and some javascript cleanup remain before I could consider a release.  Then it will be time for some more curation and thoughts of updating the curation tool (python 3.5 and pyramid and mysql -> postgres).

Owlbuilder and server backend code are gradually migrating to Java 1.8.  Meanwhile Oracle’s cloud-related noises are making me nervous about the future of Java on non-Oracle environments.   Events conspire to make me nervous about Java every couple of years, so maybe it’s not time to worry yet.