Last few weeks

Writing lots of unit tests and associated refactoring.  Part of this was inspired by reading though Martin’s (2008) Clean Code,  which had been sitting on my shelf for a couple of years.  Most useful thing I found was Martin’s admission that even he writes big ugly functions on the first pass.  Definitely a lot of cleanup in the new arachcurator editor.  Also triggering some simplification in the database – I’ve removed a many-to-many mapping table (participant2claim) since the relation is really many-to-one.  I think there are a couple of other tables that will suffer the same fate.

I should get back to cleaning up my list of terms as well.

img_20160615_102730841_hdr-1

Advertisements

Summer Ontologizing

I’ve been quiet since April, but I’ve also been pretty busy.  Still fighting with the reimplementation of claim editing (a big, messy web page that if I could figure how to simplify further, I would).  I have also been focusing my efforts on a new ontology specifically for spider behavior (something to find in between the NBO/ABO and the data in arachnolingua).  I gave a talk about it at the 20th International Congress of Arachnology a few weeks ago.  The slides, rendered as PDF, are available here.

There is a link to the work in progress on the arachnolingua home page.   It is currently just the initial google doc sheet I used to collect usages across the two source texts.  I am finishing the first cleanup pass over the data and will provide an updated (and cleaned and better organized) sheet linked from a proper landing page in the coming week.

 

Quick Note

I am reworking the semantics of claims (statements about behavior dispositions or behavior events).  In the existing database (mysql arachadmin on figshare), there is a table called ‘participant_type’.  This table has been renamed to ‘expression_type’ in the updated psql database.  This is a better name since the values are really types of OWL expressions (e.g., some, individual, conjunction, etc.).   This morning I am adding a new table called ‘participant_type’ which has two values: individual or class.  A participant is either an individual or a class, although there are some funny, potentially confusing cases.  For example, some portion of tissue that is part_of an individual or an area that is part_of the surface of an individual.  In these cases, the portion of tissue or surface area may not constitute the entirety of the part (so not comprising an anatomical structure).  For the moment, I am thinking of modeling this as a class (though certainly not a universal) that still can participate in an individual event.  The other other seems to be to generate an anonymous individual as a instance of that class, but that seems even more wrong.  Dispositions of individuals towards a class (e.g. ‘subject tarantula showed increased consumption of crickets’) seems less problematic – it’s a statement about an individual spider, but not an individual event.

 

Progress

tan_jumper

This post is a bit overdue.  Good things are happening, the arachcurator tool continues to move forward – windows for browsing and editing entities (individuals, claims, narratives) are mostly functional, though they still require a lot of polish.  I’ve included some screen shots below.  There is also progress on the behavior ontology front – we may finally have smoothed the path forward to getting the ABO and NBO merged (I’ll probably post more on the other blog when I more certain that we can start moving into a production phase).

Continue reading

News from Sesame

Sesame is the triple store I’m currently using to back querying on the arachnolingua web page.  I just saw the message that Sesame has been adopted by the Eclipse Foundation as RDF4J.  This is probably a good thing, but it’s wait and see for now.   Nothing will break anytime soon, I’m still using the previous version (2.8).  Meanwhile fixing table permissions and starting on actual editing in arachcurator.

 

Progress with Pyramid

I’ve been pushing the conversion of arachadmin over to Pyramid pretty hard the past few weeks.   I started with the SQLAlchemy tutorial and then started with the project template, tried setting up the database (in Postgresql) in both Core and ORMS and chose the later (mostly for the more complete transaction support).  I rewrote the initdb script in the template to load and copy every (used) table in the current MySQL database and stuff it in the postgreSQL database defined by the schema.  Lots of problems with ordering of transactions due to primary keys that were also foreign keys points to a table of identifiers (goodbye mandatory integer indexes).  Just of note – I used a master table of URIs that were used as primary keys for records that persist objects (pubs, terms, properties, claims, individuals, taxa) as a straightforward way to guarantee uniqueness.  This is an issue because most of these classes of objects can have generated ids and this is an easy way to backup the id generation system.

Once I had the database, I picked up enough of Chameleon and Deform to implement (very) basic listing and browsing pages for publications, taxa, and claim (still in progress).  I haven’t done enough with Deform past the default display to have much to say, but the Chameleon templating system is working well – the basic templating and the macro system just work and I haven’t had any nasty surprises in each.  As long as I remember what belongs in the template and what needs to get passed in from the view function, everything is fine.

I’ve been trying to figure out why I feel more comfortable tinkering with the guts in this code than I was with web2py.  I think the difference is that a lot more is exposed in the process of getting started, especially moving beyond the basic ‘hello world’ app.  Web2py hides a lot more and it seems less transparent about the documentation examples are doing.  For example, specifying widths of strings in database columns vs. forms: because the packages for database and forms are separate in pyramid (and there are options) it is clearer what specifying a width in a particular place means.  In web2py it wasn’t clear whether a string length was specifying a database column width, a form display width, or a form validation constraint.  In summary, there is a lot of functionality packed into web2py, but I had trouble building a mental model of what was going on with a lot of the functionality (especially views).

The lack of serious support for unit-testing (beyond doc tests) didn’t encourage experimentation either.  I’m also happy that pylint, despite some complaints, seems a lot happier with the pyramid code I’ve feed it than when I tried web2py code.  With web2py I had to limit myself mostly to pep8 formatting if I didn’t want to drown in lists of problems due to the mismatch between pylint and web2py.  There is a pylint init file that claims to work with web2py code but I never saw any effect of including it (it said it loaded, then reported the same problems).

A new entry in the world of python code-checkers (or linters if you prefer) is the online tool from QuantifiedCode.  I haven’t tried this tool (or their new code ‘corrector’) on the new arachadmin code because I’m waiting for Pylons to officially release Pyramid 1.6 before I commit everything to github (just simpler that way).  The beta has been out for two weeks and no problems reported (I am actually using an older, alpha release of 1.6, no problems to report).

Enough rambling.  Wayne Maddison published a taxonomic revision of the Salticidae last month and at the same time released a large collection of his specimen photos.  So, since I have nothing new, have a northern Phidippus (not audex, which I didn’t find any pictures of).

 

na13-6898

 

 

PostgreSQL conversion continues

I’ve been writing the conversion script to pull the arachadmin database out of MySQL and into PostgreSQL.  Along the way I’ve fixed some problems I hadn’t noticed, so there will be at least one final update of the MySQL export on figshare.  I will commit the postgreSQL versions as a separate figshare document.  At the very least this will avoid confusion – one document if you want the legacy mysql and a different document for the maintained postgresql.  Getting this to work will probably close a number of the tickets in the arachadmin tracker.  Note that the associated wiki has a more-or-less complete coverage of the tables as of last August.  Expect an update once the database conversion has settled out.

Another advantage of moving to pyramid is the hope that the arachadmin editor will be in a state to distribute as a package – set up to run with Python 3.5.  Install postgres, the package, and the database from figshare and anyone should be able to browse a clone of the backend database.  A small step from ‘open source’ towards reproduciblity.

I plan call the new version of the editor app ‘arachcurator’.  I was planning to make the initial github commit this week, but since pyramid is promising that version 1.6 will come out of beta in a week or so, I’ll wait for that and build from that.  Changing the name is both more accurate (it does very little administration, but lots of curation) and means it won’t share a name with the database itself.   Perhaps I’ll change the name of the database when I commit it to figureshare (perhaps ‘backend’ term from spider anatomy).