Library update

Server issues

It’s not a secret that the server has been having issues. Problems are being fixed, but it could take a while. As a counter-measure, a fully up-to-date mirror has been setup at mirror.anarhisticka-biblioteka.net (thanks to anarhija.org)

It’s not a dynamic site (i.e., you can’t edit or post texts), but it’s browsable and it’s exactly the same setup of the DVD/torrent edition.

Markup engine

The markup engine has been rewritten (again) to be more maintainable and to produce LaTeX instead of ConTeXt. I’m giving the last touches and it’s growing well. The code is available at and will be released to the CPAN soon. Please contact me if you’d like to test it (I would welcome it, but no point in explaining here what to do, if nobody is going to do it).

See this previous post for a rationale

Page imposing

One of the core features of theanarchistlibrary.org is the ability to produce PDFs ready to be printed and folded. The process is called “imposing”. To do that, so far we called the TeX engine again on the produced PDF, to rearrange the pages. Now, this has been fixed and using some native Perl module, it is possible to do the job in a cleaner way, and to open the way to more possibilities (like printing A6 in A4 for smaller and cooler booklets).

The code is fully decoupled from the markup engine, so it could be used by anyone to arrange the pages of any format and size. There are probably bugs lurking around.

Wiki engine

The multilingual wikifarm is basically in the same conditions (i.e., not existent) than some months ago, because I wanted to fix the above issues first. I’m still unsure about the roadmap (if plugging the fresh code above in the existing code, or waiting for this thing to come up).

Posted in Devel | Comments closed

What’s going on at the library

Ok, it’s been a while I’ve written something here, but things are far from stalled.

First, I’ve been busy setting up and bootstrapping the other libraries, which now include, beside the English, the Russian, the Finnish and the Serbo-Croatian ones, even a Spanish one.

This is for sure a great news, but it also required time and efforts, which summed up with life issues, it’s taking a toll. Probably more libraries are coming, but I can’t know this in advance.

Today the server was moved, so we had something like 12 hours of downtime (not scary, for sure). Anyway, there are some considerations to do.

Resources

The new way of the web development, with persistent applications behind a proxy, is cool and shiny, and CPU-wise is better, as the apps are loaded only once, but it also cost a lot of RAM. Some figures:

MEM %
1.5  starman master 
1.7   \_ starman worker 
1.5   \_ starman worker 
1.9  starman master 
2.1   \_ starman worker 
2.1   \_ starman worker 
1.9  starman master 
2.4   \_ starman worker 
2.4   \_ starman worker 
1.9  starman master 
3.7   \_ starman worker 
3.7   \_ starman worker 
1.9  starman master 
2.1   \_ starman worker 
2.4   \_ starman worker 
2.3  starman master 
6.2   \_ starman worker 
6.3   \_ starman worker 
6.7   \_ starman worker 

Considering that the server has 2 GB of memory, that’s almost 400Mb only for the greedy English library. It’s unclear if it’s my code leaking (but I checked that), the way Perl manages memory, or don’t-know-what. I believe I can further lower the number of workers to save memory (as Nginx does a fantastic job in buffering the requests) but this on the long run must be fixed. I’ve started working on a “wikifarm”, but given the life issues I’m currently experiencing, this could take a while.

The TeX side

Two years ago (I can’t remember exactly when this happened), I moved away from (Xe)LaTeX to ConTeXt/luaTeX. It seemed a nice move, as it’s easily customizable, and gives full freedom to tweak the layout of the things. But…

Rant follows.

As a matter of fact, installing a functional copy of ConTeXt requires a good dose of luck. Things break. I follow the mailing list and almost every day there is a message along the lines of “Broken beta”. So you desperately need a feature and you don’t know what to do. You have to ask politely and hope for the best.

The only safe version right now, for the library project, it’s the TeXlive2012 one, installed from the .iso file. I can’t do a selective installation from the web, without the 2Gb file, because fonts and stuff are totally out of sync. So that’s it. It will have to last for a while. No new features. No experiments.

I’ve tried to build a testsuite for that. The testsuite works, but it’s almost impossible to follow the development, which happens without any kind of version control (in the Git epoch, when *everything* is under version control). I don’t know if the developer(s?) uses it. So I just use the testsuite against one single file to check whatever is safe to upgrade or not (mostly, it’s not).

ConTeXt suffers of featuritis. Shortly it will begins to read e-mails. It uses the Lua scripting to achieve various results, but as stated above, it breaks continually (no wonder, given the lack of testing, that’s it, let the user report the breakage). The LuaTeX version of ConTeXt it’s called MkIV. The developer releases so called “beta”, and once a while (it’s been a long time since then) a “current”, which is more or less a snapshot of the “beta”. Well, I think there is a misunderstanding here. Beta software in my book means testing before releasing. But here there is no release at sight. It’s more like an old “svn version”, but without the SVN part.

Rant ends.

So, I’m a little frustrated, as I would like to encourage people to download and tweak the source files of the library, but how can be done, if the installing of the software itself fails?

So I believe the moment is come to go back to the LaTeX environment. There are a couple of feature lacking (microtypography, conditional running, built-in indexes), but I came to the conclusion that stability is more important than features, and LaTeX has a tradition of stability (mostly because it’s intended for an academic audience, where you can’t imagine people getting crazy and asking the developer to fix the thing because it doesn’t work, but the developer is busy on its own stuff) — while ConTeXt basically comes from a publishing house which has its own agenda.

Also ConTeXt is monolithic and only a few persons (if not only the developer himself) knows how to fix it, while LaTeX has a way larger developer community and has the CTAN (like Perl has the CPAN).

I’ve added the code to output (Xe)LaTeX (without removing the existing output, which will be kept, just in case) to the current code. The codebase could be better, but it’s not so messy.

Like with the wikifarm, it’s unclear when the thing will be completed, don’t hold your breath too much.

Posted in Devel | Comments closed

The Anarchist Library on torrent released!

I’ve rolled up a new torrent bundle. See here for details

I admit I have neglected this blog for a while, while there was plenty of news on the Library’s side.

The library’s engine has been rewritten from scratch, using new, better tools, adopting a new markup language which is WAY more human-readable and human-editable. It’s basically a wikish markup and produces the usual formats (TeX, PDF, HTML, EPUB) in a safer way (as far as I can see). If you’re into this kind of things, you may want to take a look at the code repository on Gitorious.

If you’re asking why I’ve given 3 months of life for this, that’s easy: to support other languages. Before this upgrade the only other parallel project was the Russian site, and maintaining it was a pain. Lesson learned, code written. Now there are 4 libraries, they have full autonomy and they are reasonably user-friendly from the librarians point of view.

From the casual user’s point of view, nothing changed (the more attentive could have notice some lack of movement on the RSS).

The new torrent ships all the libraries as seen on May 22th, 2012, packed in 4 DVD/CD images. More information about what this means here

Posted in Devel | Comments closed

Quick update

While I was at this, I’ve changed a bit the CSS of the development version of the site. I’m pleased to offer some screenshot for comparison. It looks more 201x now 🙂

Old and new front page

Old and new front page

Text page

Text page

Posted in Devel | Comments closed

Upcoming feature: custom PDF creator

Looking at the “Book” feature of wikipedia, I was wondering if the library too can have something similar… or even better!

http://en.wikipedia.org/wiki/Special:Book

I’ve started working on some code, and so far it’s doing well… well, a bit messy, but definitively doable. Here there’s a preview of the rough interface I’ve built so far:

Basically, the user can choose its favorite texts, merge them, choose font and paper, and have the PDF ready in about 1 hour. I’ve thought about preparing them on the fly, but due to the not infinite server resources, it would expose too much the library to denials of service (imagine only 10 users wanting a custom PDF at the same time, the server would quickly become unresponsive).

Much is still to do: documentation for the user, the backend, the ConTeXt styles, and so on… So stay tuned!

Posted in Devel | Comments closed

Zagreb bookfair and the next torrent

Well, it’s time for another torrent release, with the fresh PDFs produced by ConTeXt. The DVD iso (2.1 Gb) is ready, has been tested, and now has been burned on 30 physical DVD, to be distributed at
the Zagreb bookfair

The printed stuff is also ready.

Once I’ll come back home, the torrent will be release, shipping the same ISO of the DVD. Hurray!

Posted in Devel | Leave a comment

Finished?

As today, December 22, 2010, http://theanarchistlibrary.org runs the new engine, using ConTeXt as typesetting system. Hope you’ll enjoy.

P.S. To celebrate I’ve printed out Nightmares of Reason (in its A4 imposed variant).

Posted in Devel | Leave a comment

Last touches and general update

Hello there.

I’ve not updated this blog for a while, but I was very busy coding and polishing and testing. Now I’m very, very close. While at the first glance the site will look the same, and the PDFs will look very similar, under the hood a lot of things changed.

Some other new features:

1) multilanguage support. This means that is possible to build a sister site in no time (just translate the interface and some tuning via a configuration file written in basic perl, something like $setting="value"). It’s also possible to use the library format (as described in the guidelines) as a standalone converter. And the library format is just a subset of html, just using blank lines as paragraph delimiter instead of the tag <p>

2) microtypography. The new PDFs, even if to the untrained eye looks more or less the same, have their quality really boosted. Btw, a side effect is that the copy and paste from the pdf will work better (no more ligatures shown as weird characters).

3) It’s possible to define new pdf formats with no pain. I was thinking about a web version (horizontal) and a “POD candidate” (Print on Demand candidate, for people that wants an hard copy printed by professional people) which should replace the unimposed a4/letter. Or even to be added. I don’t know, but at this point the possibilities are infinite.

4) Web interface: no changes for the end-user, minor changes for us; there’s a news editor (locked for the rest of the world) and a spool manager, to avoid the use of the shell just to move a couple of files in an archive directory.

5) Support for file renaming. Renaming (due to titles that get fixed) is/was a pain, because it leaves behind broken links and outdated files. I’ve added a “router” that searches in a file (which is possible to compile in no time thanks to the revision control) in the root of the “alibrary” git, and redirect to the correct links. As fallback, it will route to the search.

6) Some other tags support, including <em> and <br /> in the title and in the author fields (keeping the output formats standard compliant). Basic tables are now supported. Images are handled better. The purpose is to get rid of most of the <pre> </pre> shit.

7) General code cleaning that nobody cares about.

8.) the tag cloud will pull from both the authors and the topics, picking the one with more texts. Tags with names should be at this point deprecated. This will help keeping the tags to a sensible number.

9) There will be a new field, with “Author for sorting”. It should be set to Last, First, to have a proper index.

Posted in Devel | Leave a comment

Updated the repository

After a week of reflection, I’ve decided to merge the two repositories of the library code. I’ve pushed it to gitorious and I’ve updated the link.

The development will happen in the “ng” branch.

Posted in Devel | Leave a comment

More on testing

It turned out that I’m sick of testing against over 1200 files. So with a little of patience I’ve compiled a dozen of little files with the various cases.

I’ve packed them and uploaded on the server. The idea is to create a test file for each problem I could find.

If you want to check them, you can download them here

Posted in Devel | Leave a comment