How Else to Create Lemmatized Text for Topic Modeling

Here it is at last, the first post in the new blog for the SeNeReKo project. On this blog, we will write about aspects of our work, ongoing experiments and lessons learned. Our work heavily builds on the work of others, and we hope it will be of use beyond our project as well. That is why I want to open this blog with a response to another blog post.

Crowds gather to watch cranes joining two ends of an oil pipeline before the official ceremony commemorating the joining of the pipeline of an oil tanker terminal, Portland, Maine, with refineries in Montreal, Quebec.

In his post How to Create Lemmatized (French) Text for Topic Modeling, Christof described how to make use of the TreeTagger output when preparing texts for Topic Modeling. As a by-product of our own work in SeNeReKo, we aimed for a more generic and hopefully simpler approach to deal with annotations of this kind. We chose WebLicht as a starting point, because it allows to easily build annotation toolchains using independent components. Besides others, it supports TreeTagger for lemmatizing French, so generally WebLicht would be a good match for Christof’s task.

But there are a few issues that we encountered with WebLicht:

  • New services have to be registered officially to be available in WebLicht. This is often not possible when developing a new service that does not already run on publicly accessible infrastructure.
  • The graphical toolchain editor is really nice to annotate individual texts, but less suited for batch-processing larger collections. The tools are accessible using a REST interface, but writing shell scripts with a series of relatively obscure curl commands is not exactly helpful.
  • While there is good support for Java in the official WebLicht infrastructure, it is not so easy to implement new components in other languages, e.g. Python.

To overcome these limitations, I wrote TCFlib, a Python library for writing and using WebLicht annotation services. (The library is called not very imaginatively after the XML exchange format for WebLicht.) While it makes it easy to write new annotation services, it also makes it easy to use WebLicht services from Python.

Building an annotation pipeline with TCFlib

So Christof’s task of using TreeTagger to prepare texts for Topic Modelling could be solved like this in TCFlib:

from glob import glob
import os.path

from tcflib.service import RemoteWorker, Read, Write
from tcflib.examples.nltk_tokenizer import NltkTokenizer
from tcflib.examples.remote_tcf_converter import ToTCFConverter
from tcflib.examples.mallet_exporter import MalletExporter

files = glob('input/*.txt')
for infile in files:
    print('Processing ' + infile)
    prefix = os.path.basename(infile)
    Read(infile) \
        | ToTCFConverter(language='fr') \
        | NltkTokenizer() \
        | RemoteWorker(url='') \
        | MalletExporter(prefix=prefix) \
        | Write(infile + '_mallet')

So basically a pipeline in TCFlib consists of a series of individual annotators (called workers) that are chained together using the pipe syntax. In a pipeline, local and remote workers can be combined seamlessly. In this case, the ToTCFConverter actually uses the remote WebLicht service. Besides the provided example workers, one can use any WebLicht service by using the generic RemoteWorker. The URLs of these services can be found in the WebLicht tool list.


WebLicht tool list

Preparing input

Using a local pipeline instead of the WebLicht application allows to easily exchange some services by custom versions. So for example, the WebLicht tokenizers support only word and sentence tokenization, but not text structure like paragraphs or scenes. But this information might be useful for splitting the input into smaller chunks for topic modeling. The example tokenizer in TCFlib uses NLTK for word and sentence tokenization, but in addition to that adds paragraph annotations based on empty lines in the input file.

So using the structural information in the original TEI files, it is possible to create plain-text input that carries this basic structural information. The mallet exporter can then use it to export each section – in this case: scene – as an individual document.

This small script takes all the TEI files and extracts the lines as plain text, adding an empty line between the scenes:

from glob import glob
from lxml import etree

files = glob('input/*.xml')
for infile in files:
    tree = etree.parse(infile)
    with open(infile + '.txt', 'w') as outfile:
        for scene in tree.xpath('//div2'):
            for line in scene.xpath('.//l/text() | .//s/text()'):
                outfile.write('\n')  # line ending
            outfile.write('\n')  # scene ending

These plain text files can then be used as input for the annotation pipeline. The result are lemmatized files in mallet’s input format which contain each scene as a document. The mallet files for all plays can be concatenated and fed into mallet:

cat input/*mallet > collected_input
mallet import-file --input collected_input --keep-sequence --token-regex '\S+' --output output/collected_input.mallet
mallet train-topics --input output/collected_input.mallet --num-topics 20 --optimize-interval 20 --output-state output/topic-state.gz --output-topic-keys output/topic-keys.txt --output-doc-topics output/doc-topics.txt


The TCFlib API is usable, but it is still in development. At this moment, there is no guarantee that the API will not change. It also does not cover all annotation layers that are covered by the TCF file format.

If you want to try it out now, I would recommend you use the branch feat-fasttcf. It’s implementation of TCF reading is more pythonic and far more efficient, making it 55× faster with this annotation pipeline. It will be merged into the master branch as soon as our own services that use TCFlib are ported to the new API.

I appreciate any feedback with regard to TCFlib. Contributions and extensions are welcome. Please let me know if you use TCFlib in your own project and if you have any questions or suggestions.

Update (2014-10-27):

The fasttcf branch has been merged into master, so this article should apply to the recently released TCFlib 0.2. A more systematic documentation is also available.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.