Some other items around it to give it a sense of scale:

You can order them here: http://www.mynanoblock.com/site/

One note: some parts don’t stick together too well (notably the little pieces that attach under large pieces), so for longevity it might be worth to use some kind of glue to permanently attach the pieces.

I’m thinking of getting the giant 5000+ piece castle, but unfortunately it’s around $300…

Basic project structure is set up.

I have been looking at the OSM data further and thinking in the back of my mind about how to work with the large amount of data with the WPS. The main problem is that the planet.osm data is fairly large, thus will take time to download, and then process. This can be further broken down into two main subissues, namely:

• how to download/store the data such that WPS has access to it immediately
• how to modify the datastore quickly such that the WPS provides the most up-to-date information from OSM

The latter issue is easy to solve as planet.osm provides incremental daily or weekly updates of roughly 300MB in size. This small amount of data (compared to 21GB compressed, 303GB uncompressed fully planet.osm data) will be easy and fast to process given average server resources/speed (need to provide benchmarks for this). Integrating this is potentially a problem.

The former issue is more difficult as this is quite clearly the initialization step. In terms of storage, there are several options:

• Use a db-based backend, e.g. PostgreSQL database with PostGIS extensions installed; There exists osm2pgsql, but it does lossy conversion. There is also osmosis.
• Use a OSM-native backend. Examples of this are using the Overpass API or other APIs
• Roll our own datastore/backend within WPS. This is easily feasible for small osm data, but not so much for large osm data. This would probably work with GeoTools as this is what WPS primarily uses for conversion to different formats. This might be slow, however.

If we go with the last option, this would probably involve rolling out our own class extending the AbstractFileDataStore class in GeoTools or that of IData in WPS. The specifics of how this would work would be (depending on what sort of defaults we’d want) store general features, and create some sort of hierarchy from general to specific for particular regions? This is dependent on how large particular regions are in terms of data available, and would require a significant preprocessing stage. Currently it is unknown if the planet.osm data is sorted.

It also turns out that the osmosis tool is written in Java, and has some useful benchmarks, so it might be worth looking at some of the classes within osmosis on github to see how fast processing of osm data is done. Despite the large amount of data, it appears that it is possible to process them rather fast without a significant memory footprint.

It appears more and more that this will have to be a fork of trunk at some point possibly. I can think of one way to organize the different components of osmtransform in the project:

• the preprocessing part of osmtransform to be part of the IO module, namely in the parsers package. Depending on the user needs, it may be worth to set a separate backend/process to handle this. Basically:
  • Two possible parsers: one for file-based OSM data, one for API-based OSM data.
    • The file-based OSM data will need some preprocessing based on the size of the OSM data. This is converted into some kind of intermediary format, with some preprocessing to make requests faster. Probably some hierarchal sorting (i.e. by blocks, or by region) will be useful.
    • The API-based OSM data probably will need minimal preprocessing, and instead serve more as a passthrough interface to the OSM process
  • Two possible processes/AlgorithmRepositories: one for file-based OSM data, one for API-based OSM data
    • The file-based OSM data will require an AlgorithmRepository that can handle the internal converted format. The capabilities which it supports is… ???? There will exist options for which mirror to pull the OSM data from, and how often it updates
    • The API-based OSM data will require an AlgorithmRepository that handles the OSM data directly as queried by the OSM provider. There will exist options for the source of the API-based OSM data, and whether it has write access (and what API format it uses)
• the ui part of osmtransform would likely be a separate web page that performs some basic operations on OSM data, that are forwarded to the WPS itself.

Diagram of workflow would look like this:

PROBLEMS

Some details of the ui may not quite be as clearcut as it would initially seem. As mentioned before, there was discussion as to whether the UI component would be Java Swing-based or web-based. If the former option, it would act like one of the existing clients that interact with the WPS. If the latter option, then would probably be some additional webpage for the WPS? Assuming we go with the latter option, it is quite possible that we can have certain kinds of requests be served over a web interface, e.g. integrate OpenLayers into the WPS. This might facilitate easy testing, and allow for uploads of smaller osm data for conversion.

TODO FOR THE WEEK

Start writing the Parser classes for planet.osm data and OSM3S data. Look at the osmosis code and get hints for possible speedups. Document all the facilities that OSM3S provides, and start composing the AlgorithmRepository subclass that provides the capabilities of OSM3S servers.

We had a phone call with Benjamin on Friday, to discuss where I was regarding to schedule, and to discuss the preliminaries of UI mockup. We are slightly behind schedule in that the UI mockup should probably have been completed last week, so hopefully by the middle of this week this should be sorted out. The question posed to Benjamin was whether or not this would be Java Swing-based or whether would be web-based, the consensus being that since this was an addition to WPS, it would probably have to be web-based.

PROBLEMS

So one problem that prevented work from getting done this weekend was sporadic power outages in the Waterloo region. There were some work lost as various parts of campus and residences either cycled power at various points of the day, or lost power for several hours, which was rather unfortunate. This was extremely unusual, and also highly frustrating. It is also a holiday today (Victoria Day).

TODO FOR THE WEEK

As per the schedule, the TODO for this week would be to flesh out the preliminaries of that UI mockup (and get feedback from Benjamin and/or other parties). This will probably be put on the wiki somewhere. Also the other TODO for this week is to start committing code to the repository, basically as a skeleton structure for the different components of the project.

This project will be divided into three main components:

• User Interface component – this is a visual component where the user can select a portion of a map that he/she intends to convert between formats
• Data Preprocessing component – this is a backend/systems component which is in charge of taking an OSM data file and converting it into an internal representation understandable by WPS
• Data Postprocessing component – this is a backend/systems component which is in charge of taking the features selected by the user, doing any sanity checks/processing, then performing the final conversion into a GIS format specified by the user

The final product from this project is intended to be a plugin for WPS with the three discrete components as mentioned above, designed/organized in such a way that it will be easy to migrate this into the core WPS should 52°North wish this to be a core feature.

The input format will be any valid OSM files, which are primarily XML-based but may also be PBF-based. Such files may involve data for the entire planet, or discrete regions; the plugin should be able to handle both sizes. While the primary focus of the project will be to translate the OSM/XML data into a usable format understandable by the WPS, investigation of the conversion of the PBF file format may be a possibility as time permits.

The two main output formats that are desired for the conversion process are: Shapefiles and GML files.

Additional post about GSoC and 52north: http://52north.org/52north/news/52dnorth-to-mentor-4-google-summer-of-codetm-student-programmers

This is aimed at the more advanced users of Linux who wish to use/install Arch Linux on their OpenVZ VPS. This is basically a compilation of solutions to the different problems I have encountered thus far, and may be updated as I find more issues that need to be resolved.

Several assumptions are made as to the Linux background of the VPS user:

• You know basic shell and filesystem commands (i.e. filesystem traversal and file manipulation)
• You know how to use a text editor
• You understand all of the words in this post so far
• You know how to use pacman, Arch’s package manager
• You are using Arch Linux 32-bit on your VPS

Background
One of the reasons why this exists is due to the fact that Arch cannot use the most up to date kernel available in the repository. The VPS in fact relies on the kernel running on the virtualization system set up by x10Hosting (OpenVZ), currently at 2.6.18 (2.6.18-238.5.1.el5.028stab085.5). This causes a host of problems to spawn, primarily with glibc and udev, which can result in an unusable system. Thus, workarounds must be sought.

What is Arch Linux?
Arch Linux is a distribution whose core philosophy centers around simplicity, minimalism, and code elegance. It is not aimed at the beginner to Linux, and is not a derivative of Debian/Ubuntu or Red Hat/Fedora. Arch’s packages tend to be bleeding-edge, i.e. providing binary versions of the latest versions of applications from the Linux community.
Read more at http://www.archlinux.org/about/.

Issue 1: FATAL: kernel too old
(source: https://wiki.archlinux.org/index.php/VPS_Repo)
This problem stems from trying to upgrade your system to bleeding edge. The latest version of glibc in most repositories will break your system as soon as it is installed, causing these messages to appear whenever you try to run a command. The solution is to use a version of glibc that has been specially compiled to work on x10′s kernel. The following steps must be done before any further pacman actions are carried out:

• Add the following in /etc/pacman.conf above [core]

[glibc-vps]
Server = http://dev.archlinux.org/~ibiru/openvz/glibc-vps/i686


• Example:


...snip...

[glibc-vps]
Server = http://dev.archlinux.org/~ibiru/openvz/glibc-vps/i686

[core]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrorlist

[extra]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrorlist

[community]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrorlist

...snip...


• Edit /etc/pacman.d/mirrorlist as necessary
• Run the following to upgrade your system:

pacman -Syu

Issue 2: PTY allocation request failed on channel 0
(source: http://forums.quantact.com/viewtopic.php?f=35&t=1149)
This problem stems after performing a system upgrade on the VPS (or upgrading udev), rebooting the VPS, then trying to log in via SSH (or the VPS console). This is a result of updating udev, which for some reason doesn’t allow for the necessary virtual terminals to be created. To fix, the following must be done before rebooting the system:

• Add the following to /etc/rc.local


rm -rf /dev/ptmx
mknod /dev/ptmx c 5 2
chmod 666 /dev/ptmx
umount /dev/pts
rm -rf /dev/pts
mkdir /dev/pts
mount /dev/pts


That’s it for today.