When writing software, you often analyze several ways to accomplish a task and embark on what seems like the best one; but on occasion, the stars align and the meaning of life becomes crystal clear. Your solution is so perfect and elegant that you know, even feel, that it is right and it shouldn’t be any other way. I detail my experience with this below.

First, let me describe the problem. I was tasked with writing the search subsystem of our project (which was not a trivial task if you check our schema). I was excited for the task, and I spent a great deal of time working on the details. I decided to do a filter/restriction approach, where people would specify the data they wanted to search (filter) and the data they wanted to find (restrictions). Using this method, you could add multiple filters and even multiple restrictions per filter quite easily. Given our schema, I was not immediately sure how it was going to come together. However, in order to get something semi-workable early on for other members to work with I wrote a simple, yet fairly flexible query generator. It worked quite well when searching a single table, as it could handle multiple restrictions, and would prove quite useful later.

While discussing the project with Allyn, we decided it might save some time to use Sphinx, an open source search engine. I then spent hours installing it, working with examples and trying to set it up to search our fresh, semi-complete class database. About ten or fifteen hours of work later, I determined it was great for searching text fields, but it did not suit our project’s needs.

Frustrated, I took a some time off to fiddle with other things that needed to be done. Suddenly I realized there was serendipity in my Sphinx research, as I followed a tutorial that forced me to use views, a mechanism in SQL that allows you to compose data from multiple sources into a single table. While I knew of them, I did not know an actual use for them. Then it hit me like a brick wall: I knew how to search a single table easily, and with views I could easily bring all of the data I desired into a single table! I nearly shouted “Eureka!” as, in a very brief moment, I mentally saw all of the pieces fall into place. As I had not committed any of my Sphinx related code, I took a chance: I deleted ten plus hours of work and hacked up some code. In two hours I cut the first version of the search system we used for the rest of the project (revision 113 out of 243).

Later changes were mostly trivial additions or refactoring. As I composed all of the data into a single table for searches, I also took all of the data we required to be presented and combined them as well (the views are here). With this done, we mostly had cosmetic changes and testing to deal with. I integrated verification functions and created a paging system to simplify work on the user interface. But the true feature of this system was its flexibility, and sadly was the biggest thing we forgot to mention to the class during our presentation (a story for another time). Adding a new filter to both the user interface and search is a matter of adding a row to the filters table in the database, and if any more search data is required you merely update the view. By changing one line of SQL and adding a row to a table you can create filters rapidly, remove them, deactivate/reactivate, change, update. Nothing is hard-coded. In fact, we removed some functionality because we only sported data from one semester, so filters for semester and year were removed since all data was from Spring 2010.

Search was the core functionality that drove the project, and without it the system would be pointless, so I was very excited to see it come together. It has some issues still, but for only a few weeks worth of work I am impressed with the system we built. Feel free to give it a test run: http://allynbauer.com/seproject. If you get a dinosaur, try hitting the link again

–

Ideas we wanted to talk about and learn about: distributed processing, high performance computing, formal software engineering process and techniques, AI programming and emergent behavior.

Ideas we did not want to work on in-depth: graphics programming, user interface creation and design, user input.

When Dr. Gray allowed us to borrow the LittleFe, we instantly decided to use it for this project. This provides us with a high performance, transportable medium to do our testing (although I would later assemble a simple, low-end machine to run smaller MPI-emulated simulations). After much discussing, and even consideration of OpenCL and/or Cuda, we opted for an MPI (Message Passing Interface) based application for this project.

The goal itself is not to make a video game, but a large-scale battle simulation involving thousands (hopefully tens or hundreds of thousands) of units. We want a lot of units to readily see the effects of AI changes and to stress the system processing the updates. Ultimately a simulation will consist of several players, each with a defined rule set (which will be simple and defined by us, nothing too fancy) that will govern their units as each side faces off. We hope someday to have several unit types, terrain considerations, random catastrophic events (meteors, nukes, tornados, etc), and an event tracking system.

ABE will have two core parts, the simulation and the player. The simulation will run on the LittleFe or some medium and will load the rulesets, run the simulation, and then output information step-by-step to text files. We will then have a player that can read these generated text files and somehow visually display each step so we can actually see what is happening. The player should be able to play, pause, and advance one step at a time.

So what does this give us? Anyone who is interested but doesn’t know how to program will be able to talk about and influence our decisions and design, and even create their own AI rule-sets. This can lead to a lot of student interaction without a lot of domain knowledge. We realize not everyone wants to get into the technical aspect, so we want to take that out of the picture and talk more about our design, process, and issues. We figure this will make the project more marketable.

As the software process is an important learning part of this project, we plan to make extensive use of things not usually seen in student projects. DISTek, my employer, has graciously agreed to let us use their conference rooms for design and code reviews, so we will try to take full advantage of this. I don’t expect many people who will help with this project to understand the code I write, so their ability to review it effectively may be low, but it is a chance to help them learn none the less. The project is hosted in a Subversion repository and we will try to keep it up to date as possible. If things go well we will probably release the code under GPL or a similar license and allow people to use it once we have a stable, working version. Also, as we are following a more formal software engineering process, I personally will try out CPPUnit and attempt some test driven development. This is something I personally am interested in and would like to try on something beyond homework-scale.

Currently I am set to design and write the simulation. Dan is tasked with creating the player and, perhaps more importantly, setting up the LittleFe in a way we can use it. He is more of a Linux junkie then me these days, and he seemed quite interested in it so it should be fun for him. I personally can’t wait to dig into MPI and get things moving, but I don’t want this project to fall on its face. Thus, I will probably spent some large amounts of time designing things once I have a good idea how to setup the simulation correctly to take full advantage of MPI.

AA Tree’s are very similar to Red-Black trees. They add one rule in addition to the traditional Red-Black tree rules, and that is that a Red node may only be a Right child. This eliminates a substantial amount of special cases, making this algorithm far easier to implement. Interestingly enough, AA trees perform as well as Red-Black trees. Since the AA algorithm is simpler, it takes less time to compute, but since it makes more rotations then Red-Black trees the performance evens out. Due to the nature of the algorithm (which shapes the tree), AA trees also tend to have better retrieval times then Red-Black trees. For those unfamiliar with the tree (which is probably most people), there are only two operations required for maintaining the tree (compared to the red-black tree’s five). A right rotation occurs when an insertion or deletion creates a red left child; this is known as a Skew. Split’s perform a (conditional) left rotation when a function creates two red links in a row.

My program implements a string to integer map using AA Trees. As an example, it counts word frequencies in a file. The string handling is rather rudimentary (converts to lower case and tosses out punctuation), so some of the words are kind of funky looking, but it works pretty well. I was surprised at how fast this program ran. I didn’t time it exactly, but it scanned a 11.7MB text file full of text in less than 30 seconds.

During my coding I hit several roadblocks. Coding the algorithm went well, but when I was programming my insertion function it all went screwy. I rewrote it three or four times only to realize that my initialization in my constructor was wrong (while cascading assignments for pointers I forgot the NULL at the end). While stepping through the program for an hour using GDB, I was rather clueless as to what was going on. Everything was wrong. None of the pointers were right, none had key data stored, and it crashed randomly. After fixing this I ran into an error with Skew running on my root pointer, which should never happen. Once I got passed these problems all remaining changes were pretty cosmetic.

I would love to package this up using templates and reuse this tree implementation elsewhere. It is pretty applicable, but again, the STL has a very good built-in map class.

About the code:
I did not flesh this code out much. The tree lacks a deletion function, and I show no examples of how to retrieve a word’s count using Lookup, though it is pretty straight forward. I use Lookup primarily as a wrapper for Insert (and for clarity), differentiating the names for their intended use within the code. There is also a large amount of pointer manipulation in this code, which I tried to document, but it may still be confusing to those new to pointers. Interestingly enough, while the algorithm itself contains a significant amount of references to red and black nodes (which this algorithm is based on), there is no reference to such in the code. The code assumes all Right links are red nodes, and instead everything is based off of the “level” of a node, which is the number of left links that must be taken to reach a NULL pointer.

It comes with a few text files as well. Data.txt is a smaller text file I used to test, being about 500KB or so, which resulted in out.txt. The file test.txt was a small test, and bigout.txt is the result of scanning the whole 11.7MB file.

Code -> http://www.nick-cash.com/download/cs3/tree

Note: A wonderful summary, analysis, and comparison between some of the most common efficient data types can be found at http://www.upgrade-cepis.org/issues/2004/5/up5-5Mosaic.pdf. I recommend at least a glance at the diagrams. It is obvious that AA trees are outperformed on many levels, but they hold steady and beat data types on various challenges. Considering the simplicity of the algorithm and ease of implementation compared to the others, I’d say its far more useful.

What I ended up with after this assignment was a class that replicated most features of STL’s built-in list class. It is templated to allow for any data type, and given a couple tests it seemed to work well. I purposefully avoided the use of any STL data types to drive the class, as that would be counter productive in the learning and showing of knowledge about lists.

I experienced many oddities with lists while I programed. After, again, fighting with template syntax, and I had an initial implementation I realized I was indeed missing a vital feature of a list data type: the ability to traverse it. When I gave the Element class the ability to cough up its pointers, it broke all list class functions that needed to set them since that data was now private. I am not totally happy with this implementation because it gives everyone the ability to modify a given elements pointers, not just its data. This is not good since anyone using those functions could potentially screw up the list, and things like that can be very, very hard to track down (nothing like spending hours stepping through a program to evaluate pointers at every step).

However, this was a very vital feature, so once I added it I found I was able to do quite a bit more with the list. An interesting side effect of my implementation is that you can add and remove things to the list without adversely affecting any current iteration (supposing some pointers are handled correctly). I demonstrate this slightly in the example programs provided by inserting an element into a list when a certain point is reached (and then that element is eventually processed like the rest).

When implementing the ability to traverse the list I found it made the code far more obscure. Lines that were already strange looking, like:

x->next->prev = x->prev;

became

(x->GetNext())->SetPrev(x->GetPrev());

And that, as far as readability of code, is very bad. However, its fairly simple to reason out, and with comments I feel no need to separate it into multiple lines of code. Also, each member function of the class is pretty specific and short, and they would normally appear multiple times within code, so each function is inline. I have no real way to tell if this is faster then outline code (is that even what its called?), but it seems reasonable that it should be inline.

Note: The code below is separated into different files for readability. It should be noted that all the code in list.h and list.cpp -NEED- to be in whatever file they will be used by. Therefore, all of the test files (main_int.cpp and main_string.cpp) would need this code copied and pasted into them to work. This has to do with the nature of how C++ uses templates and the code must be present in the source file it is used by. Due to the way I program this is awfully inconvenient, as I love my larger number of smaller file type projects. Also, the output from the two given test programs is provided in text format for examination.

Code -> http://www.nick-cash.com/download/cs3/list

In any event, I decided to jump into hashing, and hash tables more specifically. Hash tables are something I’m very used to because when I started programming with SMAUG it had a hash table built into it (if you have a thousand instances of a mob it is an awful waste of space to allocate the same string a thousand times). So I learned the in’s and out’s required to use it, but like all of my knowledge at the time, I knew how to make it work without knowing -why- it worked. That has changed.

Data Structure Techniques by Thomas A. Standish had a lot of good information on hash tables. It taught me many things, even with my preconceived notions of what a hash table did and what its purpose was. I never even considered adding a key by a method other then division (modulo). His text on the multiplicative method is very interesting, though it seems a little too narrow to be useful in a large number of instances.

I initially had trouble testing my different hashing functions for one very simple reason: my computer. Since my last computer bit the dust (the eMachine lasted 5 years with no problems, I’d say thats pretty good quality), I’ve finally invested the money into a -real- computer (more on that here). Simply put, with the original test cases my independent, worst-case variable, the list structure, performed so fast I had to come up with another solution.

As I was in node mood to figure out and implement the use of a high performance timer (known as QueryPerformanceCounter on Windows), I merely used google and grabbed an easy implementation (doesn’t get easier then one structure!). After implementing that, I increased the number of items hashed by about 100 fold, bring most of the searches up to 15 million items. This gave a sufficient look at the differences between hashing functions and their performance compared to a list.

To compare hash functions, I went to Wikipedia (I think) and grabbed a recommended hash function, which happened to be Jenkin’s One-At-A-Time Hash. I then implemented two more very basic hashes. One was based off the SMAUG version, which hashes a string based on its length. The other is based on the first letter of the string. Oddly enough I named that CaseHash, when really it has nothing to do with the case of the first letter, but the letter itself.

The data was rather astounding. Jenkin’s hash almost always performed as well as or better then all other methods. Sometimes it would take a little longer (we are talking less then hundredths of a second longer though), but almost always was the top performer. Note in the final screen shot how it outperforms the other has functions, or performs as well. When doing a particularly hard lookup it would perform much, much better.

Provided via the link below is the source and a couple of screen shots (some have different output because I was busy tinkering). The first time is provided using the time() function, the second time is using the performance timer (with a precision of 15).

Code and screen shots -> http://www.nick-cash.com/download/cs3/hash

Notes:
1) I intended for this hash table implementation to be far more flexible, but after fighting with templates for quite some time I decided to merely call it quits and hard code the strings in.
2) In screen shots 1,2, and 3 it appears to be sorting numbers. These are not numbers, but in fact strings that contain incrementing numbers. This was the product of the program I used to generate the data.
3) The programs I used to generate the data are also included in the link above. They are small and simple.
4) All code listed above is public domain. Use it as you like, or don’t use it. Your choice.