NightHawk Database

NightHawk uses the latest in database technology to securely store your data. I thought I'd take a little time and talk about our data architecture.

Heart of the 'Hawk

When I was designing NightHawk, one of my first decisions was our database. Databases have come a long way since Now Up-to-Date's and Now Contact's databases were written in the early '90s. The NUD and NC databases are flat-file databases designed for high speed and small memory footprint. While they were excellent back in "The Day", requirements change. For NightHawk, I wanted a database that:

• is an embedded SQL database
• is FAST for data retrieval and reasonably quick for storing new data
• has a reasonable and predictable memory footprint
• has a reasonable startup time
• can store millions of records
• runs on Macintosh, Windows, and Linux
• is open source (or at least I can get the source code if need be)
• integrates with other tools for data mapping, data analysis, and data interchange

Wow, that's a tall order! From my survey of the field, there are only a handful of databases that meet those requirements, including SQLite, HSQLDB, H2, and Derby. Of those, I like SQLite and HSQLDB the most. Although I'd used SQLite on my proof-of-concept for NightHawk and was familiar with its source, I opted to use HSQLDB mainly because it is better supported by other tools that we're using.

Object Mapping

NightHawk is an object-oriented application, so my goal was to store my data as objects in a database. Unfortunately there's a small (yet significant) disconnect in mapping objects into SQL. To make things easier, I decided to use an ORM ("Object Relationship Mapping") layer. All an ORM does is provide an object-oriented interface to the SQL database. The ORM I chose is called "Hibernate", and it does a surprisingly good job. Using Hibernate I create a file that describes my data objects and then have Hibernate generate the SQL for me. This is similar to Apple's CoreData in Tiger, except Hibernate works on Macintosh + Windows + Linux, and runs on 10.3 and later. Hibernate + HSQLDB is also pretty performant. On my MacBook Pro, NightHawk loads a 100,000 record contact list in less than a second. That should be fast enough for most people's needs!

Pick Your Database

Hibernate also has another useful (and required) benefit. Since Hibernate generates the SQL for me, and since Hibernate supports a wide range of SQL databases, I am free to try different databases with NightHawk. Although NightHawk is currently configured to use HSQLDB, it is relatively easy for me to use a different database like PostgreSQL or MySQL or Oracle. I'm planning on trying some of the other databases as we implement the pro-level server since they may have better performance characteristics for extremely large databases.

Syncing

When I designed the database schemas for NightHawk, I decided to take a different approach than we did with NUDC. (Sidebar: a database schema is the definition of the fields in a database table, and the relationship between tables. For example, the NightHawk database has tables for contacts, addresses, etc. where each contact can have any number of addresses.) The NUDC databases were designed before vCard and iCalendar formats came on the scene. (In fact, members of the NUDC development team served on the committees that helped design the precursors to those formats.) NightHawk has the advantage of being designed to share data with modern data formats, so the database schemas used in NightHawk are designed to be a superset of the AddressBook and iCal Sync Services schemas (which, in turn, are closely patterned after vCard and iCalendar). The result is that it is easy for NightHawk to sync to AddressBook and iCal and vCard and iCalendar.

As it turns out, it is far harder for NightHawk to read data from Now Contact and Now Up-to-Date files than it is to read data from vCard or iCalendar files. As a result, we only import data from Now Contact and Now Up-to-Date files into NightHawk.

By supporting the industry-standard data formats, there is a huge win for NightHawk:

1. It's easy to sync to AddressBook and iCal
2. It's easy to sync to PDAs and phones supported by SyncServices and other sync standards, such as SyncML
3. We can leverage third party code to read and write vCard and iCalendar format files, so we're more likely to inter-operate with other products.

Database of the Database

One of the problems with NUDC in a group environment was that anyone could change anything. One day you'd have a meeting on your calendar and next day it would be gone because someone decided to delete it. Unlike NUDC, NightHawk contains a full history of every edit made on the data. If someone changes an entry, NightHawk knows who did it, when they did it, and what it used to read. You can then undo specific edits or collections of edits. This should go a long way to pinpointing changes in your calendar or contact file!!

Conclusion

NightHawk's database gives us a solid platform to build the rest of the application. It contains all of your data records, all of your preferences, and manages all of the syncing to the NightHawk server and to your mobile phones and PDAs. At its core, NightHawk contains a fast, flexible and advanced database architecture. And if I do my job right, that database will be humming smoothly and silently in the background. It'll just work.

If you have questions about our database or other parts of NightHawk, write to me at nighthawk@nowsoftware.com.

Thanks!
John

NightHawk Intro

Folks have been asking me: Why do a new calendar & contact program? Why not just do another rev of Now Up-to-Date & Contact? Well, here goes...

About NUDC

Now Up-to-Date & Contact first came into people's lives in the early '90s. Over the last decade-and-and-half, NUDC has grown from System 6 to System 7 to Mac OS 8 to Mac OS 9 to Mac OS X. It's been adapted to Windows 9x/NT/XP. NUDC has its own database code, its own networking code, its own UI management code. It even implements its own web server for its web publishing.

At this point NUDC contains about 1,200,000 lines of code developed by Now Software -> QualComm -> Power On Software -> Now Software engineers. And while the code quality is reasonably good, it only takes a few needles in a 1.2M line haystack to cause problems. Just modifying the code to keep up with OS changes takes a lot of effort. Plus (as you've seen) it is difficult to make sweeping changes to the code, especially graphical user interface (GUI) changes. If we stayed with the current code base we'd have to continue taking small steps. We want more.


NightHawk Overview

Our new software base (codename: "NightHawk") is a complete re-implementation of NUDC from the bottom up. It has a SQL database backend, industry standard networking code, industry standard encryption code, and runs on an Aqua GUI for Mac and a .NET GUI for Windows. NightHawk forms the basis for all of our future development projects. The first product released on the NightHawk code base will be our new integrated calendar and contact manager. This new release implements the current feature set of NUDC plus essential new features. But NightHawk is more than just a replacement for NUDC5. NightHawk is a platform for business management software where we can add integrated support for whole new kinds of functionality.


NightHawk Roadmap

NightHawk is currently in active development. At this time, all of our NUDC engineers have moved over to NightHawk and we'll only be doing further releases under the NUDC5 code base as necessary. NightHawk will go through a series of releases, where our first public beta is scheduled for June 2007. At that point we expect the code to be usable and reasonably solid.


NightHawk Blog

This blog is a place where I and members of my team can talk about NightHawk and the software industry, and where you can give us your feedback and insights. NightHawk is near and dear to my heart. It's been a 100+ hour per week project where I can pursue software done right. I hope you enjoy using it as much as I and my team enjoy developing it!


--John