Page 1 of 1

.PW* file format inconsistencies

Posted: Fri Apr 04, 2008 5:58 pm
by andrixnet

As I was trying to write a style file for GPSBabel's xcsv generic interpreter, I hit several really bad problems, some which may have to do with the design of the file format or the implementation, and a bit about the documentation as well, hope my findings help.

I was using the information you gave me at ... Format.htm as reference for the .pw* file formats.
I had a point database in a .pwpts file and I wanted to convert it using GPSBabel.
First I'd like to point out in your the documentation some details :

Database Record Structure:

- Records are Comma Delimited text as follows:

AFAIK, the terminology commonly used is rather the following:

Fields - delimited by COMMA
Records - delimited by CRNEWLINE (DOS/Windows \r\n)
Records : one record contains one or more fields and is equivalent to a table row, while a field is a table cell or table column on the current row.
(ok, your terminology confused me for a moment)
Documentation does not specify which fields are enclosed in DOUBLEQUOTEs.

Latitude,Longitude Field:
- Stored in decimal degrees (hdd.ddddd) or degrees and decimal minutes (hdd mm.mmmm).
- d=degrees, m=minutes
- Positive latitudes are Northern hemisphere, positive longitudes are Eastern hemisphere.

The "h" character is not explained as "d" and "m" are. Intuitively I guessed it must be the sign indicating the hemisphere.
No rule is given as to which format (hdd.ddddd or hdd mm.mmmm) is used.
Personally, when I exported from Pathaway the points database, in prefs I had "Primary coordinates = DD MM SS.SS"
In the resulting file however, the hdd.ddddd format was used.

Time Field:
- Time format is<space>yyyymmdd.
The date may be missing in older databases.
Time field may be omitted. Esp. for waypoint files.

I did meet a small problem because strptime does not know of miliseconds, but this is a problem with GPSBabel.
Indeed, the date in points saved by an older version only have the "time" information saved, and my file contains a mix of both.
However, my file was a Point database in Pathaway and it does contain the time field. Did you mean "route point" by "waypoint" here? (I haven't tested this yet)
Records that contain time information with both time & date seem to contain an arbitrary number of spaces after date. (1, 2, 4, 7 ...)
After further testing, the number of spaces added at the end of this field is related to the number of times the point was edited inside Pathaway.

Name Field:
- Can be up to 32 characters.
- Any Ascii value except comma is allowed.

Documentation does not say that this field is enclosed in DOUBLEQUOTEs and weather the DOUBLEQUOTE (") character is allowed.
Testing, I found out that DOUBLEQUOTE written in Pathaway point name is NOT exported, not even escaped, it is discarded.
This also happens in Pathaway's the internal .PDB format
Which is good, however I think it should be mentioned in the docs.
Testing, I found that COMMA written in Pathaway point name is REPLACED by a SPACE. (both internally and in the exported file)
Pathaway 4 (PALMOS) User manual :
Name: Name for the point. Maximum length is 79 characters. All characters
except (commas and quotes) are valid.
Here are more inconsistencies.
Pathaway 4, by the manual, supports 79 characters for the point name. The corresponding field in the .PW* file only supports 32? Why?
Pathaway 4 manual says COMMA and QUOTEs (does not specify which ones) are valid. It does not say what will Pathaway do should the user actually type such a character.
In the .PW* file, COMMA is not allowed, no mention about QUOTEs
By trial and error :
COMMA typed in Pathaway is replaced by SPACE
DOUBLEQUOTE typed in Pathaway is discarded.

Note Field:
- Can be up to 15000 characters.
- Any Ascii value except comma is allowed, unless contained inside double quotes.
- For PathAway 4 and higher, the Note may also contain XML tags which represent the sound and picture attachments to the point record. The xml is appended to the end of the description and prefixed with "?xml?".

Pathaway 4 (PALMOS) User manual :
Note: Describe the point. Maximum length is 4000 characters.
Again, inconsistent. However, this time it's better, because the file format supports more characters then the program.
I assume that if a note is longer the 4000 characters (maybe related to PALMOS clipboard size?), on import, it is simply truncated?
No mention in the Pathaway Manual about forbidden characters, but by testing, I found that the behaviour is in fact similar with the Name field.
A DOUBLEQUOTE is discarded by Pathaway
A COMMA is replaced by Pathaway with a SPACE
A COMMA inside DOUBLEQUOTEs is in fact discarded by Pathaway, together with the rest of the text in that field
(I am talking here about how Pathaway behaves when editing points, not importing data from a .PW* file)
I assume the XML bit may be related to the Windows Mobile version? (because the Palm version does not generate any such tags) But won't this be a source for incompatibility between the two? Also, since there already are dedicated fields for this (sound, picture), what's the use for XML tags here?
This field is also enclosed by DOUBLEQUOTEs in the file.
The BIG problem :
Pathaway does NOT escape in any way CR and/or NEWLINE in this field (they are allowed) when exporting to a .PW* file.
This in fact breaks the usability of this file format which by definition, has one record per line.

Extended Format ID:
- Version ID of extended delimited format

No explanation is given as to what this field stands for.
In the file, at least for Pathaway 4.00.80/PALM, seem to be always 1

Picture File attachment:
- Filename of Picture file

Such a picture attachement is exported to a .PW* file as follows:
Please note the underlined part. What is the meaning of those characters after the filename? (Other then the fact that Pathaway is unable to display any attached picture from the card)

I've sent this info also as a sort of bug-report via email.