Dear TrapNZ,
I have been having coding issues when downloading csvs of projects that have macrons (ā, ē, ī, ō, ū) in them. I have been battling quite a weird one recently where ā reads as 2x unicode characters U+00C4 (Ä) and U+0081 (non-printing C1 control character).
We are dealing with it at the moment but some help in future would be amazing.
Cheers,
Nikita
This also applies to the ‘My Projects’ list which sorts alphabetically by default . I have a project which starts A with a macron but is sorted to the bottom of the list.
It would be great to have this addressed.
It is up to you to handle Unicode files correctly. That means the software that you are using to load the CSV files has to understand it is a Unicode and not just ASCII. For backward compatibility all the ASCII codes for the alphabet are the same in Unicode so you are getting away with it for legacy characters.
Perhaps use Notepad++ to change the data type of the CSV. Excel and most other software supports Unicode natively so I wonder what you are using that does not handle Unicode.
There are multiple Unicode standards. UTF-8 is the simplest but there are others. The sort order is not the same in each system. The convention for Māori macrons is different to the underlying codes if sorted numerically.
Many New Zealand systems that use Māori collation treat:
a < ā < e < ē < i < ī < o < ō < u < ū
so macron vowels are grouped with their corresponding base vowels rather than being pushed after z.
The answer to “where does ā sort relative to ASCII?” is:
a.ā immediately after a).It appears that TrapNZ software uses binary codepoint sorting.
If you’re using a specific platform (JavaScript, Python, SQL Server, PostgreSQL, Excel, .NET, etc.), Each environment sorts mixed ASCII and macron characters differently.
The best way to keep sorting predictable is to avoid unicode characters in sorting fields, particularly codes.