Plain text files in macOS for Nyquist plug-ins in Audacity

Sometimes interesting investigations arise completely unexpectedly! Here's a happenstance that caught me out recently...

(Yes, this screen shot shows the Audacity that I haven't updated in quite a while...)

I have written a few plug-ins for Audacity, the amazing free and open-source cross-platform audio editor software, and have never had any problems . Well, other than trying to get my head around the fascinating programming approach that it uses - an extension of the XLISP dialect of Lisp. For some reason, my brain doesn't seem to be wired to think about things the way that Lisp does, and this makes developing plug-ins quite a challenge! But I'm not used to problems with the way that the plug-ins interact with the host application (Audacity in this case)...

It happened when I was adding some functionality to an existing plug-in, the RUSS Harmonic Rejector, to cope with some unusual audio signals that I was processing. I copied across the Nyqvist text files to the appropriate 'Plug-Ins' folder on my MacBook Pro, edited the Nyqvist plug-in code, saved it, and went into Audacity to load it via the 'Effect>Add/Load Plug-Ins' menu option - and this is where things got interesting.

Here's what I saw in the dialogue box that opened:

Now those extra characters aren't there in the Nyqvist code:

;nyquist plug-in
;version 1
;type process
;name “a_RUSS_Harmonic_Filter_lp2"

Also, it turns out that in the Audacity support pages for Nyquist, there's a warning box that I hadn't paid much attention to...

Now that I read that warning again, it seems to be quite important!

But what is the warning actually saying? It says to use a plain text editor rather than a word processor, and I was using TextWrangler as my editor, and that is a plain text editor, isn't it?

It is at this point that the 'question everything, trust nothing' big red warning light starts flashing in my head, and I realise that I'm guilty of 'making assumptions'. Yep, an unquestioned, obvious assumption that you don't realise you are making until it trips you up... So I looked at the files in the Finder:

Now, the Finder is designed for 'finding' files, and it isn't particularly good at telling you much about the files themselves. There's 'Get Info...' for that, and it told me that I had two 'documents' and a document that was last saved as a 'SimpleText' document, which used to be the built-in editor for MacOS from System 7 to System 9 (I think). Unfortunately, a description like 'SimpleText' document doesn't directly indicate if it is a plain ASCII text file. ASCII, by the way, is one of the early ways of mapping numbers to keyboard characters, numbers and other functions, and so a value of 65 represents the character 'A', 49 is '1', and 10 is 'Line Feed (LF)', which is one of several characters that can indicate a new line (Carriage Return (CR) is another), and 7 is the 'Bell' character, which makes a noise (and was originally an actual physical bell!).

Anyway, I opened the Harmonic Extractor file in TextWrangler (my current but 'about-to-be-outdated' 32 bit text editor), and looked at the 'info' bar on the lower edge of the window:

Aha! This isn't a plain ASCII text file! Somewhere along the line, it has been converted to Unicode (UTF-8) encoding instead of ASCII (maybe when it was stored on Google Drive?). So what about the SimpleText document? TextWrangler said:

Now this says it is a text file, but there's an interesting 'Mac OS Roman' encoding, and it has Windows 'end of line' Carriage Return, Line Feed characters, so my tendency to use whatever text editor is around at the time is now biting me back!

Here's what the info bar should say for the 'Plain text' that the Audacity warning box was hinting at:

This is a text file, with ASCII encoding of the symbols on the keys, and with a Unix-style 'Line Feed' to indicate the end of each line. So I tried to save the Nyqvist files as Plain ASCII text files using these settings and up popped a warning that some Unicode characters could not be saved. It turns out that these were the 'smart' doubles quotes around the "title". So I took these out and replaced them with the normal less curvy double quotes and re-saved. All was well again! The plug-ins now had the correct names and were in the correct alphabetically sorted position in the menu. I also tried saving the file using the Windows 'End of Line' convention (CRLF = a Carriage Return character, followed by a Line Feed character), and that worked fine as well.

Solved

In summary:

1. You need to make very sure that your Nyquist files are 'plain ASCII text'!
2. 'Plain ASCII text' requires that your editor be set to save using the 'Western (ASCII)' encoding, and ideally to not add a hidden '.txt' extension to the end.
3. Your Nyquist files must have the '.ny' suffix (Remember that the default on macOS is to hide extensions, so use 'Get Info' to make sure that there isn't a hidden '.txt' suffix at the end of the '.ny' file.)
4. Remember that the Mac's 'Textedit' utility makes it very easy to save files as '.rtf' files, and these are NOT plain ASCII text, and just adding '.ny' at the end does not guarantee that there isn't a hidden '.txt' suffix as well...

If this was a forum, then there would be a 'SOLVED' indicator somewhere around here!

There are several things to note here. The first is that I should have thought more about how to edit a file, and especially how to save it - I'm fully guilty of this, and probably rely far too much on default settings to be appropriate to what I'm doing. The second is that Apple's 'it just works' approach sometimes works against you - in this case, setting up TextWrangler so that it saved plain ASCII text was reasonably simple, but only when you knew what was required (and even then, it is easy to end up with a hidden '.txt' extension suffix on the end of what appears to be a '.ny' file. The third is that the Audacity documentation indicated that there might be something that you should be aware of abut editing Nyqvist files, but it didn't give any explicit details, and so required more investigation. The fourth is that just adding '.ny' to the end of a filename does not make that 'extension' disappear, even if  the macOS 'Hide Extensions' default is active - but that Audacity will still accept the '.ny' as an extension even if the Mac's Finder doesn't.

What follows are some of the mitigations that can be used to try and ensure that files are created, edited and saved as plain ASCII text files, in a world where such files seem to be increasingly uncommon.

Utilities

1. TextUtil

There is a utility that can be accessed via the macOS Terminal called textutil that can convert files to plain ASCII text, and remove smart quotes. Here's a template command line for use in Terminal on a file called 'rich text.rtf' that will convert it to a plain text file called 'rich text.txt':

   >textutil richtext.rtf -convert txt

Note that there's now a '.txt' extension suffix at the end of the file. Audacity is looking for '.ny' files, so you need to edit the filename...

2. Shift + Command + T in TextEdit

Also, inside TextEdit, you can use the command: 'Shift + Command + T' to change from Rich Text format to plain text. When you do this, then the 'ribbon' of buttons at the top of the window disappears and the font changes to Courier:

Notice that the double quotes are vertical in the above screen shots...

If you get a double quote that is non-vertical, then it is a smart quote, and could cause problems if you require plain ASCII text - as Audacity does for Nyqvist files:

To make it easier to see the curved 'smart' and vertical 'non-smart', then here are some screenshots with larger text:

(Notice that the 'ribbon' of buttons and the ruler vanish when you are in the 'Plain text' mode.)

To prevent any problems, then you need to turn off the 'Smart Quotes' option in the Text Edit preferences:

You also need to add 'Western ASCII' to the encodings for saving - you do this via the 'Plain Text Encoding' selector in the 'Save' dialogue box:

And then add 'Western (ASCII):

It also seems to be good to start typing in Plain text mode to avoid any smart quotes! As you can see, it isn't entirely straightforward to adjust textEdit so that it saves files as plain ASCII text! 

3. Coteditor

Despite quite a bit of searching, I didn't find a dedicated 'plain ASCII text' editor for MacOS, and unfortunately, the free and very capable generic editor that I've been using for many years, TextWrangler from Bare Bones Software, is a 32-bit applications, and so whilst it works fine in the El Capitan version of macOS that I use on this MaxBook Pro, the future is 64-bit, and so I'm going to need to change to another editor eventually. BBEdit from Bare Bones Software would be the obvious choice, but I wondered if there was an alternative simple text editor...

In my searching I found Sublime Text, BBEdit, Atom, Vim, Brackets (from Adobe), UltraEdit, Coda and Microsoft's Visual Studio Code. Most of these are serious, professional editors, and some of them have free versions (or trials), but none of them seemed to be at the simpler end of the range. However, I did find Coteditor (from Coteditor.com) that seems to be quite simple and straightforward, and it puts the save format right at the top of the main window, where even I would probably eventually notice it!   

4. Get Info

I mentioned using the Command-I 'Get Info' utility before, so here's what it shows when you look at some edited Nyqvist files:

It turns out that the 'Get Info' utility can be very useful! The 'Hide Extension' option is the default in macOS, and so even though the nyquist files have '.ny' at the end, the '1-...' test file also has a '.txt' at the end, which means that Audacity does not include it in the 'Add/remove Plug-Ins' dialogue box. The 'Name & Extension' field can be used to remove the '.txt' suffix - just click in the field and remove the '.txt', then press 'Return'.

Let's recap that:

The '1-RUSS_Harmonic_Extractor_hi2.ny' file the you see in the Finder in macOS is actually called: '1-RUSS_Harmonic_Extractor_hi2.ny.txt' (yes, with two 'extension' suffixes), because macOS hides extensions by default. The '.ny' that looks like an extension is misleading, because the '.txt' suffix that follows is the actual extension.

If you use the 'Get Info' utility to remove the '.txt' at the end of the filename, then you get a file which looks like it has a real extension of '.ny'. The 'a-RUSS_Harm_Extract_hi2.ny' file does not have '.txt' at the end, but it appears in the Finder as 'a-RUSS_Harm_Extract_hi2.ny', which means that the '.ny' isn't a real extension, because if it was, then macOS would not show it, and the file would appear as  'a-RUSS_Harm_Extract_hi2'... However, Audacity treats the 'a-RUSS_Harm_Extract_hi2.ny' file as if it had an extension of '.ny', and thus displays it in the 'Add/Remove Plug-Ins' dialogue box.   

Conclusion

Making sure that Nyqvist files are plain ASCII text files is very easy to get wrong, and if you do get it wrong, then your plug-in may not appear in the 'Add/Remove Plug-Ins' dialogue box, or Audacity may not display the name of the plug-in correctly. Having spent a lot of time in the integrated development environment that Max offers (for MaxForLive plug-ins, as well as Max code), then it is very interesting to see how a non-integrated development system requires more thought and organisation.

I hope that this blog post will help anyone else who experiences problems with Nyqvist files in Audacity on macOS, assuming that they can find this blog post, of course!

Buy me a coffee