BASHing data 2

Replace the last N occurrences of a pattern in a string

Fri, 03 Jan 2025 00:00:00 GMT

Drive sed with a for loop

A Unicode normalisation problem

Fri, 10 Jan 2025 00:00:00 GMT

How to get rid of full-width characters

MAD about the median

Fri, 17 Jan 2025 00:00:00 GMT

That's Median Absolute Deviation, a useful statistic

Adding the missing keys and values in a key-value series

Fri, 24 Jan 2025 00:00:00 GMT

Easily done with an AWK array

The ìèñëèâñüêå mystery

Fri, 31 Jan 2025 00:00:00 GMT

The killer was... the Microsoft Corporation

AWK's view of existence

Fri, 07 Feb 2025 00:00:00 GMT

AWK — by which I mean GNU AWK, or "gawk" — sometimes doesn't distinguish between a zero and an empty string. I blogged about this peculiarity in 2020 but this post dives a bit deeper.

Four exercises with data art

Fri, 14 Feb 2025 00:00:00 GMT

Colorful fun with PPMs

Extract the year from a date string without using the date command

Fri, 21 Feb 2025 00:00:00 GMT

Demonstrating 2 methods and 4 variations for each

Does string A contain string B? Ask AWK's index

Fri, 28 Feb 2025 00:00:00 GMT

Demonstrating a handy use for the index function

Permutations and combinations of pairs with AWK

Fri, 07 Mar 2025 00:00:00 GMT

Easy ways to get results with and without repetition

Find all data points "X" km or less from a given point

Fri, 14 Mar 2025 00:00:00 GMT

I have a table with about 10000 records of millipede occurrences in Tasmania. Almost every record has the latitude and longitude of the point where a particular species of millipede was recorded, although in a few cases the lat/lon entries are blank (inadequately georeferenced occurrences, like locality = "Tasmania"). Occasionally I need to identify any records that are 5 (or 10) km or less from a particular point in the Tasmanian landscape. Previously, finding those records meant importing the latest occurrences table into a GIS program, adding the point of interest, buffering the point with a 5 (or 10) km circle, then intersecting the records layer with the circle; i.e. "cookie-cutting" out the millipede records found within the circle. As a final step I'd export the cookie-cut records to a new text file. To save time and avoid opening a GUI GIS program I wrote a simple shell script ("circle") with a YAD entry dialog. This post explains how the script works.

New code for my translation box

Fri, 21 Mar 2025 00:00:00 GMT

Back in 2017 I blogged about a script for quick-and-easy translation. I highlighted a block of non-English text in an application window (for example, on a webpage in my browser) and launched the script with a keyboard shortcut. A YAD dialog popped up next to the application, and a second or two later an English translation appeared in the YAD window. The YAD dialog had automatically gained focus and I could dismiss it by pressing Esc. I've used the script a lot over the past 8 years, but YAD has changed a little and I wanted to have both the original and the translated text together in the YAD window for copying. The revised script shown here does that and has another useful feature.

Text processing with xargs and jot

Fri, 28 Mar 2025 00:00:00 GMT

The xargs and jot commands are mainly useful for jobs other than manipulating text. I'll feed a list of files to xargs to get another command to do something with each of the files, and I'll call on jot to generate some random numbers. Nevertheless, just as GNU datamash can do text processing as well as descriptive statistics, both xargs and jot have niche uses for the handling of plain text, as shown in this post.

Rename time-series files for chronological sorting

Fri, 04 Apr 2025 00:00:00 GMT

What I call "time-series files" are documents like monthly financial statements, monthly newsletters and quarterly reports. I don't modify them after I receive them, and they get filed and archived in an appropriate folder, like "[Organisation name]-newletters". Often the time-series file doesn't have an informative name. My last quarterly water bill from TasWater, for example, arrived as "document15176075861143268989.pdf" (I am not making this up). Even when the filename is informative, the date component might be less than obvious: "Smalltown Tennis Club Dec 24 news.pdf". I've recently gotten into the habit of renaming time-series files to make them chronologically sortable by filename, as in "2024-12-18-TasWater-bill". I wanted to do the same for the contents of folders-ful of legacy documents, so I turned to the command line and "date -r ', which returns the last-modified date of a file or folder.

A launcher for occasionally used applications

Fri, 11 Apr 2025 00:00:00 GMT

There are no launch icons on my desktop because I prefer using keyboard shortcuts (memorised) to launch frequently used applications and scripts. To start applications that I use rarely, I turn to the Xfce applications menu, which I open with the Super (Windows) key or by right-clicking on the desktop. There I navigate through lists and sub-lists to find the application I want. For a half-dozen occasionally used applications I have a mouse-less alternative. It's the script "occapps" (shown here) and it's launched from the keyboard. Before writing the script I looked at the many existing application launchers for Linux. Most of them have a search box in which you type the first few letters of the application name. That doesn't seem efficient in my case, because I'm only interested in a few applications that aren't already tied to keyboard shortcuts. The "occapps" script is very simple and easily editable if I need to change an application or command.

Extreme reformatting: a vertical calendar

Fri, 18 Apr 2025 00:00:00 GMT

I've used the "cal" program for many years, especially with the "-3" option, which returns the current month and the two on either side. I now use a homemade vertical calendar built from "ncal", with the current date highlighted.

How to add trailing spaces and zeroes

Fri, 25 Apr 2025 00:00:00 GMT

It's pretty easy to add leading spaces or zeroes with printf, since leading spaces and zeroes are a kind of right-justifying. How about left-justifying? Can you add trailing spaces and zeroes just as easily? Spaces yes, zeroes no.

csvlens: a delimited text file viewer for the terminal

Fri, 02 May 2025 00:00:00 GMT

I downloaded the latest csvlens (v.0.11.0) from its GitHub site as a ready-to-use, 8.7 MB executable and tested it on CSVs (comma and semicolon separated), TSVs and PSVs. TL, DR: it works very well as a viewer!

Two more tweaks for the ranger file manager

Fri, 09 May 2025 00:00:00 GMT

Wrap text in preview, and improve the default colors

Multiple-line records to a simple table

Fri, 16 May 2025 00:00:00 GMT

The data file was irregular, with multiple-line records and variable numbers of lines within records. A single AWK command tidied it up nicely and reformatted the file as an easily parse-able table

How to ignore everything but numbers

Fri, 23 May 2025 00:00:00 GMT

So far as AWK is concerned, arithmetic is done with the numbers it first sees, and everything else is just noise to be ignored

What a long, strange trip it's been

Fri, 30 May 2025 00:00:00 GMT

When I wrote my first code, computer users were computer programmers

Making an archive job a lot easier

Fri, 06 Jun 2025 00:00:00 GMT

Selectively unzip and rename, all on the command line

How to hide a number in plain sight

Fri, 13 Jun 2025 00:00:00 GMT

A simple cryptographic trick

Beware these characters in a terminal

Fri, 20 Jun 2025 00:00:00 GMT

I haven't tried every modern terminal emulator, but the ones I've looked at will all interpret a C1 control character as though they were ANSI-standard teletype machines. This is really annoying behaviour when working with text files in a terminal.

Five ways to pass a shell variable to AWK

Fri, 27 Jun 2025 00:00:00 GMT

My shell scripts often contain a (GNU) AWK command, and that command often references a shell variable defined earlier in the script. My usual way of passing the shell variable to AWK is to define an AWK variable with it. There are several other ways, though.

Data entry with unknown data categories

Fri, 04 Jul 2025 00:00:00 GMT

Both Microsoft Excel and Google Sheets allow users to write data entry forms. The forms aren't very easy to create, but they help data workers to avoid entry errors when compiling data in spreadsheets. However, data entry forms require that the structure of the records is already known. What if you don't know in advance all the possible "columns" you might need?

The script command for tinkerers

Fri, 11 Jul 2025 00:00:00 GMT

I like to tinker on the command line, and if something interesting (or bad!) happens I like to review what I've been doing. One way to do that is to go back through my BASH history, but if I've been using other commands in the meantime, the relevant bits of the history might be scattered and hard to find. For this reason I use the "script" command to record tinkering sessions.

Format musings 1: NestedText and indentation

Fri, 18 Jul 2025 00:00:00 GMT

"NestedText" is a relatively new format (2020 >) for sharing data between applications as uncompressed strings. Strings are its only data types and the "NestedText" rules allow for special characters (like "{") and syntactical elements (like "") to be parsed as literal strings. The "NestedText" documentation is very good and the philosophy behind the format is clearly explained. Interested readers should look at the "NestedText" rules but what I'd like to muse about in this post is the "Nested" in "NestedText". A key element in "NestedText" structuring is indentation, and relying on indentation can be risky.

GNU sed's handy -z option

Fri, 25 Jul 2025 00:00:00 GMT

In 2012 the GNU sed developers added a new option, -z. Its effect is to replace newlines in the stream fed to sed with NULL characters. That might sound a little ho-hum, so I'll rephrase it this way: the -z option converts the input into a single long line while sed works on it. This means you can take advantage of sed's "numbered occurrence" feature to do replacements on whole files.

zet for sets

Fri, 01 Aug 2025 00:00:00 GMT

This post demonstrates the "zet" utility for doing command-line set operations. It does unions, intersects etc on multiple files, all with the same simple syntax and without changing line order.

Data noise

Fri, 08 Aug 2025 00:00:00 GMT

All the datasets I audit are UTF-8 encoded and tab-separated text files. They contain letters, numbers, punctuation and spaces, and they're correctly separated into records and fields. They're what I call "tidy", but many of them still aren't "clean" because they contain unnecessary data noise. This post describes five kinds of data noise and how I deal with it.

Square root days, prime years and maximum-factor years

Fri, 15 Aug 2025 00:00:00 GMT

Square Root Day happens nine times a century. It's the day on which the square root of the last two digits in the year is both the month number and the day number, and this year it was 5 May (2025-05-05). It's easy to generate a list of Square Root Days in your head, but I wondered whether there might be other numerical date questions best answered with a bit of coding, like "Which of the year numbers are prime?"

Converting lat/lon from DMS to DD without screaming

Fri, 22 Aug 2025 00:00:00 GMT

Some of the datasets I audit have coordinates in two formats, namely degree-minute-second (DMS) and decimal degrees (DD). The DMS data are the originals in a single data field, and the DD data are split in the dataset between decimal latitude and decimal longitude fields.Checking that DMS-to-DD was done correctly could be easy but usually isn't. The problem is that the DMS might be in any one of many formats, and working with all the possible variations is enough to give you the screaming meemies. What's needed on the command line is a magic program that picks out just the numbers and the directions in the DMS and ignores punctuation and spaces. That magic program is AWK.

Filling down blanks in multiple fields

Fri, 29 Aug 2025 00:00:00 GMT

A spreadsheet can't do this in one pass, but AWK can

5, 7, 8, 9, 10, 12, 14, 15, 17. Any advance on 17?

Fri, 05 Sep 2025 00:00:00 GMT

Nine websites try to identify the most common data problems, up to 17 of them. I reckon I could do better than 17 data problems, but I'm suspicious of headlines with numbers. They remind me of the Confucian "Four Cardinal Principles and Eight Virtues" and all those silly "Top Ten" lists. Nevertheless, I'll go numerological here and ask: Are there 3 data quality issues that these 9 websites all agree are troublesome?

Change of habit: Geany out, Mousepad in

Fri, 12 Sep 2025 00:00:00 GMT

Geany was my everyday text editor for many years. When I first started with Geany, I saw it as an upgrade from Gedit, which came with the Gnome desktop environment. Like many other disappointed users, I abandoned Gnome ca 2011 (Gnome 3) for Xfce. Geany seemed faster than Gedit, had lots of programming aids and was good at loading and editing very large text files. Meanwhile the developers of Mousepad, the default editor for Xfce, had been quietly adding features and fixing bugs. Mousepad now has what I need and I've changed from Geany for plain-text work, including coding.

Another tricky formatting problem

Fri, 19 Sep 2025 00:00:00 GMT

How to merge lines, differently

My shell and my browser don't understand each other

Fri, 26 Sep 2025 00:00:00 GMT

A small annoyance in writing this blog is that some characters in commands need to be replaced with other characters when representing them in HTML. I don't trust browsers to correctly display non-ASCII characters, even though I have "meta charset="utf-8"" in the webpage header. For that reason I always write "special" characters in webpage code as their HTML named-entity equivalent. The annoyance goes both ways, too. If I take webpage text (from the webpage code) and paste it into my terminal, the shell complains. So both my shell and my browser understand those exceptional characters, but their spelling is a bit different. That raises an interesting question: is there a shared spelling? In other words, is there a single way to represent those characters that both shell and browser understand?

Too many keyboard shortcuts to remember easily?

Fri, 03 Oct 2025 00:00:00 GMT

Being more of a tap-tap person than a clicky-GUI one, I have a lot of keyboard shortcuts in my head. At last count I had 21 custom shortcuts, plus a few of the many that came with my Xfce desktop, plus within-application shortcuts like "Ctrl+k", which toggles the spellchecker in Mousepad text editor. Most of the time I can remember these, but my senior-age memory isn't as good as my junior-age one used to be. What I needed was a GUI pop-up display with a list of regularly used keyboard shortcuts. The display could be launched with a keyboard shortcut, of course, sort of like a master password to open a file of all my passwords. After reviewing some of the options for GUI pop-ups I settled on Timothy Musson's "gxmessage", which is a GNU utility in the repos and is based on GTK2.

Four ways to prepend a line of text

Fri, 10 Oct 2025 00:00:00 GMT

In BASH, it's easy to add a line or lines to the bottom of a text file, because BASH has an "append" operator, ">>". It's not so easy to add a line or lines to the top of a text file, because there isn't a "prepend" operator. "Not so easy" doesn't mean "horribly difficult", though, and there are some simple ways to do this job.

Mojibake detective: the case of the Greek claw

Fri, 17 Oct 2025 00:00:00 GMT

There's something a little strange about this mojibake, and Microsoft is again the chief suspect in the character murder

The difference between two dates: easy solutions and hard

Fri, 24 Oct 2025 00:00:00 GMT

On the BASH command line, getting the duration between two dates is straightforward (sort of) using the "date" command. It can be made a lot easier with a function.

A data table full of ghosts

Fri, 31 Oct 2025 00:00:00 GMT

It's a coincidence that this post was uploaded on Halloween and that the subject is ghosts. By "ghosts" I mean invisible characters (whitespaces) representing visible ones that have been deleted. In a recent real-world case, the author of a data table had replaced ca 150,000 "NA" null values in the table's numerical fields with blanks. The "blanks" weren't empty strings, though. Instead the author had replaced each NA with a single whitespace. The NAs had disappeared, but their replacements were different from the genuine blanks in non-numerical fields in the table. The whitespaces were the "ghosts" of the NAs.

Not on the keyboard. How to type it?

Fri, 07 Nov 2025 00:00:00 GMT

Unless you've done some clever keymapping or have a most unusual keyboard, you can't type superscript or subscript characters, or the degree symbol. That's annoying if you work in STEM.

Format musings 2: CSV vs KVR (key-values in rows)

Fri, 14 Nov 2025 00:00:00 GMT

Everyone knows the CSV file format, except they probably don't. The field separator has to be a comma? Nope. So-called ".csv" files often have a colon, semicolon or tab as field separator. The first line needs to have fieldnames? Nope. That's optional, according to the relevant RFC spec. Details aside and ignoring the world of Big Data, CSV is probably the world's most widely used file format for sharing data tables. It has some annoyances, though, of which the biggest (in my mind) is that commas and quotes within data items need to be escaped. Another issue is that the field to which a data item belongs may not be obvious when you're looking at line 5819 in a ten-thousand-row, 20-column CSV. A CSV alternative is what I call a "KVR" (key-values in rows) file, in which each row has a horizontal list of key-value pairs. There's no need to escape characters within data items and each value is preceded by its key, so you always know what each value means.

THE escape character, not AN escape character

Fri, 21 Nov 2025 00:00:00 GMT

An escape character is any character that tells a computer program or communications protocol that the next character or characters have special meaning. The escape character plus those next meaningful characters are together called an "escape sequence". On the command line, for example, the escape character "\" followed by the "n" character means "newline". There's another escape character, though, namely 00011011 as an 8-bit binary number, 27 as a decimal, 1b as a hexadecimal and 33 as an octal. This character is part of the basic ASCII character set and has been absorbed into Unicode. It's used in escape sequences, like the ones that colorise text. How do you enter it?

Similar to "most distant pair of points", sort of

Fri, 28 Nov 2025 00:00:00 GMT

I had a long, sorted, tab-separated list of spatially clustered sampling sites and their latitudes and longitudes, and I wanted to check that none of the distances within a cluster was larger than a few hundred metres. My solution used everyday shell tools and AWK, not GIS.

Where there's a shell, there's a (usually simpler) way

Fri, 05 Dec 2025 00:00:00 GMT

Something I've learned from the wonderful "Rosetta Code website" is that there are lots of different ways to code solutions to a problem. They all work, and which one you'd use depends more on your programming language preferences and requirements than on the shortness of the solution, or its ability to be generalised as an algorithm.

Serial numbering based on changing values in another field

Fri, 12 Dec 2025 00:00:00 GMT

My problem was to make up a special, sortable, unique ID field for a table that was based on values in a couple of existing fields. I couldn't do this by hand in the real-world case because there were way too many records. But on the command line the job was simple.

Copy selected items from a terminal to a text file

Fri, 19 Dec 2025 00:00:00 GMT

Sometimes when I'm looking through a list in my terminal I see data items that I'd like to follow up in some way. To build a small text file with those selected items as a list I have a tiny BASH script based on the "xclip" utility.

Format musings 3; the last "BASHing data" post

Fri, 26 Dec 2025 00:00:00 GMT

This last "BASHing data" post looks at lightweight markup languages (LMLs). There are more than 20 LMLs, and they all do the same job, which is to annotate plain text with characters already on your keyboard. The annotations tell some other program how the plain text should be formatted.