banner

For a full list of BASHing data blog posts, see the index page.     RSS


How to delete, insert and replace whole lines

The trick to successful operations with whole lines in data tables is the correct use of line addresses. You need to be sure not only that the operation is done where you want it, but also that it isn't done where you don't want it. sed and AWK are the right tools, because with both you can easily specify the line address or addresses for the operation.


Single lines. From the table "lines" (below), I'll delete line 3:

This is line 1.
This is line 2.
This is line 3.
This is line 4.
This is line 5.

sed '3d' lines
awk 'NR!=3' lines

lines1

Inserting and replacing are similar operations, because both involve adding a new line to the table. I'll insert "This is a new line." after line 4...

sed '4a\This is a new line.' lines
awk -v x="This is a new line." 'NR==4 {$0=$0 RS x} 1' lines

lines2

The "append" instruction (a) for sed adds a line just after the line address.
 
In the AWK command, line 4 is redefined to be line 4, a newline (the default record separator, or RS) and the line to be added.

...and replace line 4 with the new line:

sed '4s/.*/This is a new line./' lines
awk -v x="This is a new line." 'NR==4 {$0=x} 1' lines

lines3
 

Multiple lines. Suppose I want to delete lines 1, 3 and 5 from "lines". (Three lines isn't a lot, but this method works for any number of lines.) First I'll create a file called "ops" with a list of those line numbers:

1
3
5

With sed, I generate a virtual file of instructions from "ops", and use sed's "-e" option to make that a list of jobs to do on "lines":

sed -e "$(sed 's/$/d/' ops)" lines

lines4

Something similar works for inserting and replacing. I'll insert "This is a new line." after lines 1, 3 and 5, or replace those lines with the new one:

sed -e "$(sed 's/$/a\This is a new line./' ops)" lines
sed -e "$(sed 's/$/s\/.*\/This is a new line.\//' ops)" lines

lines5

Since I'm using a forward slash as a delimiter in the sed commands, I need to escape the forward slashes in the "statement" s/.*/This is a new line./.

An easy AWK method is to first put the contents of "ops" into an array, then do a deletion, insertion or replacement based on whether or not the current line number is in the array:

awk 'FNR==NR {a[$0]; next} !(FNR in a)' ops lines
 
awk -v x="This is a new line." \
'FNR==NR {a[$0]; next} FNR in a {$0=$0 RS x} 1' ops lines
 
awk -v x="This is a new line." \
'FNR==NR {a[$0]; next} FNR in a {$0=x} 1' ops lines

lines6

FNR is the record number (records are lines by default) of the current file and NR is the total number of records. These will only be the same for the first file processed, so the action in curly brackets (a[$0]; next) only applies to the first file, namely "ops". That action is to build an array "a" indexed with the whole lines in the file, i.e. the listed line numbers. When that's done, next tells AWK to move on to the next file, "lines".
 
If the current line number in "lines" (FNR) is not indexed in "a" (!(FNR in a)), it's printed.
 
If the current line number in "lines" (FNR) is indexed in "a" (FNR in a), AWK does the commanded action, either inserting or replacing. The final "1" is AWK shorthand for "print all lines".


Deleting, inserting or replacing based on a pattern match. This is where operations on whole lines can go wrong, because line addresses aren't used. For example, suppose I want to delete any lines containing the string "tomorrow" in "file". I can use sed or AWK:

sed '/tomorrow/d' file
 
awk '$0 !~ /tomorrow/' file

But what if there were lines in "file" that I didn't want to delete and that I didn't realise contained "tomorrow"? To avoid disaster, I always check with a general grep search to make sure that the only lines to be targeted are the ones containing the pattern to be matched.

grep "tomorrow" file

If some of those lines need to be left alone, I get the line numbers of the ones to be deleted and do the deletion with just those line addresses. It's safer.


Last update: 2019-05-12