Categories
in-place sed shell text-parsing

How to delete from a text file, all lines that contain a specific string?

2168

How would I use sed to delete all lines in a text file that contain a specific string?

0

    3327

    To remove the line and print the output to standard out:

    sed '/pattern to match/d' ./infile
    

    To directly modify the file – does not work with BSD sed:

    sed -i '/pattern to match/d' ./infile
    

    Same, but for BSD sed (Mac OS X and FreeBSD) – does not work with GNU sed:

    sed -i '' '/pattern to match/d' ./infile
    

    To directly modify the file (and create a backup) – works with BSD and GNU sed:

    sed -i.bak '/pattern to match/d' ./infile
    

    12

    • 16

      Thanks, but it doesn’t seem to erase it from the file but just print out the text file contents without that string.

      Mar 23, 2011 at 20:03

    • 128

      @A Clockwork: yes, you need to redirect the output either to a new file with something like sed '/pattern to match/d' ./infile > ./newfile or if you want to do an in-place edit then you can add the -i flag to sed as in sed -i '/pattern to match/d' ./infile. Note that the -i flag requires GNU sed and is not portable

      – SiegeX

      Mar 23, 2011 at 20:16

    • 16

      For some flavor’s of sed; sed’s “-i” flag required an extension to be provided. (e.g. sed -i.backup '/pattern to match/d' ./infile) That got me across with in-place edits.

      – avelis

      Jan 31, 2013 at 21:45


    • 10

      @SiegeX Better yet, don’t apply commands like sed to any files that aren’t version-controlled.

      – Tyler

      Feb 27, 2013 at 23:55

    • 91

      One more note for Mac OS X users: for some reason, the -i flag requires an argument to be passed, even if it’s just an empty string, like sed -i '' '/pattern/d' ./infile.

      Oct 2, 2013 at 14:55

    700

    There are many other ways to delete lines with specific string besides sed:

    AWK

    awk '!/pattern/' file > temp && mv temp file
    

    Ruby (1.9+)

    ruby -i.bak -ne 'print if not /test/' file
    

    Perl

    perl -ni.bak -e "print unless /pattern/" file
    

    Shell (bash 3.2 and later)

    while read -r line
    do
      [[ ! $line =~ pattern ]] && echo "$line"
    done <file > o
    mv o file
    

    GNU grep

    grep -v "pattern" file > temp && mv temp file
    

    And of course sed (printing the inverse is faster than actual deletion):

    sed -n '/pattern/!p' file
    

    8

    • 4

      how to delete a particular line with a pattern and also the line immediately above it? I have a fine with thousands of such lines in between different data.

      Aug 6, 2013 at 22:41


    • 1

      On OS/X, the shell variation doesn’t preserve leading spaces, but the grep -v variation worked well for me.

      Feb 3, 2014 at 23:31

    • 16

      the sed example have a different behaviour, it only greps! it should be something like sed -n -i '/pattern/!p' file.

      – caesarsol

      Mar 28, 2014 at 16:41

    • 10

      The grep version does not work when every line matches the pattern. Better do: grep -v "pattern" file > temp; mv temp file This might apply to some of the other examples depending on the return value.

      Jun 20, 2014 at 14:43


    • 4

      “printing the inverse is faster than actual deletion” – Not on my machine (2012 MacBook Air, OS X 10.13.2). Create file: seq -f %f 10000000 >foo.txt. sed d: time sed -i '' '/6543210/d' foo.txt real 0m9.294s. sed !p: time sed -i '' -n '/6543210/!p' foo.txt real 0m13.671s. (For smaller files, the difference is larger.)

      Jan 22, 2018 at 2:11

    314

    You can use sed to replace lines in place in a file. However, it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original.

    e.g.

    sed -i '/pattern/d' filename      
    

    or

    grep -v "pattern" filename > filename2; mv filename2 filename
    

    The first command takes 3 times longer on my machine anyway.

    10

    • 28

      Voting up your answer too, just because you tried a performance comparison!

      – anuragw

      Apr 12, 2013 at 7:11

    • 4

      +1 for offering option to overwrite current file with the grep line.

      – Rhyuk

      May 6, 2013 at 20:43

    • 2

      The second ‘grep’ solution is also better for large files

      – simoes

      Jan 2, 2014 at 4:50

    • 3

      I’m curious what the performance difference would be if it were sed '/pattern/d' filename > filename2; mv filename2 filename

      – Pete

      Apr 8, 2014 at 1:00

    • 12

      (using ubuntu’s /usr/share/dict/words) grep and mv: 0.010s | sed in place: 0.197s | sed and mv: 0.031s

      Feb 10, 2015 at 14:22