Categories
awk bash shell

How do I use shell variables in an awk script?

367

I found some ways to pass external shell variables to an awk script, but I’m confused about ' and ".

First, I tried with a shell script:

$ v=123test
$ echo $v
123test
$ echo "$v"
123test

Then tried awk:

$ awk 'BEGIN{print "'$v'"}'
$ 123test
$ awk 'BEGIN{print '"$v"'}'
$ 123

Why is the difference?

Lastly I tried this:

$ awk 'BEGIN{print " '$v' "}'
$  123test
$ awk 'BEGIN{print ' "$v" '}'
awk: cmd. line:1: BEGIN{print
awk: cmd. line:1:             ^ unexpected newline or end of string 

I’m confused about this.

3

  • 2

    I like the -v as shown below, but this is really a great exercise in thinking about how to protect things from the shell. Working through this, my first cut use backslashes on spaces and dollar signs. Needless to say the examples here were well worth my time.

    – Chris

    Dec 20, 2016 at 21:00

  • If your awk search needs regular expression, you can’t put /var/. Instead, use tilde: awk -v var="$var" '$0 ~ var'

    May 7, 2020 at 9:29


605

#Getting shell variables into awk
may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below.                                                                                    v1.5


Using -v (The best way, most portable)

Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)

variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two

This should be compatible with most awk, and the variable is available in the BEGIN block as well:

If you have multiple variables:

awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

Warning. As Ed Morton writes, escape sequences will be interpreted so \t becomes a real tab and not \t if that is what you search for. Can be solved by using ENVIRON[] or access it via ARGV[]

PS If you like three vertical bar as separator |||, it can’t be escaped, so use -F"[|][|][|]"

Example on getting data from a program/function inn to awk (here date is used)

awk -v time="$(date +"%F %H:%M" -d '-1 minute')" 'BEGIN {print time}'

Example of testing the contents of a shell variable as a regexp:

awk -v var="$variable" '$0 ~ var{print "found it"}'

Variable after code block

Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:

variable="line one\nline two"
echo "input data" | awk '{print var}' var="${variable}"
or
awk '{print var}' var="${variable}" file
  • Adding multiple variables:

awk '{print a,b,$0}' a="$var1" b="$var2" file

  • In this way we can also set different Field Separator FS for each file.

awk 'some code' FS=',' file1.txt FS=';' file2.ext

  • Variable after the code block will not work for the BEGIN block:

echo "input data" | awk 'BEGIN {print var}' var="${variable}"


Here-string

Variable can also be added to awk using a here-string from shells that support them (including Bash):

awk '{print $0}' <<< "$variable"
test

This is the same as:

printf '%s' "$variable" | awk '{print $0}'

P.S. this treats the variable as a file input.


ENVIRON input

As TrueY writes, you can use the ENVIRON to print Environment Variables.
Setting a variable before running AWK, you can print it out like this:

X=MyVar
awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'
MyVar /bin/bash

ARGV input

As Steven Penny writes, you can use ARGV to get the data into awk:

v="my data"
awk 'BEGIN {print ARGV[1]}' "$v"
my data

To get the data into the code itself, not just the BEGIN:

v="my data"
echo "test" | awk 'BEGIN{var=ARGV[1];ARGV[1]=""} {print var, $0}' "$v"
my data test

Variable within the code: USE WITH CAUTION

You can use a variable within the awk code, but it’s messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.

This works by extracting the variable within the code, so it becomes a part of it.

If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.

variable="line one\nline two"
awk 'BEGIN {print "'"$variable"'"}'
line one
line two

Here is an example of code injection:

variable="line one\nline two" ; for (i=1;i<=1000;++i) print i""
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
1
2
3
.
.
1000

You can add lots of commands to awk this way. Even make it crash with non valid commands.

One valid use of this approach, though, is when you want to pass a symbol to awk to be applied to some input, e.g. a simple calculator:

$ calc() { awk -v x="$1" -v z="$3" 'BEGIN{ print x '"$2"' z }'; }

$ calc 2.7 '+' 3.4
6.1

$ calc 2.7 '*' 3.4
9.18

There is no way to do that using an awk variable populated with the value of a shell variable, you NEED the shell variable to expand to become part of the text of the awk script before awk interprets it.


Extra info:

Use of double quote

It’s always good to double quote variable "$variable"
If not, multiple lines will be added as a long single line.

Example:

var="Line one
This is line two"

echo $var
Line one This is line two

echo "$var"
Line one
This is line two

Other errors you can get without double quote:

variable="line one\nline two"
awk -v var=$variable 'BEGIN {print var}'
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ backslash not last character on line
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ syntax error

And with single quote, it does not expand the value of the variable:

awk -v var="$variable" 'BEGIN {print var}'
$variable

More info about AWK and variables

Read this faq.

9

  • 4

    “messy and hard to read” ignores the more important security concern of code injection when directly substituting strings into awk code.

    Feb 18, 2016 at 20:21

  • reading the answer above I can run my script without errors but it doesn’t do the job: awk -v repo=”$1″ -v tag=”$2″ ‘{sub(/image: registryabx.azurecr.io\/{print repo}:([a-z0-9]+)$/,”image: registryabc.azurecr. io/{print repo}:{print tag}”);}1’ ./services/appscompose.yaml >> newcompose.yaml. Is because of the nested parenthesis {?

    Oct 29, 2019 at 11:10

  • @DarionBadlydone Try this awk -v repo="$1" -v tag="$2" 'BEGIN {print "repo="repo,"tag="tag}'. It will see if it prints the variable. Post a own question if you can not figure the out.

    – Jotne

    Oct 29, 2019 at 12:13

  • @Jotne yes it print the values so i tried in this way: awk -v repo=”$1″ -v tag=”$2″ ‘{print “{sub(/image: registryabc.azurecr.io/”repo”:([a-z0-9]+)$/,\”image: registryabc.azurecr.io/”repo”:”tag”\”);}1″}’ ./services/appscompose.yaml >> newcompose.yaml but is not working as aspected. It replace each line of the source file with the printed command

    Oct 29, 2019 at 13:38

  • 1

    I strongly disagree that -v is the “best, most portable way”. awk -v a=b cmds path1 path2 is (almost) equivalent to awk cmds a=b path1 path2, but there is no good way to use -v to emulate awk cmds path1 a=b path2 Defining variables in the arguments is an extremely useful technique which is equally portable and which I will argue is “better”.

    Feb 8, 2021 at 21:48

32

It seems that the good-old ENVIRON built-in hash is not mentioned at all. An example of its usage:

$ X=Solaris awk 'BEGIN{print ENVIRON["X"], ENVIRON["TERM"]}'
Solaris rxvt

3

  • 4

    This is a good suggestion because it passes the data verbatim. -v doesn’t work when the value contains backslashes.

    Feb 23, 2016 at 21:45

  • 2

    @thatotherguy I did not know that! I thought that if I use awk -v x='\c\d' ... then it will be used it properly. But when x is printed awk drops the famous: awk: warning: escape sequence '\c' treated as plain 'c' error message… Thanks!

    – TrueY

    Feb 24, 2016 at 9:11


  • 1

    It does work properly – properly in this context means expand escape sequences because that’s how -v was designed to work so you can use \t in the variable and have it match a literal tab in the data, for example. If that’s not the behavior you want then you don’t use -v you use ARGV[] or ENVIRON[].

    – Ed Morton

    Jul 7, 2019 at 15:02

15

Use either of these depending how you want backslashes in the shell variables handled (avar is an awk variable, svar is a shell variable):

awk -v avar="$svar" '... avar ...' file
awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}... avar ...' "$svar" file

See http://cfajohnson.com/shell/cus-faq-2.html#Q24 for details and other options. The first method above is almost always your best option and has the most obvious semantics.