Categories
bash quoting sh shell

I just assigned a variable, but echo $variable shows something else

140

Here are a series of cases where echo $var can show a different value than what was just assigned. This happens regardless of whether the assigned value was “double quoted”, ‘single quoted’ or unquoted.

How do I get the shell to set my variable correctly?

Asterisks

The expected output is /* Foobar is free software */, but instead I get a list of filenames:

$ var="/* Foobar is free software */"
$ echo $var 
/bin /boot /dev /etc /home /initrd.img /lib /lib64 /media /mnt /opt /proc ...

Square brackets

The expected value is [a-z], but sometimes I get a single letter instead!

$ var=[a-z]
$ echo $var
c

Line feeds (newlines)

The expected value is a a list of separate lines, but instead all the values are on one line!

$ cat file
foo
bar
baz

$ var=$(cat file)
$ echo $var
foo bar baz

Multiple spaces

I expected a carefully aligned table header, but instead multiple spaces either disappear or are collapsed into one!

$ var="       title     |    count"
$ echo $var
title | count

Tabs

I expected two tab separated values, but instead I get two space separated values!

$ var=$'key\tvalue'
$ echo $var
key value

4

178

In all of the cases above, the variable is correctly set, but not correctly read! The right way is to use double quotes when referencing:

echo "$var"

This gives the expected value in all the examples given. Always quote variable references!


Why?

When a variable is unquoted, it will:

  1. Undergo field splitting where the value is split into multiple words on whitespace (by default):

    Before: /* Foobar is free software */

    After: /*, Foobar, is, free, software, */

  2. Each of these words will undergo pathname expansion, where patterns are expanded into matching files:

    Before: /*

    After: /bin, /boot, /dev, /etc, /home, …

  3. Finally, all the arguments are passed to echo, which writes them out separated by single spaces, giving

    /bin /boot /dev /etc /home Foobar is free software Desktop/ Downloads/
    

    instead of the variable’s value.

When the variable is quoted it will:

  1. Be substituted for its value.
  2. There is no step 2.

This is why you should always quote all variable references, unless you specifically require word splitting and pathname expansion. Tools like shellcheck are there to help, and will warn about missing quotes in all the cases above.

2

  • it’s not always working. I can give an example: paste.ubuntu.com/p/8RjR6CS668

    – recolic

    May 5, 2019 at 3:08

  • 1

    Yup, $(..) strips trailing linefeeds. You can use var=$(cat file; printf x); var="${var%x}" to work around it.

    Jun 6, 2019 at 19:17

24

You may want to know why this is happening. Together with the great explanation by that other guy, find a reference of Why does my shell script choke on whitespace or other special characters? written by Gilles in Unix & Linux:

Why do I need to write "$foo"? What happens without the quotes?

$foo does not mean “take the value of the variable foo”. It means
something much more complex:

  • First, take the value of the variable.
  • Field splitting: treat that value as a whitespace-separated list of fields, and build the resulting list. For example, if the variable
    contains foo * bar ​ then the result of this step is the 3-element
    list foo, *, bar.
  • Filename generation: treat each field as a glob, i.e. as a wildcard pattern, and replace it by the list of file names that match this
    pattern. If the pattern doesn’t match any files, it is left
    unmodified. In our example, this results in the list containing foo,
    following by the list of files in the current directory, and finally
    bar. If the current directory is empty, the result is foo, *,
    bar.

Note that the result is a list of strings. There are two contexts in
shell syntax: list context and string context. Field splitting and
filename generation only happen in list context, but that’s most of
the time. Double quotes delimit a string context: the whole
double-quoted string is a single string, not to be split. (Exception:
"[email protected]" to expand to the list of positional parameters, e.g. "[email protected]" is
equivalent to "$1" "$2" "$3" if there are three positional
parameters. See What is the difference between $* and [email protected]?)

The same happens to command substitution with $(foo) or with
`foo`
. On a side note, don’t use `foo`: its quoting rules are
weird and non-portable, and all modern shells support $(foo) which
is absolutely equivalent except for having intuitive quoting rules.

The output of arithmetic substitution also undergoes the same
expansions, but that isn’t normally a concern as it only contains
non-expandable characters (assuming IFS doesn’t contain digits or
-).

See When is double-quoting necessary? for more details about the
cases when you can leave out the quotes.

Unless you mean for all this rigmarole to happen, just remember to
always use double quotes around variable and command substitutions. Do
take care: leaving out the quotes can lead not just to errors but to
security
holes
.

    11

    In addition to other issues caused by failing to quote, -n and -e can be consumed by echo as arguments. (Only the former is legal per the POSIX spec for echo, but several common implementations violate the spec and consume -e as well).

    To avoid this, use printf instead of echo when details matter.

    Thus:

    $ vars="-e -n -a"
    $ echo $vars      # breaks because -e and -n can be treated as arguments to echo
    -a
    $ echo "$vars"
    -e -n -a
    

    However, correct quoting won’t always save you when using echo:

    $ vars="-n"
    $ echo "$vars"
    $ ## not even an empty line was printed
    

    …whereas it will save you with printf:

    $ vars="-n"
    $ printf '%s\n' "$vars"
    -n
    

    3

    • Yay, we need a good dedup for this! I agree this fits the question title, but I don’t think it’ll get the visibility it deserves here. How about a new question à la “Why is my -e/-n/backslash not showing up?” We can add links from here as appropriate.

      Sep 21, 2018 at 23:24

    • Did you mean consume -n as well?

      – PesaThe

      Jan 7, 2019 at 15:32

    • 1

      @PesaThe, no, I meant -e. The standard for echo does not specify output when its first argument is -n, making any/all possible output legal in that case; there is no such provision for -e.

      Jan 7, 2019 at 15:46