Categories
batch-file pipe

Why does delayed expansion fail when inside a piped block of code?

39

Here is a simple batch file that demonstrates how delayed expansion fails if it is within a block that is being piped. (The failure is toward the end of the script) Can anyone explain why this is?

I have a work-around, but it requires creation of a temporary file. I initially ran into this problem while working on Find files and sort by size in a Windows batch file

@echo off
setlocal enableDelayedExpansion

set test1=x
set test2=y
set test3=z

echo(

echo NORMAL EXPANSION TEST
echo Unsorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
)
echo(
echo Sorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
) | sort

echo(
echo ---------
echo(

echo DELAYED EXPANSION TEST
echo Unsorted works
(
  echo !test3!
  echo !test1!
  echo !test2!
)
echo(
echo Sorted fails
(
  echo !test3!
  echo !test1!
  echo !test2!
) | sort
echo(
echo Sort workaround
(
  echo !test3!
  echo !test1!
  echo !test2!
)>temp.txt
sort temp.txt
del temp.txt

Here are the results

NORMAL EXPANSION TEST
Unsorted works
z
x
y

Sorted works
x
y
z

---------

DELAYED EXPANSION TEST
Unsorted works
z
x
y

Sorted fails
!test1!
!test2!
!test3!

Sort workaround
x
y
z

    51

    As Aacini shows, it seems that many things fail within a pipe.

    echo hello | set /p var=
    echo here | call :function
    

    But in reality it’s only a problem to understand how the pipe works.

    Each side of a pipe starts its own cmd.exe in its own ascynchronous thread.
    That is the cause why so many things seem to be broken.

    But with this knowledge you can avoid this and create new effects

    echo one | ( set /p varX= & set varX )
    set var1=var2
    set var2=content of two
    echo one | ( echo %%%var1%%% )
    echo three | echo MYCMDLINE %%cmdcmdline%%
    echo four  | (cmd /v:on /c  echo 4: !var2!)
    

    Update 2019-08-15:
    As discovered at Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?, cmd.exe is only used if the command is internal to cmd.exe, if the command is a batch file, or if the command is enclosed in a parenthesized block. External commands not enclosed within parentheses are launched in a new process without the aid of cmd.exe.

    EDIT: In depth analysis

    As dbenham shows, both sides of the pipes are equivalent for the expansion phases.
    The main rules seems to be:

    The normal batch parser phases are done
    .. percent expansion
    .. special character phase/block begin detection
    .. delayed expansion (but only if delayed expansion is enabled AND it isn’t a command block)

    Start the cmd.exe with C:\Windows\system32\cmd.exe /S /D /c"<BATCH COMMAND>"
    These expansions follows the rules of the cmd-line parser not the the batch-line parser.

    .. percent expansion
    .. delayed expansion (but only if delayed expansion is enabled)

    The <BATCH COMMAND> will be modified if it’s inside a parenthesis block.

    (
    echo one %%cmdcmdline%%
    echo two
    ) | more
    

    Called as C:\Windows\system32\cmd.exe /S /D /c" ( echo one %cmdcmdline% & echo two )", all newlines are changed to & operator.

    Why the delayed expansion phase is affected by parenthesis?
    I suppose, it can’t expand in the batch-parser-phase, as a block can consist of many commands and the delayed expansion take effect when a line is executed.

    (
    set var=one
    echo !var!
    set var=two
    ) | more
    

    Obviously the !var! can’t be evaluated in the batch context, as the lines are executed only in the cmd-line context.

    But why it can be evaluated in this case in the batch context?

    echo !var! | more
    

    In my opionion this is a “bug” or inconsitent behaviour, but it’s not the first one

    EDIT: Adding the LF trick

    As dbenham shows, there seems to be some limitation through the cmd-behaviour that changes all line feeds into &.

    (
      echo 7: part1
      rem This kills the entire block because the closing ) is remarked!
      echo part2
    ) | more
    

    This results into
    C:\Windows\system32\cmd.exe /S /D /c" ( echo 7: part1 & rem This ...& echo part2 ) "
    The rem will remark the complete line tail, so even the closing bracket is missing then.

    But you can solve this with embedding your own line feeds!

    set LF=^
    
    
    REM The two empty lines above are required
    (
      echo 8: part1
      rem This works as it splits the commands %%LF%% echo part2  
    ) | more
    

    This results to C:\Windows\system32\cmd.exe /S /D /c" ( echo 8: part1 %cmdcmdline% & rem This works as it splits the commands %LF% echo part2 )"

    And as the %lf% is expanded while parsing the parenthises by the parser, the resulting code looks like

    ( echo 8: part1 & rem This works as it splits the commands 
      echo part2  )
    

    This %LF% behaviour works always inside of parenthesis, also in a batch file.
    But not on “normal” lines, there a single <linefeed> will stop the parsing for this line.

    EDIT: Asynchronously is not the full truth

    I said that the both threads are asynchronous, normally this is true.
    But in reality the left thread can lock itself when the piped data isn’t consumed by the right thread.
    There seems to be a limit of ~1000 characters in the “pipe” buffer, then the thread is blocked until the data is consumed.

    @echo off
    (
        (
        for /L %%a in ( 1,1,60 ) DO (
                echo A long text can lock this thread
                echo Thread1 ##### %%a > con
            )
        )
        echo Thread1 ##### end > con
    ) | (
        for /L %%n in ( 1,1,6) DO @(
            ping -n 2 localhost > nul
            echo Thread2 ..... %%n
            set /p x=
        )
    )
    

    9

    • +1 Thank you jeb for finally opening my eyes to what is happening with SET /P. I used to think it was a peculiarity of how it reads input. Now I realize it is actually working, but of course we can’t pass the result back to the parent environment, so it is kind of useless.

      – dbenham

      Nov 19, 2011 at 15:41

    • This answer still does not explain why delayed expansion is disabled within the CMD context. See my own “answer” to this question.

      – dbenham

      Nov 19, 2011 at 16:55

    • 1

      +1 Very interesting! This behavior modifies some basic Batch rules: cd . | BatSub returns to current Batch file after BatSub.bat ends even if it was not called via CALL nor CMD /C (we know now that there is an implicit CMD /C here). Also, we know now that is faster to do two redirections com1 > file & com2 < file instead of a pipe: com1 | com2; I will avoid pipes in favor of two redirections from now on. All this stuff sound very strange to me! @jeb: just one detail, the execution of right side of pipe is not ascynchronous…

      – Aacini

      Nov 20, 2011 at 17:08


    • 1

      @jeb: You are right! The execution of both sides in a pipeline ARE asynchronous! See the addendum in my answer (this becomes stranger every time…)

      – Aacini

      Nov 20, 2011 at 18:19


    • 1

      Great stuff jeb. Everything indeed makes sense now.The %%cmcmdline%% trick really helps explain things. One other thing not mentioned yet: CMD.EXE does not inherit the parent delayed expansion state; it defaults this state based on the registry setting. Presumably the same is true for the command extensions state.

      – dbenham

      Nov 20, 2011 at 22:46

    11

    I wasn’t sure if I should edit my question, or post this as an answer.

    I already vaguely knew that a pipe executes both the left and the right side each in its own CMD.EXE “session”. But Aacini’s and jeb’s responses forced me to really think about and investigate what is happening with pipes. (Thank you jeb for demonstrating what is happening when piping into SET /P!)

    I developed this investigative script – it helps explain a lot, but also demonstrates some bizarre and unexpected behavior. I’ll post the script, followed by the output. Finally I will provide some analysis.

    @echo off
    cls
    setlocal disableDelayedExpansion
    set var1=value1
    set "var2="
    setlocal enableDelayedExpansion
    
    echo on
    @echo NO PIPE - delayed expansion is ON
    echo 1: %var1%, %var2%, !var1!, !var2!
    (echo 2: %var1%, %var2%, !var1!, !var2!)
    
    @echo(
    @echo PIPE LEFT SIDE - Delayed expansion is ON
    echo 1L: %%var1%%, %%var2%%, !var1!, !var2! | more
    (echo 2L: %%var1%%, %%var2%%, !var1!, !var2!) | more
    (setlocal enableDelayedExpansion & echo 3L: %%var1%%, %%var2%%, !var1!, !var2!) | more
    (cmd /v:on /c echo 4L: %%var1%%, %%var2%%, !var1!, !var2!) | more
    cmd /v:on /c echo 5L: %%var1%%, %%var2%%, !var1!, !var2! | more
    @endlocal
    @echo(
    @echo Delayed expansion is now OFF
    (cmd /v:on /c echo 6L: %%var1%%, %%var2%%, !var1!, !var2!) | more
    cmd /v:on /c echo 7L: %%var1%%, %%var2%%, !var1!, !var2! | more
    
    @setlocal enableDelayedExpansion
    @echo(
    @echo PIPE RIGHT SIDE - delayed expansion is ON
    echo junk | echo 1R: %%var1%%, %%var2%%, !var1!, !var2!
    echo junk | (echo 2R: %%var1%%, %%var2%%, !var1!, !var2!)
    echo junk | (setlocal enableDelayedExpansion & echo 3R: %%var1%%, %%var2%%, !var1!, !var2!)
    echo junk | (cmd /v:on /c echo 4R: %%var1%%, %%var2%%, !var1!, !var2!)
    echo junk | cmd /v:on /c echo 5R: %%var1%%, %%var2%%, !var1!, !var2!
    @endlocal
    @echo(
    @echo Delayed expansion is now OFF
    echo junk | (cmd /v:on /c echo 6R: %%var1%%, %%var2%%, !var1!, !var2!)
    echo junk | cmd /v:on /c echo 7R: %%var1%%, %%var2%%, !var1!, !var2!
    

    Here is the output

    NO PIPE - delayed expansion is ON
    
    C:\test>echo 1: value1, , !var1!, !var2!
    1: value1, , value1,
    
    C:\test>(echo 2: value1, , !var1!, !var2! )
    2: value1, , value1,
    
    PIPE LEFT SIDE - Delayed expansion is ON
    
    C:\test>echo 1L: %var1%, %var2%, !var1!, !var2!   | more
    1L: value1, %var2%, value1,
    
    
    C:\test>(echo 2L: %var1%, %var2%, !var1!, !var2! )  | more
    2L: value1, %var2%, !var1!, !var2!
    
    
    C:\test>(setlocal enableDelayedExpansion   & echo 3L: %var1%, %var2%, !var1!, !var2! )  | more
    3L: value1, %var2%, !var1!, !var2!
    
    
    C:\test>(cmd /v:on /c echo 4L: %var1%, %var2%, !var1!, !var2! )  | more
    4L: value1, %var2%, value1, !var2!
    
    
    C:\test>cmd /v:on /c echo 5L: %var1%, %var2%, !var1!, !var2!   | more
    5L: value1, %var2%, value1,
    
    
    Delayed expansion is now OFF
    
    C:\test>(cmd /v:on /c echo 6L: %var1%, %var2%, !var1!, !var2! )  | more
    6L: value1, %var2%, value1, !var2!
    
    
    C:\test>cmd /v:on /c echo 7L: %var1%, %var2%, !var1!, !var2!   | more
    7L: value1, %var2%, value1, !var2!
    
    
    PIPE RIGHT SIDE - delayed expansion is ON
    
    C:\test>echo junk   | echo 1R: %var1%, %var2%, !var1!, !var2!
    1R: value1, %var2%, value1,
    
    C:\test>echo junk   | (echo 2R: %var1%, %var2%, !var1!, !var2! )
    2R: value1, %var2%, !var1!, !var2!
    
    C:\test>echo junk   | (setlocal enableDelayedExpansion   & echo 3R: %var1%, %var2%, !var1!, !var2! )
    3R: value1, %var2%, !var1!, !var2!
    
    C:\test>echo junk   | (cmd /v:on /c echo 4R: %var1%, %var2%, !var1!, !var2! )
    4R: value1, %var2%, value1, !var2!
    
    C:\test>echo junk   | cmd /v:on /c echo 5R: %var1%, %var2%, !var1!, !var2!
    5R: value1, %var2%, value1,
    
    Delayed expansion is now OFF
    
    C:\test>echo junk   | (cmd /v:on /c echo 6R: %var1%, %var2%, !var1!, !var2! )
    6R: value1, %var2%, value1, !var2!
    
    C:\test>echo junk   | cmd /v:on /c echo 7R: %var1%, %var2%, !var1!, !var2!
    7R: value1, %var2%, value1, !var2!
    

    I tested both the left and right side of the pipe to demonstrate that processing is symmetric on both sides.

    Tests 1 and 2 demonstrate that parentheses don’t have any impact on delayed expansion under normal batch circumstances.

    Tests 1L,1R: Delayed expansion works as expected. Var2 is undefined, so %var2% and !var2! output demonstrates that the commands are executed in a command line context, and not a batch context. In other words, command line parsing rules are used instead of batch parsing. (see How does the Windows Command Interpreter (CMD.EXE) parse scripts?) EDIT – !VAR2! is expanded in the parent batch context

    Tests 2L,2R: The parentheses disable the delayed expansion! Very bizarre and unexpected in my mind. Edit – jeb considers this an MS bug or design flaw. I agree, there doesn’t seem to be any rational reason for the inconsistent behavior

    Tests 3L,3R: setlocal EnableDelayedExpansion does not work. But this is expected because we are in a command line context. setlocal only works in a batch context.

    Tests 4L,4R: Delayed expansion is initially enabled, but parentheses disable it. CMD /V:ON re-enables delayed expansion and everything works as expected. We still have command line context and output is as expected.

    Tests 5L,5R: Almost the same as 4L,4R except delayed expansion is already enabled when CMD /V:on is executed. %var2% gives expected command line context output. But !var2! output is blank which is expected in a batch context. This is another very bizarre and unexpected behavior. Edit – actually this makes sense now that I know !var2! is expanded in the parent batch context

    Tests 6L,6R,7L,7R: These are analogous to tests 4L/R,5L/R except now delayed expansion starts out disabled. This time all 4 scenarios give the expected !var2! batch context output.

    If someone can provide a logical explanation for results of 2L,2R and 5L,5R then I will select that as the answer to my original question. Otherwise I will probably accept this post as the answer (really more of an observation of what happens than an answer) Edit – jab nailed it!


    Addendum: In response to jeb’s comment – here is more evidence that piped commands within a batch execute in a command line context, not a batch context.

    This batch script:

    @echo on
    call echo batch context %%%%
    call echo cmd line context %%%% | more
    

    gives this output:

    C:\test>call echo batch context %%
    batch context %
    
    C:\test>call echo cmd line context %%   | more
    cmd line context %%
    


    Final Addendum

    I’ve added some additional tests and results that demonstrate all the findings so far. I also demonstrate that FOR variable expansion takes place before the pipe processing. Finally I show some interesting side effects of the pipe processing when a multi-line block is collapsed into a single line.

    @echo off
    cls
    setlocal disableDelayedExpansion
    set var1=value1
    set "var2="
    setlocal enableDelayedExpansion
    
    echo on
    @echo(
    @echo Delayed expansion is ON
    echo 1: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^!, !var2!, ^^^!var2^^^!, %%cmdcmdline%% | more
    (echo 2: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
    for %%a in (Z) do (echo 3: %%a %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
    (
      echo 4: part1
      set "var2=var2Value
      set var2
      echo "
      set var2
    )
    (
      echo 5: part1
      set "var2=var2Value
      set var2
      echo "
      set var2
      echo --- begin cmdcmdline ---
      echo %%cmdcmdline%%
      echo --- end cmdcmdline ---
    ) | more
    (
      echo 6: part1
      rem Only this line remarked
      echo part2
    )
    (
      echo 7: part1
      rem This kills the entire block because the closing ) is remarked!
      echo part2
    ) | more
    

    Here is the output

    Delayed expansion is ON
    
    C:\test>echo 1: %, %var1%, %var2%, !var1!, ^!var1^!, !var2!, ^!var2^!, %cmdcmdline%   | more
    1: %, value1, %var2%, value1, !var1!, , !var2!, C:\Windows\system32\cmd.exe  /S /D /c" echo 1: %, %var1%, %var2%, value1, !var1!, , !var2!, %cmdcmdline% "
    
    
    C:\test>(echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
    2: %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"
    
    
    C:\test>for %a in (Z) do (echo 3: %a %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
    
    C:\test>(echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
    3: Z %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"
    
    C:\test>(
    echo 4: part1
     set "var2=var2Value
     set var2
     echo "
     set var2
    )
    4: part1
    var2=var2Value
    "
    var2=var2Value
    
    C:\test>(
    echo 5: part1
     set "var2=var2Value
     set var2
     echo "
     set var2
     echo --- begin cmdcmdline ---
     echo %cmdcmdline%
     echo --- end cmdcmdline ---
    )  | more
    5: part1
    var2=var2Value & set var2 & echo
    --- begin cmdcmdline ---
    C:\Windows\system32\cmd.exe  /S /D /c" ( echo 5: part1 & set "var2=var2Value
    var2=var2Value & set var2 & echo
    " & set var2 & echo --- begin cmdcmdline --- & echo %cmdcmdline% & echo --- end cmdcmdline --- )"
    --- end cmdcmdline ---
    
    
    C:\test>(
    echo 6: part1
     rem Only this line remarked
     echo part2
    )
    6: part1
    part2
    
    C:\test>(echo %cmdcmdline%   & (
    echo 7: part1
     rem This kills the entire block because the closing ) is remarked!
     echo part2
    ) )  | more
    

    Tests 1: and 2: summarize all the behaviors, and the %%cmdcmdline%% trick really helps to demonstrate what is taking place.

    Test 3: demonstrates that FOR variable expansion still works with a piped block.

    Tests 4:/5: and 6:/7: show interesting side effects of the way pipes work with multi-line blocks. Beware!

    I’ve got to believe figuring out escape sequences within complex pipe scenarios will be a nightmare.

    5

    • +1, I love exhaustive tests, but some of your conclusions seems to be wrong. IMHO your interpreation of 1LR and 5LR is wrong

      – jeb

      Nov 19, 2011 at 22:53


    • @jeb ??? where did my analysis go wrong? Especially with regard to 1LR, since the 3LR results seems to support my conclusion. 5LR is still a mystery to me.

      – dbenham

      Nov 19, 2011 at 23:12

    • I edit my answer, and hopefully it explains the complete behaviour now 🙂

      – jeb

      Nov 20, 2011 at 13:23


    • @jeb – Absolutely! It all makes sense now. Take a look at some of the side effects I demonstrate in my final addendum. They deal with how multi-line blocks are processed. It’s nasty! But it all makes sense.

      – dbenham

      Nov 20, 2011 at 23:30

    • I added a solution in my answer for the & behaviour with REM or quotes

      – jeb

      Nov 24, 2011 at 13:59

    8

    Funny thing! I don’t know the answer, what I know is that the pipeline operation have consistent failures in Windows Batch that should not be present in original MS-DOS Batch (if such features could be executed in old MS-DOS Batch), so I suspect that the error was introduced when the new Windows Batch features were developed.

    Here are some examples:

    echo Value to be assigned | set /p var=

    Previous line does NOT assign the value to the variable, so we must fix it this way:

    echo Value to be assigned > temp.txt & set /p var=< temp.txt

    Another one:

    (
    echo Value one
    echo Value two
    echo Value three
    ) | call :BatchSubroutine
    

    Doesn’t work. Fix it this way:

    (
    echo Value one
    echo Value two
    echo Value three
    ) > temp.txt
    call :BatchSubroutine < temp.txt
    

    However, this method DO work in certain cases; with DEBUG.COM for example:

    echo set tab=9> def_tab.bat
    (
    echo e108
    echo 9
    echo w
    echo q
    ) | debug def_tab.bat
    call def_tab
    echo ONE%tab%TWO
    

    Previous program show:

    ONE     TWO

    In which cases works and which not? Only God (and Microsoft) may know, but it seems to be related to new Windows Batch features: SET /P command, delayed expansion, code block in parentheses, etc.

    EDIT: Asynchronous Batch files

    NOTE: I modified this section to correct an error of mine. See my last comment to jeb for details.

    As jeb said, the execution of both sides of a pipeline create two asynchronous processes, that made possible to execute asynchronous threads even if START command is not used.

    Mainfile.bat:

    @echo off
    echo Main start. Enter lines, type end to exit
    First | Second
    echo Main end
    

    First.bat:

    @echo off
    echo First start
    
    :loop
        set /P first=
        echo First read: %first%
    if /I not "%first%" == "end" goto loop
    echo EOF
    
    echo First end
    

    Second.bat:

    @echo off
    echo Second start
    
    :loop
        set /P second=Enter line: 
        echo Second read: %second%
        echo/
    if not "%second%" == "EOF" goto loop
    
    echo Second end
    

    We may use this capability to develop a program equivalent to Expect application (working in a similar way of pexpect Phyton module) that could control any interactive program this way:

    Input | anyprogram | Output
    

    Output.bat file will achieve the “Expect” part by analysing the output from the program, and Input.bat will achieve the “Sendline” part by providing the input to the program. The backwards communication from Output to Input modules will be achieved via a file with the desired information and a simple semaphore system controlled via the presence/absence of one or two flag files.

    3

    • jeb’s answer explains why a pipe can’t call a batch :function – The CALL :FUNCTION command is not executed within the batch context, so it can’t possibly work.

      – dbenham

      Nov 19, 2011 at 16:44

    • Your edit about the asynchronous behaviour is nice, but not completely correct. Even the pipe starts first in a cmd context, but if you start there a batch, you are in a batch context again.

      – jeb

      Nov 20, 2011 at 20:07

    • 1

      @jeb: You are right again! When I was developing my example a strange error happens and I was confused by that 🙁 I corrected my example, deleted the erroneous text about the context and added a possible application of this: an Expect-like program.

      – Aacini

      Nov 20, 2011 at 23:45