‘Delayed Expanded’ Environment Variables

June 16, 2008

Run the examples in this article using ‘Running Steps’ by Stepping Software (http://www.steppingsoftware.com).  It will help you visualize very easily the concepts here explained.

So, what are they, and do they serve any special purpose?  Environment variables make part of any normal batch script that reaches a minimal level of complexity.  Why the need for delayed expansion?  Let’s consider the typical example

set Missingfiles=
for %%p in (required1.exe required2.exe required3.exe) do (
    if “%%~$PATH:p”==”” set MissingFiles=%MissingFiles% %%p )

This piece of codes searches for the 3 required files (required1.exe … required3.exe) in the path, and if not found, it will be added to the list.  Let’s assume for the sake of the example that the 3 files are missing.  What is the value of MissingFiles after this code has executed?  It will not be the 3 files, it will be

required3.exe

So, if the files did not exist, how come only the 3rd (last) file got into the list?  Because the value %MissingFiles% is resolved before the for command is executed.  Therefore, the loop is the same as replacing the value before the loop is executed.  The loop instruction is

set MissingFiles=  %%p   (equal sign – space -space – %%p)

So MissingFiles  will be set 3 times with each file name, and the last one erases the 2 previous ones. 

OK, so the intention of the code is to make a list.  How should it be changed to do what it’s intended to do?  By using delayed expansion, like this: (ENABLEDELAYEDEXPANSION only works in batch files.  It does not work in the command line).

Setlocal ENABLEDELAYEDEXPANSION
set Missingfiles=
for %%p in (required1.exe required2.exe required3.exe) do (
    if “%%~$PATH:p”==”” set MissingFiles=!MissingFiles! %%p )

Now, at the end of the execution, the value of MissingFiles will be ‘required1.exe required2.exe required3.exe’  without the quotes, of course.  By using delayed expansions and indicating the variable name within exclamation marks, the expression !MissingFiles! is resolved when the statement is executed, and not when the statement is parsed.  Even with delayed expansion turned on, any variable expressed with percent signs (e.g. %MissingFiles%) will be resolved at parse time.

Cool!  Now, are there any other benefits from using delayed expansion?  Well yes.  There is a very subtle and not well known benefit.  Since the values are taken post-parse and during execution, any special characters will live literally.  This can be explained with an example.  Assume you want to write some xml file using a batch file (you may think me crazy, right?  so many conflicting symbols like <, >, &, quotes, etc).  Here we go.

set line=^<MovieData title=”Pillow Talk” Classification=”Comedy”^>Out of Stock^</Movie^>
echo processing line ‘%line%’
if “%line%”==”” echo Done Processing Data.  Terminating

In a real world example the value of line would not be hardwired but most likely read from a file.  Since here it’s hardwired, I added carets (^) before the special characters.

The output to the 2nd line is: The system cannot find the file specified.

The output to the 3rd line is: Talk” Classification=”Comedy”>Out of Stock</Movie>”==”” was unexpected at this time.

Why?  Well, all the < and > and quotes made the commands go way wrong.  Even when the carets were there in line 1.  But how could this be?  Well the carets in line 1 instructed the command processor when setting the variables, but that’s it.  The caret is not part of the value of variable ‘line’.  Symbols like < and > are used for redirection, and will make the command mean something different that what we intended.  Quotes will drive the if command nuts, since the parser for the command will think it’s the closing quote.  Quotes should appear twice in the condition for the command to work.  Like this:

if “he said:””I’m typing”””==”he said:””I’m typing””” echo strings match

The parser sees double quotes as a quote inside quotes.  The above works fine.  The following is wrong:

if “he said:”I’m typing”==”he said:”I’m typing” echo strings match

will yield the results ‘typing”==”he was unexpected at this time.

To avoid all these complications, and when special symbols will make part of variable values, use delayed expanded variables.  Using the same example, try the following code:

Caution:  If you are running this from the command line and not a batch file, replace the first line with cmd.exe /V:oninstead.  The options for setlocal only works inside a batch file and not in the command line.

setlocal ENABLEDELAYEDEXPANSION
set line=^<MovieData title=”Pillow Talk” Classification=”Comedy”^>Out of Stock^</Movie^>
echo processing line ‘!line!’
if “!line!”==”” echo Done Processing Data.  Terminating

Now line 2 produced this output:

processing line ‘<MovieData title=”Pillow Talk” Classification=”Comedy”>Out of Stock</Movie>’

That’s exactly what we wanted.  Since the string is parsed prior to execution, all redirections have already taken place and the special characters are taken as literals.  The same is true for strings with quotes.  No need to have double quotes when using delayed expansions.

And now, I will close with another example

setlocal ENABLEDELAYEDEXPANSION
set x=echo ^> is the greater than symbol
%x%
!x!
endlocal

Line 2 (%x%) echoed the string ‘the greater than symbol’ to a file named is.  Line 3 produced the following output in the console: > is the greater than symbol

So now you know.  If  you are processing file contents, and you want to be on the safe side.  Always use delayed expansion.

Enjoy!

GISkier

Variables in Batch Files

June 3, 2008

I will be writing about different topics all related to batch files.  Today I picked variables.  The variables used in batch files are environment variables.  Every process has an environment associated with it, and every environment has a list of variables.  To see the list of variables in your machine, open a command prompt window and type set.  It will give an alphabetical listing of all the variables.  Some of the common ones are USERNAME, USERDOMAIN, PROCESSOR_ARCHITECTURE, COMPUTERNAME, and SystemRoot

In batch files you can use these and create more of them.  The variables you create won’t affect the environment from which the batch file started running.  They are unique to the process running your batch file.  Variables can be defined to store information to be used later on, just like in any other language.  Most of the times variables are strings, but they can be integers if defined using the /A option, as follows:

set /a FileCount=5

In this case variable FileCount is 5.  Variable names are not case sensitive.  That means that I can refer to FILECOUNT, or filecount and both are the same thing.  A mix of upper case and lower case is the same also.

When a variable is defined as an integer, it can also be used as a string.  So, the following is valid

set /a FileCount=5
if “%FileCount%”==”5″ (echo yes) else (echo no)

This code will echo ‘yes’ to the console.  As you can see, to get the value of an environment variable you sorround it by percent signs.  (e.g.  %varibleName%).  If the variable you use is not defined you will get empty string when the batch file is running.  From the command line you get the literal (e.g. %nonExistentVar%).

Numeric Values

When variables store numeric values, you can perform operations with them.  Batch commands support grouping, arithmetic, logical, assignment operators and more.  You can see the list of operators by typing set /? in the command line.   All these operators give you the ability to keep count of things.  Something like set /a FileCount+=1 is perfectly valid.  So is set /a FileCount=%Previouscount% + 17.  When you use arithmetic expressions, you can avoid using the percent signs to denote variables.  That means that the following 2 expressions are equivalent:

set /a FileCount=%PreviousCount% + 17
set /a Filecount=PreviousCount + 17

This is only true when the /a option is used in a set command.   So what happens if a variable does not store a numeric value?  What happens is the value of PreviousCount in the example is the string ‘Yellow’.  Then the value is replaced by zero.

Some of the operators can be confused with special characters intended for pipelines and redirection (|, >>, <<).  If you are using arithmetic expressions involving any of these characters, use double quotes to avoid confusions.  Like this:

set /a myVar=”foo >> 2 | 7″

Without the quotes you get an error when running this command.  The if command provides some operators to compare numeric variables.  These are NEQ, EQU, LSS, LEQ, GTR, GTE, which stand for “not equal”, “equal to”, “less than”, “less than or equal”, “greater than” and “greater than or equal to” respectively.  An example on how to use these operators can be

if %fileCount% GTR 10 echo Too many files opened.  Process will abort.

String Variables

When variables hold strings, they can be used to save values, folder names, etc.  They can be used in comparisons just like numbers, and denoted with replacements too.  Another interesting point is that variables can store parts of instructions to be executed.  This can be useful when you have a value entered by the user, and based on it define where the code should execute, as this:

goto Option%chosenOption%

if the value for ChosenOption is ‘Label1′ the code would jump to label OptionLabel1.  This makes the value of variables more powerful and versatile than in other languages.   Complete instructions can be executed this way, as follows:

set myInstruction=echo This data comes from a variable
%myInstruction%

These 2 instructions will result in the string being echoed.  This makes environment variables even more powerful than in some other languages.  You can create code on the fly and execute it.  Pretty cool.

Watching out for common pitfalls

There are some things to be careful about.  Watch the following assignment:

set myVariable = myValue

At first sight it may seem fine.  But it’s not.  The name of the variable just created is ‘myVariable ‘ (with a space at the end), and it stores the value ‘ myValue’ (with a space at the beginning).  Running Steps will warn you if it finds this type of constructions in your files.   If you accidentally had spaces after ‘myValue’ on the line, the spaces will become part of the variable’s value.  Running Steps will warn you about that too.  Be careful not to fall into those.

Special Variables

There are some special variables which are predefined in your environment.  These are: errorlevel, cd, time, date, comspec, Random, and CmdExtVersion.  You should never create variables with these names.  If you do, you will be creating a variable that shadows out the special variable with the same name.  This leads to very hard to find bugs in your code.  Running Steps will warn you again this practice when it finds it.

Setlocal and Endlocal commands

Finally, I will briefly mention setlocal and endlocal commands.  When you execute a setlocal command, a new set of environment variables is created.  A copy of your set of variables.  From there on, any changes to your variables are applied in this new set.  Where did the initial set go?  It’s still there, and it will become active when an endlocal command executes.  As you can see these commands will effectively stack sets of variables (setlocal is like a push (push current and get a copy of it) and endlocal pops the old set from the stack and throws away the previous active set).   If you return from a routine (using goto :EOF or exit) you will return to the caller.  Upon returning, any new sets of variables will be discarded as if you had called endlocal several times.  My advice: don’t rely on this behavior.  Using as many endlocal instructions as setlocal will make your code more readable and maintainable.  Running Steps will count how many setlocal instructions a file has, and also how many endlocal‘s.  If the numbers do not match, a warning will be issued.  Just for fun, try the following:

call :MyRoutine
.
.
:Myroutine
set myVariable=myVariable version 0
setlocal
set myVariable=myVariable version 1

setlocal
set myVariable=myVariable version 2
goto :EOF

There are things related to variables that I did not mention here.  Specifically delayed expansion syntax.  I will write about that topic some other day.  It’s a whole topic in itself.

The topic of variables is very extensive and long.  I know.  But it’s the foundation for doing things with batch files.  I hope you stayed with me till the end.  If not, it’s OK too.   You may come and use this article as reference some other time.

I hope this will be helpful.  Enjoy!

GISkier

Batch files: Still here after so many years

June 2, 2008

I believe Batch files were available when the first IBM PC hit the market around 1980.   They have evolved a little.  Batch commands have changed to handle more options, a little more versatility has been added here and there.  How come, after 28 years they are still around, and they don’t seem to be going away any time soon?  Well, if the task to be performed is not large enough to write code in a compiled language, and not simple enough to do it manually, the batch file is the right answer!  It’s fast to write, easy to understand (well, not always), and easy to change.  They do have an expensive aspect, though.   They are hard to tune up and get running right.  A simple mistake can be very hard to find and correct:  misspelling a label can result in the wrong execution path;  a space in an assignment will result in the wrong assignment, a space in a file name can change execution fatally if the code is not solid.   Yes, but with all their frailties and difficulties, we continue to use them.

So who uses batch files?  Home users?  maybe here and there for some repetitive tasks.  But the main users are large companies.  They are used in backup processes, in file purging processes, program installations (before setup.exe is ready for prime time), changing user permissions, preparing machines for special code runs, copying data, etc.

Well, batch files can do many things, and be very powerful.  Some batch files have thousands of lines of code.  They can handle easily jobs which are more cumbersome in other languages, like starting and stopping services.

I will write about different aspects of batch files.  Today, writing batch files is easier with the introduction of a very cool product: Running Steps.  It’s a full featured debugger and code analyzer for batch files.  It can cut your development time significantly.  It will tell you about your mistakes before you need to dig into your code to find them because something is not executing right.  The code analyzer will check your code as you type.  It’s a syntax check, even though it goes beyond syntax.  Let’s say, I have been programming in C# or C++ all day, and move on to change a batch file.  I type the following line:

set MyVariable = new value

Looks fine at first sight, right?  Weill, it’s not.  Running Steps will display a wiggly line after the variable name.  Both the variable name and the value have a space which may not be intended to be there.  The variable name is ‘MyVariable ‘ (with a space at the end).  Catching this type of issues during execution is a pain.  The simpler the mistake, the harder it is to catch.

That is why having a tool to debug batch files is awesome.  It will help identify problems before they execute.  Running Steps will also allow me to step through my code, set breakpoints, watch/edit variable names, etc.  Just what you would expect from a good debugger.

So, I invite you to check it out.  You can find it at http://www.steppingsoftware.com .  I will be writing more about batch files, common faults, cool programming practices (check some cool information in the ‘Batch File Coding’ forum Stepping Software has created), use of variables, common pitfalls, etc.

I guess this is a good starting point for a blog on Batch Files.   Eventually cool tricks and techniques will be all found here.

Enjoy!

GISkier


Follow

Get every new post delivered to your Inbox.