قالب وردپرس درنا توس
Home / Tips and Tricks / Using Xargs in conjunction with bash -c to create complex commands – CloudSavvy IT

Using Xargs in conjunction with bash -c to create complex commands – CloudSavvy IT



Shutterstock / ksb

xargs is fireworks for your shell commands. Any output generated by any shell command can be sent to xargs for further processing to another command line. Learn to harness this amazing power of xargs today!

xargs Introduction

My colleague Dave McKay wrote an interesting article How to Use the xargs Command on Linux, which you may want to read first for a detailed introduction and exploration of xargs in general.

This article addresses a specific problem: what to do if you encounter the limitations of traditional tube-based stacking, or normal (; delimited) stacking Linux commands, and when even using xargs doesn̵

7;t seem to give an answer right away?

This is often, if not often, the case when writing one-liner scripts on the command line and / or when processing complex data or data structures. The information presented here is based on research and years of experience with this particular method and approach.

Tubes and xargs

Anyone who learns Bash will grow in their command-line scripting skills over time. The great thing about Bash is that all skills learned on the command line can be easily translated into Bash scripts (which are generally marked with a .sh suffix). The syntax is almost identical.

And as skills improve, newer engineers will usually discover the Pipe idiom first. Using a pipe is simple and straightforward: the output from the previous command is ‘piped’ to the input for the next command think of it as a water pipe taking water from an output source to an input source elsewhere, for example water from a dam via pipes to a water turbine.

Let’s look at a simple example:

echo 'a' | sed 's/a/b/'

A simple Bash pipe example with sed regex text replacement

Here we simply repeated ‘a’ and then changed the same thing using the text flow editor and. The output is of course ‘b’: the sed-substituted (s command) ‘a’ to ‘b’.

Some time later, the engineer will realize that pipes have limited options, especially if one wants to pre-process data into a format that is ready for the next tool. For example, consider this situation:

Encountering challenges with a pipe combined with kill

Here we start to sleep in the background. Then we use pidof to get the PID of the sleep command being executed, and try to kill it with kill -9 (Think of -9 as a destructive mode to end a process). It fails. We then try to use the PID provided by the shell when we started the background process, but this also fails.

The problem is that kill does not accept the input directly, whether or not it comes from ps or even a simple one echo. To solve this problem, we can use xargs to get the output of the ps or the echo command) and provide them as input for kill, by giving them arguments for the kill command. So it is as if we were executed kill -9 some_pid straight away. Let’s see how this works:

sleep 300 &
pidof sleep | xargs kill -9

xargs fixes the problem with the pipe kill command seen earlier

This works perfectly and achieves what we wanted to do: end the sleep process. A minor change to the code (ie just add xargs before the command), but a major change to how useful Bash can be for the developer anyway!

We can also use the -I option (defining the substitute string argument) to kill to make it a little clearer how we pass arguments to kill: i12b Here we define {} as our replacement string. In other words, when will xargs see {}, it will replace {} whatever input it received from the last command.

Yet even this has its limitations. How about if we want to provide some nice debugging information printed inline and from the statement? So far it seems impossible.

Yes, we can post-process the output with a sed regex, or add a subshell ($()) somewhere, but all of these still seem to have limitations, and especially when it comes time to build complex data streams, without using a new command and without using intermediate temporary files.

What if we could – once and for all – leave these limitations behind and be 100% free to create each Bash command line we like, just using pipes, xargs and the Bash shell, without any temporary intermediate files and without starting a new command? It is possible.

And it’s not complicated at all if someone shows it to you, but the first time it took some time and discussion to figure it out. I especially want to thank and thank my previous Linux mentor and former colleague Andrew Dalgleish – together we figured out how best to do this a little less than 10 years ago.

Welcome to xargs with bash -c

As we have seen, even when pipes are used with xargs, one will still run into some higher level scripting restrictions. Let’s take our previous example and inject some debugging information without editing the outcome. Normally this would be difficult to achieve, but not so with xargs combined with bash -c:

sleep 300 &
pidof sleep | xargs -I{} echo "echo 'The PID of your sleep process was: {}'; kill -9 {}; echo 'PID {} has now been terminated'" | xargs -I{} bash -c "{}"

A complex sequence of building xargs commands and using bash -c

Here we have used two xargs commands. The first builds a custom command line and uses as input the output of the previous command in the pipe (being pidof sleep) and the second xargs command executes the generated, custom-per-input (important!) command.

Why custom-per-input? This is because by default, xargs processes its input (the output of the previous command in the pipe) line by line and executes whatever it is instructed to do for each such input line.

There is a lot of power here. This means you can create and build any custom command and then run it, completely free of whatever format the input data resides in, and you don’t have to worry about how to run it. The only syntax to remember is this:

some_command | xargs -I{} echo "echo '...{}...'; more_commands; more_commands_with_or_without{}" | xargs -I{} bash -c "{}"

Note that the nested echo (the second echo) is only really needed if you want to redo the actual text. Otherwise, if the second echo was not there, the first echo would start running ‘The PID …’ etc and the bash -c subshell could not parse this as a command (IOW, ‘The PID…’ is not a command and cannot be executed as such, hence the secondary / nested echo).

Once you remember it bash -c, the -I{} and the way to echo from another echo (and you can optionally use escape sequences as well), you will find yourself using this syntax over and over again.

Let’s say you have to do three things per file in the directory: 1) run the contents of the file, 2) move it to a subdirectory, 3) delete it. Normally, this requires a number of steps with different staged commands, and as things get more complicated you may even need temporary files. But it is done with very easily bash -c and xargs:

echo '1' > a
echo '2' > b
echo '3' > c
mkdir subdir
ls --color=never | grep -v subdir | xargs -I{} echo "cat {}; mv {} subdir; rm subdir/{}" | xargs -I{} bash -c "{}"

A fully functioning miniscript based on xargs and bash -c in a one-liner command

First of all, quickly note that it is always a good idea to use --color=never in front of ls, to avoid problems on systems that use color coding for directory listing output (enabled by default in Ubuntu), as this often causes serious parsing problems because the color codes are actually sent to the terminal and precede entries in the directory listings.

We will first create three files, a, b and c, and a subfolder called subdir. We exclude this subdir from the directory listing with grep -v after we notice that a quick trial run (without running the commands, or in other words without the second xargs) the subdir will still be displayed.

It’s always important to test your complex commands first by observing the output before actually passing them to a bash subshell with bash -c for execution.

Finally, we use the exact same method seen before to build or command; cat (show) the file, move the file (indicated by {}) to the subdir and finally delete the file in the subdir. We see the contents of the 3 files (1, 2, 3) are displayed on the screen, and if we check the current directory, our files are gone. We can also see that there are no more files in the subdir. Everything worked fine.

Shut down

Using xargs are a great strength for an advanced Linux (and in this case) Bash user. Using xargs in combination with bash -c brings much more power; the unique ability to build custom and complex command lines 100% free, with no intermediate files or constructs and without the need to have stacked / consecutive commands.

To enjoy!


Source link