use of 'testing' function from testsuite/testing.sh

[Cristian Ionescu-Idbohrn]

On Fri, 2 May 2008, Denys Vlasenko wrote:

> On Friday 02 May 2008 16:29, Cristian Ionescu-Idbohrn wrote:
> > Documentation:
> >
> >  # The "testing" function takes five arguments:
> >  #       $1) Description to display when running command
> > -#       $2) Command line arguments to command
> > +#       $2) Command and command line arguments
>
> Command(s) to run. May have pipes, redirects, etc

That's fine.  But "Command line arguments to command" is a little bit
confusing.

> >  #       $3) Expected result (on stdout)
> >  #       $4) Data written to file "input"
> >  #       $5) Data written to stdin
> > +# At least one of $4 and $5 should not be empty, but if both are empty
> > +# an input file name should be included in $2 (how do we validate that?).
>
> "Sanity checking found bugs. More sanity checking found more bugs
> and now even more sanity checking finds bugs in sanity checking.
>
> What will be next?"

You tell me.

> Seriously. $4 and $5 _can_ be empty. Lots of commands don't use input.
> For example, mdev.

Yes, I figured that out.  I'm not complaining, but I still think it's a
bit confusing and may thus be one reason why writing tests might not
so attractive.

> "an input file name should be included in $2". What input file?
> Why it should be included? The applets we test are so diverse,
> they have wastly different behavior. For example, sleep
> applet takes no input. At all. There will be no "input file"
> in $2 for it.

Yes, I do see that writing flexible tools is not easy.  I'm not
complainig.  I'm just stating unattractiveness has side effects, like not
enough tests to cover all areas.  And, by the way, attached is a POC test
for hexdump.

> And lastly, we don't need sanity checking on sanity checking.

If you say so.  But, I agree, it may get too complex.

> > Tests not specifying $4 or $5 _and_ the input (file) in $2 (when $4 is
> > non-empty) may not work at all.  Some still do work due to a
> > "feature/bug" in the 'testing' function, which always does this:
> >
> >   echo -ne "$5" | eval "$2"
> >
> > regardless.  This should be probably done only if both $4 and $5 are
> > empty.
> >
> > If $4 is non-emty the magic word 'input' should be present in $2, but even
> > if it's present, it could be placed in the wrong spot.  And if $5 is
> > empty, then only:
> >
> >   eval "$2"
> >
> > should be executed (no echo).
>
> What will happen if test *will* read stdin?

Yes.  Loose end.

> > Are there cases where we want both stdin and input file(s)?
>
> Yes. grep etc.
>
> > Are there cases where we _don't_ want them both?
>
> Yes. There are. But "testing" doesn't need to bother!

Those questions were a bit rhethorical.

> It should just feed the stding and create "input",
> and don't worry whether thing being tested uses it or not.

How do you then test whether thing being tested _shouldn't_ use it?

> > If both $4 and $5 are non-empy, the magic word 'input' should be present
> > in $2, double input should be assumed and the 'testing' function should
> > execute:
> >
> >   echo "$5" | eval "$2"
> >
> > Still, not totally reliable :(
>
> I dont understand. _What_ is not reliable?

Bad wording, maybe.  Confusion, maybe.  Still, finding reliable ways to
write tests isn't easy att all.


Cheers,

-- 
Cristian
-------------- next part --------------
_app=hexdump

tmpf=$(mktemp /tmp/${0##*/}.XXXXXXXXXX) || {
	echo "E: can't create temp file" >&2
	exit 1
}
exec 5>$tmpf

cleanup() {
	rm -fr $tmpf $host_hxd $bb_hxd $orig
}

trap 'cleanup' EXIT

_bbwd=$(pwd)
while [ "$_bbwd" ] && [ ! -x $_bbwd/busybox ]; do
	_bbwd=${_bbwd%/*}
done
[ "$_bbwd" ] || {
	echo "E: can't busybox executable" >&2
	exit 1
}

echo "PATH=$PATH" >&5
which -a hexdump >&5

hxd_path=
IFS_old="$IFS"
IFS=:
for _d in $PATH; do
	[ -x $_d/$_app ] || continue
	[ $_d != $_bbwd ] || continue
	hxd_path=$_d/$_app
	break
done
IFS="$IFS_old"
echo "I: found this:" >&5
echo "   $hxd_path" >&5
echo "   $_bbwd/busybox" >&5

host_hxd=$tmpf.host
> $host_hxd && chmod 600 $host_hxd
cat $0 | bzip2 -c | $hxd_path -Cv >>$host_hxd 2>>$tmpf
bb_hxd=$tmpf.bb
> $bb_hxd && chmod 600 $bb_hxd
cat $0 | bzip2 -c | $_bbwd/busybox $_app -Cv >>$bb_hxd 2>>$tmpf
cmp $host_hxd $bb_hxd 2>>$tmpf

orig=$tmpf.orig
> $orig && chmod 600 $orig
$_bbwd/busybox $_app -R $bb_hxd | bunzip2 -c >>$orig
cmp $0 $orig 2>>$tmpf