From aldot at busybox.net Fri Dec 1 08:56:12 2006 From: aldot at busybox.net (aldot at busybox.net) Date: Fri, 1 Dec 2006 08:56:12 -0800 (PST) Subject: svn commit: trunk/busybox: networking util-linux Message-ID: <20061201165612.C20A048577@busybox.net> Author: aldot Date: 2006-12-01 08:56:12 -0800 (Fri, 01 Dec 2006) New Revision: 16745 Log: - provide central knob to turn off RPC related options. Modified: trunk/busybox/Config.in trunk/busybox/networking/Config.in trunk/busybox/util-linux/Config.in Changeset: Modified: trunk/busybox/Config.in =================================================================== --- trunk/busybox/Config.in 2006-11-30 23:13:59 UTC (rev 16744) +++ trunk/busybox/Config.in 2006-12-01 16:56:12 UTC (rev 16745) @@ -203,6 +203,14 @@ /etc/busybox.conf should be readable by the user needing the SUID, check this option to avoid users to be notified about missing permissions. +config FEATURE_HAVE_RPC + bool "RPC support" + default y + help + Select this if you have rpc support. + This automatically turns off all configuration options that rely + on RPC. + config SELINUX bool "Support NSA Security Enhanced Linux" default n Modified: trunk/busybox/networking/Config.in =================================================================== --- trunk/busybox/networking/Config.in 2006-11-30 23:13:59 UTC (rev 16744) +++ trunk/busybox/networking/Config.in 2006-12-01 16:56:12 UTC (rev 16745) @@ -324,10 +324,10 @@ bool "Support RPC services" default n depends on INETD + depends on FEATURE_HAVE_RPC help - Suuport Sun-RPC based services + Support Sun-RPC based services - config IP bool "ip" default n Modified: trunk/busybox/util-linux/Config.in =================================================================== --- trunk/busybox/util-linux/Config.in 2006-11-30 23:13:59 UTC (rev 16744) +++ trunk/busybox/util-linux/Config.in 2006-12-01 16:56:12 UTC (rev 16745) @@ -369,6 +369,7 @@ bool "Support mounting NFS file systems" default n depends on MOUNT + depends on FEATURE_HAVE_RPC select FEATURE_SYSLOG help Enable mounting of NFS file systems. From bugs at busybox.net Fri Dec 1 11:40:03 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Fri, 1 Dec 2006 11:40:03 -0800 Subject: [BusyBox 0001108]: Implement flock Message-ID: <19d42e8d005c5b4d62adac51178e9078@www.busybox.net> The following issue has been SUBMITTED. ====================================================================== http://busybox.net/bugs/view.php?id=1108 ====================================================================== Reported By: eswierk Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 1108 Category: New Features Reproducibility: always Severity: feature Priority: normal Status: assigned ====================================================================== Date Submitted: 12-01-2006 11:40 PST Last Modified: 12-01-2006 11:40 PST ====================================================================== Summary: Implement flock Description: I've ported H. Peter Anvin's flock implementation (http://directory.fsf.org/All/flock.html) to BusyBox. This utility is handy for implementing mutual exclusion in shell scripts. The attached patch is a rough first draft. It works, but segfaults if the fd# argument is missing. I would appreciate suggestions for improvements, especially for making option processing and usage more BusyBox-like. ====================================================================== Issue History Date Modified Username Field Change ====================================================================== 12-01-06 11:40 eswierk New Issue 12-01-06 11:40 eswierk Status new => assigned 12-01-06 11:40 eswierk Assigned To => BusyBox 12-01-06 11:40 eswierk File Added: busybox-200-flock.patch ====================================================================== From bugs at busybox.net Fri Dec 1 11:58:43 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Fri, 1 Dec 2006 11:58:43 -0800 Subject: [BusyBox 0001108]: Implement flock Message-ID: A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=1108 ====================================================================== Reported By: eswierk Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 1108 Category: New Features Reproducibility: always Severity: feature Priority: normal Status: assigned ====================================================================== Date Submitted: 12-01-2006 11:40 PST Last Modified: 12-01-2006 11:58 PST ====================================================================== Summary: Implement flock Description: I've ported H. Peter Anvin's flock implementation (http://directory.fsf.org/All/flock.html) to BusyBox. This utility is handy for implementing mutual exclusion in shell scripts. The attached patch is a rough first draft. It works, but segfaults if the fd# argument is missing. I would appreciate suggestions for improvements, especially for making option processing and usage more BusyBox-like. ====================================================================== ---------------------------------------------------------------------- eswierk - 12-01-06 11:58 ---------------------------------------------------------------------- I just noticed that my Fedora system has another implementation of flock, from util-linux. The usage is completely different from the hpa implementation, and since it's probably in much more widespread usage, I'll port this one to BusyBox instead. Issue History Date Modified Username Field Change ====================================================================== 12-01-06 11:40 eswierk New Issue 12-01-06 11:40 eswierk Status new => assigned 12-01-06 11:40 eswierk Assigned To => BusyBox 12-01-06 11:40 eswierk File Added: busybox-200-flock.patch 12-01-06 11:41 eswierk Issue Monitored: eswierk 12-01-06 11:58 eswierk Note Added: 0001851 ====================================================================== From vda at busybox.net Fri Dec 1 13:34:21 2006 From: vda at busybox.net (vda at busybox.net) Date: Fri, 1 Dec 2006 13:34:21 -0800 (PST) Subject: svn commit: trunk/busybox: include libbb loginutils Message-ID: <20061201213421.83CA14857F@busybox.net> Author: vda Date: 2006-12-01 13:34:20 -0800 (Fri, 01 Dec 2006) New Revision: 16747 Log: passwd: made smaller by ~130 bytes. size can go negative if current trend will continue ;) Modified: trunk/busybox/include/libbb.h trunk/busybox/libbb/bb_pwd.c trunk/busybox/loginutils/passwd.c Changeset: Modified: trunk/busybox/include/libbb.h =================================================================== --- trunk/busybox/include/libbb.h 2006-12-01 17:49:42 UTC (rev 16746) +++ trunk/busybox/include/libbb.h 2006-12-01 21:34:20 UTC (rev 16747) @@ -320,7 +320,7 @@ * increases target size and is often not needed on embedded systems. */ extern long bb_xgetpwnam(const char *name); extern long bb_xgetgrnam(const char *name); -extern char *bb_getug(char *buffer, char *idname, long id, int bufsize, char prefix); +/*extern char *bb_getug(char *buffer, char *idname, long id, int bufsize, char prefix);*/ extern char *bb_getpwuid(char *name, long uid, int bufsize); extern char *bb_getgrgid(char *group, long gid, int bufsize); /* from chpst */ Modified: trunk/busybox/libbb/bb_pwd.c =================================================================== --- trunk/busybox/libbb/bb_pwd.c 2006-12-01 17:49:42 UTC (rev 16746) +++ trunk/busybox/libbb/bb_pwd.c 2006-12-01 21:34:20 UTC (rev 16747) @@ -12,6 +12,35 @@ #include #include "libbb.h" + /* + * if bufsize is > 0 char *buffer cannot be set to NULL. + * If idname is not NULL it is written on the static + * allocated buffer (and a pointer to it is returned). + * if idname is NULL, id as string is written to the static + * allocated buffer and NULL is returned. + * if bufsize is = 0 char *buffer can be set to NULL. + * If idname exists a pointer to it is returned, + * else NULL is returned. + * if bufsize is < 0 char *buffer can be set to NULL. + * If idname exists a pointer to it is returned, + * else an error message is printed and the program exits. + */ + +/* internal function for bb_getpwuid and bb_getgrgid */ +static char * bb_getug(char *buffer, char *idname, long id, int bufsize, char prefix) +{ + if (bufsize > 0 ) { + assert(buffer!=NULL); + if(idname) { + return safe_strncpy(buffer, idname, bufsize); + } + snprintf(buffer, bufsize, "%ld", id); + } else if (bufsize < 0 && !idname) { + bb_error_msg_and_die("unknown %cid %ld", prefix, id); + } + return idname; +} + /* Hacked by Tito Ragusa (c) 2004 to make it more * flexible : * @@ -84,49 +113,18 @@ { struct passwd *myuser = getpwuid(uid); - return bb_getug(name, (myuser) ? - myuser->pw_name : (char *)myuser , uid, bufsize, 'u'); + return bb_getug(name, myuser ? myuser->pw_name : (char *)myuser, + uid, bufsize, 'u'); } - /* - * if bufsize is > 0 char *buffer cannot be set to NULL. - * If idname is not NULL it is written on the static - * allocated buffer (and a pointer to it is returned). - * if idname is NULL, id as string is written to the static - * allocated buffer and NULL is returned. - * if bufsize is = 0 char *buffer can be set to NULL. - * If idname exists a pointer to it is returned, - * else NULL is returned. - * if bufsize is < 0 char *buffer can be set to NULL. - * If idname exists a pointer to it is returned, - * else an error message is printed and the program exits. - */ - -/* internal function for bb_getpwuid and bb_getgrgid */ -char * bb_getug(char *buffer, char *idname, long id, int bufsize, char prefix) -{ - if(bufsize > 0 ) { - assert(buffer!=NULL); - if(idname) { - return safe_strncpy(buffer, idname, bufsize); - } - snprintf(buffer, bufsize, "%ld", id); - } else if(bufsize < 0 && !idname) { - bb_error_msg_and_die("unknown %cid %ld", prefix, id); - } - return idname; -} - unsigned long get_ug_id(const char *s, long (*__bb_getxxnam)(const char *)) { unsigned long r; - char *p; - r = strtoul(s, &p, 10); - if (*p || (s == p)) { + r = bb_strtoul(s, NULL, 10); + if (errno) r = __bb_getxxnam(s); - } return r; } Modified: trunk/busybox/loginutils/passwd.c =================================================================== --- trunk/busybox/loginutils/passwd.c 2006-12-01 17:49:42 UTC (rev 16746) +++ trunk/busybox/loginutils/passwd.c 2006-12-01 21:34:20 UTC (rev 16747) @@ -52,8 +52,7 @@ } -static char* new_password(const struct passwd *pw, const char *old_crypted, - uid_t myuid, int algo) +static char* new_password(const struct passwd *pw, uid_t myuid, int algo) { char salt[sizeof("$N$XXXXXXXX")]; /* "$N$XXXXXXXX" or "XX" */ char *orig = ""; @@ -62,12 +61,12 @@ char *cp = NULL; char *ret = NULL; /* failure so far */ - if (myuid && old_crypted[0]) { + if (myuid && pw->pw_passwd[0]) { orig = bb_askpass(0, "Old password:"); /* returns ptr to static */ if (!orig) goto err_ret; - cipher = pw_encrypt(orig, old_crypted); /* returns ptr to static */ - if (strcmp(cipher, old_crypted) != 0) { + cipher = pw_encrypt(orig, pw->pw_passwd); /* returns ptr to static */ + if (strcmp(cipher, pw->pw_passwd) != 0) { syslog(LOG_WARNING, "incorrect password for '%s'", pw->pw_name); bb_do_delay(FAIL_DELAY); @@ -76,23 +75,19 @@ } } orig = xstrdup(orig); /* or else bb_askpass() will destroy it */ - newp = bb_askpass(0, "Enter the new password (minimum of 5 characters).\n" - "Please use a combination of upper and lower case letters and numbers.\n" - "Enter new password:"); /* returns ptr to static */ + newp = bb_askpass(0, "New password:"); /* returns ptr to static */ if (!newp) goto err_ret; newp = xstrdup(newp); /* we are going to bb_askpass() again, so save it */ - if (obscure(orig, newp, pw)) { - if (myuid) + if (obscure(orig, newp, pw) && myuid) { goto err_ret; /* non-root is not allowed to have weak passwd */ - puts("\nWarning: weak password (continuing)"); } - cp = bb_askpass(0, "Re-enter new password:"); + cp = bb_askpass(0, "Retype password:"); if (!cp) goto err_ret; if (strcmp(cp, newp)) { - puts("Passwords do not match"); + puts("Passwords don't match"); goto err_ret; } @@ -116,15 +111,6 @@ } -static void set_filesize_limit(int blocks) -{ - struct rlimit rlimit_fsize; - - rlimit_fsize.rlim_cur = rlimit_fsize.rlim_max = 512L * blocks; - setrlimit(RLIMIT_FSIZE, &rlimit_fsize); -} - - #if 0 static int get_algo(char *a) { @@ -152,6 +138,7 @@ int i; int ret = 1; /* failure */ + logmode = LOGMODE_STDIO; /* New passwd file, "/etc/passwd+" for now */ new_name = xasprintf("%s+", filename); last_char = &new_name[strlen(new_name)-1]; @@ -239,6 +226,7 @@ free_mem: if (ENABLE_FEATURE_CLEAN_UP) free(new_name); if (ENABLE_FEATURE_CLEAN_UP) free((char*)username); + logmode = LOGMODE_BOTH; return ret; } @@ -252,19 +240,21 @@ OPT_delete = 0x8, /* -d - delete password */ OPT_lud = 0xe, STATE_ALGO_md5 = 0x10, - /*STATE_ALGO_des = 0x20, not yet needed */ + /*STATE_ALGO_des = 0x20, not needed yet */ }; unsigned opt; char *opt_a = ""; const char *filename; char *myname; char *name; - char *oldp; - char *newp = NULL; /* gcc happiness */ - const struct passwd *pw; + char *newp; + struct passwd *pw; uid_t myuid; + struct rlimit rlimit_fsize; + char c; - openlog("passwd", LOG_NOWAIT, LOG_AUTH); + logmode = LOGMODE_BOTH; + openlog(applet_name, LOG_NOWAIT, LOG_AUTH); opt = getopt32(argc, argv, "a:lud", &opt_a); argc -= optind; argv += optind; @@ -278,71 +268,75 @@ bb_show_usage(); myname = xstrdup(bb_getpwuid(NULL, myuid, -1)); - name = myname; - if (argc) name = argv[0]; + name = argc ? argv[0] : myname; pw = getpwnam(name); if (!pw) bb_error_msg_and_die("unknown user %s", name); if (myuid && pw->pw_uid != myuid) { - syslog(LOG_WARNING, "can't change pwd for '%s'", name); - bb_error_msg_and_die("permission denied"); + /* LOGMODE_BOTH */ + bb_error_msg_and_die("%s can't change password for %s", myname, name); } filename = bb_path_passwd_file; - oldp = pw->pw_passwd; if (ENABLE_FEATURE_SHADOWPASSWDS) { struct spwd *sp = getspnam(name); if (!sp) { - bb_error_msg("no shadow record for user %s found, " - "changing ordinary password instead", name); + /* LOGMODE_BOTH */ + bb_error_msg("no record of %s in %s, using %s", + name, bb_path_shadow_file, + bb_path_passwd_file); } else { filename = bb_path_shadow_file; - oldp = sp->sp_pwdp; + pw->pw_passwd = sp->sp_pwdp; } } /* Decide what the new password will be */ + newp = NULL; + c = pw->pw_passwd[0] - '!'; if (!(opt & OPT_lud)) { - if (myuid) { - if (oldp[0] == '!') { - syslog(LOG_WARNING, "password locked for '%s'", name); - bb_error_msg_and_die("the password for %s cannot be changed", name); - } + if (myuid && !c) { /* passwd starts with '!' */ + /* LOGMODE_BOTH */ + bb_error_msg_and_die("cannot change " + "locked password for %s", name); } printf("Changing password for %s\n", name); - newp = new_password(pw, oldp, - myuid, - opt & STATE_ALGO_md5); + newp = new_password(pw, myuid, opt & STATE_ALGO_md5); if (!newp) { - bb_error_msg_and_die("the password for %s is unchanged", name); + logmode = LOGMODE_STDIO; + bb_error_msg_and_die("password for %s is unchanged", name); } } else if (opt & OPT_lock) { - if (oldp[0] == '!') goto skip; - newp = xasprintf("!%s", oldp); + if (!c) goto skip; /* passwd starts with '!' */ + newp = xasprintf("!%s", pw->pw_passwd); } else if (opt & OPT_unlock) { - if (oldp[0] != '!') goto skip; - newp = xstrdup(oldp + 1); + if (c) goto skip; /* not '!' */ + newp = xstrdup(&pw->pw_passwd[1]); } else if (opt & OPT_delete) { newp = xstrdup(""); } - set_filesize_limit(30000); + rlimit_fsize.rlim_cur = rlimit_fsize.rlim_max = 512L * 30000; + setrlimit(RLIMIT_FSIZE, &rlimit_fsize); signal(SIGHUP, SIG_IGN); signal(SIGINT, SIG_IGN); signal(SIGQUIT, SIG_IGN); umask(077); xsetuid(0); - if (update_passwd(filename, name, newp) == 0) { - syslog(LOG_INFO, "password for '%s' changed by user '%s'", name, - myname); - puts("Password changed"); - } else { - syslog(LOG_WARNING, "cannot update password file"); - bb_error_msg_and_die("cannot update password file"); + if (update_passwd(filename, name, newp) != 0) { + /* LOGMODE_BOTH */ + bb_error_msg_and_die("cannot update password file %s", + filename); } + /* LOGMODE_BOTH */ + bb_info_msg("Password for %s changed by %s", name, myname); if (ENABLE_FEATURE_CLEAN_UP) free(newp); skip: + if (!newp) { + bb_error_msg_and_die("password for %s is already %slocked", + name, (opt & OPT_unlock) ? "un" : ""); + } if (ENABLE_FEATURE_CLEAN_UP) free(myname); return 0; } From vda at busybox.net Sat Dec 2 09:58:11 2006 From: vda at busybox.net (vda at busybox.net) Date: Sat, 2 Dec 2006 09:58:11 -0800 (PST) Subject: svn commit: trunk/busybox: editors libbb Message-ID: <20061202175811.0DD23485C2@busybox.net> Author: vda Date: 2006-12-02 09:58:10 -0800 (Sat, 02 Dec 2006) New Revision: 16754 Log: sed: improve handling of NULs Modified: trunk/busybox/editors/sed.c trunk/busybox/libbb/get_line_from_file.c Changeset: Modified: trunk/busybox/editors/sed.c =================================================================== --- trunk/busybox/editors/sed.c 2006-12-02 17:03:38 UTC (rev 16753) +++ trunk/busybox/editors/sed.c 2006-12-02 17:58:10 UTC (rev 16754) @@ -80,8 +80,8 @@ /* Bitfields (gcc won't group them if we don't) */ unsigned int invert:1; /* the '!' after the address */ unsigned int in_match:1; /* Next line also included in match? */ - unsigned int no_newline:1; /* Last line written by (sw) had no '\n' */ unsigned int sub_p:1; /* (s) print option */ + int last_char; /* Last line written by (sw) had no '\n' */ /* GENERAL FIELDS */ char cmd; /* The command char: abcdDgGhHilnNpPqrstwxy:={} */ @@ -712,56 +712,73 @@ /* Get next line of input from bbg.input_file_list, flushing append buffer and * noting if we ran out of files without a newline on the last line we read. */ -static char *get_next_line(int *no_newline) +static char *get_next_line(int *last_char) { char *temp = NULL; int len; + /* will be returned if last line in the file + * doesn't end with either '\n' or '\0' */ + *last_char = 0x100; + flush_append(); while (bbg.current_input_file < bbg.input_file_count) { - temp = bb_get_chunk_from_file(bbg.input_file_list[bbg.current_input_file],&len); + temp = bb_get_chunk_from_file( + bbg.input_file_list[bbg.current_input_file], &len); if (temp) { - *no_newline = !(len && temp[len-1] == '\n'); - if (!*no_newline) temp[len-1] = 0; + /* len > 0 here, it's ok to do temp[len-1] */ + char c = temp[len-1]; + if (c == '\n' || c == '\0') { + temp[len-1] = '\0'; + *last_char = (unsigned char)c; + } break; - // Close this file and advance to next one - } else - fclose(bbg.input_file_list[bbg.current_input_file++]); + } + /* Close this file and advance to next one */ + fclose(bbg.input_file_list[bbg.current_input_file++]); } - return temp; } -/* Output line of text. missing_newline means the last line output did not - end with a newline. no_newline means this line does not end with a - newline. */ +/* Output line of text. */ +/* Note: + * echo -n thingy >z1 + * echo -n again >z2 + * >znull + * sed "s/i/z/" z1 z2 znull | hexdump -vC output: + * gnu sed 4.1.5: + * 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| + * bbox: + * 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| + * I am not sure that bbox is wrong here... + */ -static int puts_maybe_newline(char *s, FILE *file, int missing_newline, int no_newline) +static int puts_maybe_newline(char *s, FILE *file, int prev_last_char, int last_char) { - if (missing_newline) fputc('\n',file); - fputs(s,file); - if (!no_newline) fputc('\n',file); + fputs(s, file); + if (last_char < 0x100) fputc(last_char, file); if (ferror(file)) { xfunc_error_retval = 4; /* It's what gnu sed exits with... */ bb_error_msg_and_die(bb_msg_write_error); } - return no_newline; + return last_char; } -#define sed_puts(s,n) missing_newline=puts_maybe_newline(s,bbg.nonstdout,missing_newline,n) +#define sed_puts(s, n) \ + (prev_last_char = puts_maybe_newline(s, bbg.nonstdout, prev_last_char, n)) /* Process all the lines in all the files */ static void process_files(void) { char *pattern_space, *next_line; - int linenum = 0, missing_newline = 0; - int no_newline,next_no_newline = 0; + int linenum = 0, prev_last_char = 0; + int last_char, next_last_char = 0; /* Prime the pump */ - next_line = get_next_line(&next_no_newline); + next_line = get_next_line(&next_last_char); /* go through every line in each file */ for (;;) { @@ -771,11 +788,11 @@ /* Advance to next line. Stop if out of lines. */ pattern_space = next_line; if (!pattern_space) break; - no_newline = next_no_newline; + last_char = next_last_char; /* Read one line in advance so we can act on the last line, * the '$' address */ - next_line = get_next_line(&next_no_newline); + next_line = get_next_line(&next_last_char); linenum++; restart: /* for every line, go through all the commands */ @@ -858,7 +875,7 @@ /* Write the current pattern space to output */ case 'p': - sed_puts(pattern_space,no_newline); + sed_puts(pattern_space, last_char); break; /* Delete up through first newline */ case 'D': @@ -878,25 +895,24 @@ /* Substitute with regex */ case 's': - if (do_subst_command(sed_cmd, &pattern_space)) { - substituted |= 1; + if (!do_subst_command(sed_cmd, &pattern_space)) + break; + substituted |= 1; - /* handle p option */ - if (sed_cmd->sub_p) - sed_puts(pattern_space,no_newline); - /* handle w option */ - if (sed_cmd->file) - sed_cmd->no_newline = puts_maybe_newline(pattern_space, sed_cmd->file, sed_cmd->no_newline, no_newline); - - } + /* handle p option */ + if (sed_cmd->sub_p) + sed_puts(pattern_space, last_char); + /* handle w option */ + if (sed_cmd->file) + sed_cmd->last_char = puts_maybe_newline( + pattern_space, sed_cmd->file, + sed_cmd->last_char, last_char); break; /* Append line to linked list to be printed later */ case 'a': - { append(sed_cmd->string); break; - } /* Insert text before this line */ case 'i': @@ -930,18 +946,20 @@ /* Write pattern space to file. */ case 'w': - sed_cmd->no_newline = puts_maybe_newline(pattern_space,sed_cmd->file, sed_cmd->no_newline,no_newline); + sed_cmd->last_char = puts_maybe_newline( + pattern_space,sed_cmd->file, + sed_cmd->last_char, last_char); break; /* Read next line from input */ case 'n': if (!bbg.be_quiet) - sed_puts(pattern_space,no_newline); + sed_puts(pattern_space, last_char); if (next_line) { free(pattern_space); pattern_space = next_line; - no_newline = next_no_newline; - next_line = get_next_line(&next_no_newline); + last_char = next_last_char; + next_line = get_next_line(&next_last_char); linenum++; break; } @@ -970,8 +988,8 @@ pattern_space = realloc(pattern_space, len + strlen(next_line) + 2); pattern_space[len] = '\n'; strcpy(pattern_space + len+1, next_line); - no_newline = next_no_newline; - next_line = get_next_line(&next_no_newline); + last_char = next_last_char; + next_line = get_next_line(&next_last_char); linenum++; } break; @@ -1029,7 +1047,7 @@ strcat(pattern_space, "\n"); if (bbg.hold_space) strcat(pattern_space, bbg.hold_space); - no_newline = 0; + last_char = 0x100; break; } @@ -1061,7 +1079,7 @@ { char *tmp = pattern_space; pattern_space = bbg.hold_space ? : xzalloc(1); - no_newline = 0; + last_char = 0x100; bbg.hold_space = tmp; break; } @@ -1075,7 +1093,7 @@ discard_commands: /* we will print the line unless we were told to be quiet ('-n') or if the line was suppressed (ala 'd'elete) */ - if (!bbg.be_quiet) sed_puts(pattern_space,no_newline); + if (!bbg.be_quiet) sed_puts(pattern_space, last_char); /* Delete and such jump here. */ discard_line: Modified: trunk/busybox/libbb/get_line_from_file.c =================================================================== --- trunk/busybox/libbb/get_line_from_file.c 2006-12-02 17:03:38 UTC (rev 16753) +++ trunk/busybox/libbb/get_line_from_file.c 2006-12-02 17:58:10 UTC (rev 16754) @@ -11,8 +11,8 @@ #include "libbb.h" -/* This function reads an entire line from a text file, - * up to a newline or NUL byte. It returns a malloc'ed char * which must be +/* This function reads an entire line from a text file, up to a newline + * or NUL byte, inclusive. It returns a malloc'ed char * which must be * stored and free'ed by the caller. If end is null '\n' isn't considered * end of line. If end isn't null, length of the chunk read is stored in it. */ @@ -25,7 +25,7 @@ while ((ch = getc(file)) != EOF) { /* grow the line buffer as necessary */ - if (idx > linebufsz - 2) { + if (idx >= linebufsz) { linebuf = xrealloc(linebuf, linebufsz += 80); } linebuf[idx++] = (char) ch; @@ -35,14 +35,14 @@ if (end) *end = idx; if (linebuf) { - // huh, is fgets discards prior data on error like this? + // huh, does fgets discard prior data on error like this? // I don't think so.... //if (ferror(file)) { // free(linebuf); // return NULL; //} linebuf = xrealloc(linebuf, idx+1); - linebuf[idx] = 0; + linebuf[idx] = '\0'; } return linebuf; } From bugs at busybox.net Sat Dec 2 09:59:02 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Sat, 2 Dec 2006 09:59:02 -0800 Subject: [BusyBox 0000615]: sed convert 0x00 to 0x0a Message-ID: A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=615 ====================================================================== Reported By: robang74 Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 615 Category: Other Reproducibility: always Severity: major Priority: normal Status: assigned ====================================================================== Date Submitted: 12-28-2005 01:26 PST Last Modified: 12-02-2006 09:59 PST ====================================================================== Summary: sed convert 0x00 to 0x0a Description: bash-3.00# dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out bash-3.00# echo ciao>>/tmp/test bash-3.00# dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out bash-3.00# cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 bash-3.00# wc -l /tmp/test 1 /tmp/test bash-3.00# wc -l /tmp/test2 2048 /tmp/test2 ====================================================================== ---------------------------------------------------------------------- robang74 - 12-28-05 03:18 ---------------------------------------------------------------------- sed loose the last 0x00 if it exist: bash-3.00# ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:16 /tmp/test -rw-r--r-- 1 0 0 2052 Dec 28 11:16 /tmp/test2 AFTER PATCH: / # dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out / # echo ciao>>/tmp/test / # dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out / # cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 / # wc -l /tmp/test2 1 /tmp/test2 / # wc -l /tmp/test 1 /tmp/test / # ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test2 ---------------------------------------------------------------------- landley - 01-01-06 22:47 ---------------------------------------------------------------------- Hmmm... Tricky dealing with embedded nulls when C usually considers null to be an end of string indicator. Your fix causes problems in other contexts. It was outputting a newline because there are times when that is appropriate, and this would output a null then. Something like: echo -n thingy > one echo -n again > two sed "s/i/z/" one two > three The output should be "thzngy\nagazn" and I think it would be "thzngy\0agazn". Not that I've tested it just now. I'm trying to get 1.1.0 out this friday. Not sure I'll get to this before then... ---------------------------------------------------------------------- robang74 - 01-02-06 01:25 ---------------------------------------------------------------------- patch n.2 fix the problem of newline in more files [roberto at wsraf big]$ patch -p0 < sed_2.patch patching file busybox-1.01/editors/sed.c [roberto at wsraf busybox-1.01]$ make menuconfig && make [roberto at wsraf busybox-1.01]$ echo -n thingy > one [roberto at wsraf busybox-1.01]$ echo -n again > two [roberto at wsraf busybox-1.01]$ sed "s/i/z/" one two > three [roberto at wsraf busybox-1.01]$ ./busybox sed "s/i/z/" one two > four [roberto at wsraf busybox-1.01_sed2]$ cat three thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ cat four thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ ---------------------------------------------------------------------- landley - 01-08-06 11:34 ---------------------------------------------------------------------- I tried a slightly cleaned up version of the second patch and it caused two more busybox tests to fail. To see the specific failures, cd testsuite and "./runtest -v sed". I might get around to looking at it some more today, if so I'll apply the result and close out the bug. Rob ---------------------------------------------------------------------- robang74 - 01-09-06 03:30 ---------------------------------------------------------------------- AFTER PATCH n.3 sed fails the same testa as before [roberto at wsraf testsuite]$ ./runtest sed | grep FAIL FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) BUT passes the new one [roberto at wsraf testsuite]$ ./runtest sed | grep binary PASS: sed s onto a binary input (with zeros) ---------------------------------------------------------------------- robang74 - 01-09-06 10:15 ---------------------------------------------------------------------- svn 13198 with sed_3 patch applied, defconfig [roberto at wsraf busybox_sed3]$ size busybox text data bss dec hex filename 236511 2220 28484 267215 413cf busybox svn 13201 original, defconfig [roberto at wsraf busybox]$ size busybox text data bss dec hex filename 236527 2156 28548 267231 413df busybox Between 13198 and 13201 size grows 15 bytes more tham sed_3 patch, at least on my WS but I have gcc 4.0 (good or bad, so it is). ---------------------------------------------------------------------- robang74 - 02-28-06 03:09 ---------------------------------------------------------------------- Patch n.4 applies to svn 14360, it comes in two flowers: norename: which is not include a variable rename s/lastchar/no_newline/g and the patch itself with the variable rename for improved code readibility Switching from the patch to the norename one is pretty simple (automatic): svn co ... cp -af busybox busybox.orig cat sed_4-svn14360.patch | \ sed -e "s/lastchar/no_newline/g" >sed_4-svn14360_tmp.patch patch -p0 sed_4-svn14360_norename.patch ---------------------------------------------------------------------- landley - 03-01-06 07:49 ---------------------------------------------------------------------- Did you notice the change to get_line_from_file.c you reverted, and the change to sed.c (and for that matter, sort.c) that matched that? The way you were appending the length at an arbitrary aligment would confuse platforms that don't like unaligned access, plus you hard-wired in an assumption that a pointer is 4 bytes long (not true for 64 bit platforms). ---------------------------------------------------------------------- landley - 03-01-06 07:59 ---------------------------------------------------------------------- I applied the .4-norename patch, reverted the get_line_from_file part, and made the two line fix to have get_chunk_from_file() set len. It built fine, and then I ran the sed regression test suite. Failed spectacularly. It's appending an extra blank line to most things, and in at least one case a garbage character... ---------------------------------------------------------------------- robang74 - 03-02-06 00:50 ---------------------------------------------------------------------- > Did you notice the change to get_line_from_file.c you reverted, and the > change to sed.c (and for that matter, sort.c) that matched that? The way > you were appending the length at an arbitrary aligment would confuse > platforms that don't like unaligned access, plus you hard-wired in an > assumption that a pointer is 4 bytes long (not true for 64 bit platforms). not pointers but integer, anyway to fix 64 bit platform s/-6/-2-sizeof(int)/ every platforms ignore what is wroten after \0 so appending information after the null chars is ok for me. If am wrong it necessary to report lastchar back to the caller... so change the API get_line_from_file( ..., &lastchar); > I applied the .4-norename patch, reverted the get_line_from_file part, and > made the two line fix to have get_chunk_from_file() set len. It built > fine, and then I ran the sed regression test suite. > > Failed spectacularly. It's appending an extra blank line to most things, > and in at least one case a garbage character... I know. Because you have reverted the get_chunk_from_file() part. That is the reason because that part is needed. ---------------------------------------------------------------------- robang74 - 03-02-06 01:44 ---------------------------------------------------------------------- PATCH n.5 resolve the get_line_from_file.c reverted issue: it takes get_line_from_file.c as near as possible to the original one from svn 14360 ---------------------------------------------------------------------- robang74 - 03-03-06 00:44 ---------------------------------------------------------------------- PATCH n.6 as suggested by Rob does not need to modify get_line_from_file.c ---------------------------------------------------------------------- robang74 - 03-04-06 07:11 ---------------------------------------------------------------------- patch -p0 YES cd busybox/testsuite [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | tail -n1 FAIL: sed zero support [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | wc -l 7 <--- same 6 + 1 (my test s/ciao/miao/ on 000ciao\n000) cd ../.. patch -p0 /dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed embedded NUL FAIL: sed embedded NUL g FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] After applaying patch: [roberto at GEDX0327 testsuite]$ pwd; ./sed.tests 2>/dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1_sed/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] Manual test after applaying patch dd if=/dev/zero bs=1k count=1 >/tmp/test; echo ciao>>/tmp/test; dd if=/dev/zero bs=1k count=1 >>/tmp/test; cat /tmp/test | _install/bin/busybox sed -e "s/ciao/miao/g" >/tmp/test2; wc -l /tmp/test; wc -l /tmp/test2; hexdump /tmp/test; hexdump /tmp/test2 entrati 1+0 record usciti 1+0 record entrati 1+0 record usciti 1+0 record 1 /tmp/test 1 /tmp/test2 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 6963 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 696d 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 ---------------------------------------------------------------------- robang74 - 11-30-06 04:13 ---------------------------------------------------------------------- busybox-20061130_sed.patch: apply to 20061130 snapshot version ---------------------------------------------------------------------- vda - 12-02-06 09:59 ---------------------------------------------------------------------- #!/bin/sh { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | ./busybox sed -e "s/ciao/miao/g" | hexdump -vC { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | /usr/bin/sed -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n again >z2 >znull ./busybox sed "s/i/z/" z1 z2 znull | hexdump -vC /usr/bin/sed "s/i/z/" z1 z2 znull | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | ./busybox sed -e 's/woo/bang/' | hexdump -vC echo -ne "\0bang\0woo\0" | /usr/bin/sed -e 's/woo/bang/' | hexdump -vC unpatched busybox: 00000000 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |................| 00000010 6d 69 61 6f 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |miao............| 00000020 0a 0a 0a 0a |....| 00000024 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 0a 62 61 6e 67 0a 62 61 6e 67 |.bang.bang| 0000000a 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b Patched per revision 16754: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| 0000000b 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b Issue History Date Modified Username Field Change ====================================================================== 12-28-05 01:26 robang74 New Issue 12-28-05 01:26 robang74 Status new => assigned 12-28-05 01:26 robang74 Assigned To => BusyBox 12-28-05 01:26 robang74 Issue Monitored: robang74 12-28-05 03:18 robang74 Note Added: 0000821 12-28-05 03:18 robang74 File Added: sed.diff 01-01-06 22:47 landley Note Added: 0000835 01-02-06 01:25 robang74 Note Added: 0000836 01-02-06 01:25 robang74 File Added: sed_2.patch 01-08-06 11:34 landley Note Added: 0000873 01-09-06 03:29 robang74 Note Added: 0000886 01-09-06 03:30 robang74 Note Edited: 0000886 01-09-06 03:54 robang74 File Added: sed_3.patch 01-09-06 10:15 robang74 Note Added: 0000890 02-28-06 03:05 robang74 File Added: sed_4-svn14360.patch 02-28-06 03:05 robang74 File Added: sed_4-svn14360_norename.patch 02-28-06 03:09 robang74 Note Added: 0001148 03-01-06 07:49 landley Note Added: 0001154 03-01-06 07:59 landley Note Added: 0001155 03-02-06 00:33 robang74 Note Added: 0001156 03-02-06 00:50 robang74 Note Edited: 0001156 03-02-06 01:40 robang74 File Added: sed_5-svn14360.patch 03-02-06 01:41 robang74 File Added: sed_5-svn14360_norename.patch 03-02-06 01:43 robang74 Note Added: 0001157 03-02-06 01:44 robang74 Note Edited: 0001157 03-03-06 00:43 robang74 File Added: sed_6-svn14360.patch 03-03-06 00:43 robang74 File Added: sed_6-svn14360_norename.patch 03-03-06 00:44 robang74 Note Added: 0001160 03-04-06 07:11 robang74 Note Added: 0001161 03-04-06 07:12 robang74 File Added: busybox.14360_sed7_testsuite.patch 03-04-06 07:12 robang74 File Added: busybox.14360_sed7.patch 03-04-06 09:09 robang74 Note Added: 0001162 03-04-06 09:10 robang74 File Added: busybox.14360_sed8_testsuite.patch 03-04-06 09:10 robang74 File Added: busybox.14360_sed8.patch 03-05-06 18:57 robang74 File Added: busybox.14453_sed9.patch 03-05-06 18:58 robang74 Note Added: 0001164 03-14-06 07:39 robang74 File Added: busybox.14536_sed9.patch 11-30-06 01:05 robang74 Note Added: 0001841 11-30-06 01:07 robang74 File Added: busybox-1.2.2.1_sed.patch 11-30-06 04:12 robang74 File Added: busybox-20061130_sed.patch 11-30-06 04:13 robang74 Note Added: 0001843 12-02-06 09:59 vda Note Added: 0001853 ====================================================================== From vda at busybox.net Sat Dec 2 12:12:12 2006 From: vda at busybox.net (vda at busybox.net) Date: Sat, 2 Dec 2006 12:12:12 -0800 (PST) Subject: svn commit: trunk/busybox/editors Message-ID: <20061202201212.D49AB48653@busybox.net> Author: vda Date: 2006-12-02 12:12:12 -0800 (Sat, 02 Dec 2006) New Revision: 16770 Log: sed: fix handling of files not ending in '\n' Modified: trunk/busybox/editors/sed.c Changeset: Modified: trunk/busybox/editors/sed.c =================================================================== --- trunk/busybox/editors/sed.c 2006-12-02 19:36:29 UTC (rev 16769) +++ trunk/busybox/editors/sed.c 2006-12-02 20:12:12 UTC (rev 16770) @@ -81,6 +81,7 @@ unsigned int invert:1; /* the '!' after the address */ unsigned int in_match:1; /* Next line also included in match? */ unsigned int sub_p:1; /* (s) print option */ + int last_char; /* Last line written by (sw) had no '\n' */ /* GENERAL FIELDS */ @@ -715,12 +716,9 @@ static char *get_next_line(int *last_char) { char *temp = NULL; - int len; + int len, lc; - /* will be returned if last line in the file - * doesn't end with either '\n' or '\0' */ - *last_char = 0x100; - + lc = 0; flush_append(); while (bbg.current_input_file < bbg.input_file_count) { temp = bb_get_chunk_from_file( @@ -730,18 +728,27 @@ char c = temp[len-1]; if (c == '\n' || c == '\0') { temp[len-1] = '\0'; - *last_char = (unsigned char)c; + lc |= (unsigned char)c; + break; } + /* will be returned if last line in the file + * doesn't end with either '\n' or '\0' */ + lc |= 0x100; break; } /* Close this file and advance to next one */ fclose(bbg.input_file_list[bbg.current_input_file++]); + /* "this is the first line from new input file" */ + lc |= 0x200; } + *last_char = lc; return temp; } /* Output line of text. */ /* Note: + * The tricks with 0x200 and last_puts_char are there to emulate gnu sed. + * Without them, we had this: * echo -n thingy >z1 * echo -n again >z2 * >znull @@ -750,13 +757,26 @@ * 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| * bbox: * 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| - * I am not sure that bbox is wrong here... */ static int puts_maybe_newline(char *s, FILE *file, int prev_last_char, int last_char) { + static char last_puts_char; + + /* Is this a first line from new file + * and old file didn't end with '\n'? */ + if ((last_char & 0x200) && last_puts_char != '\n') { + fputc('\n', file); + last_puts_char = '\n'; + } fputs(s, file); - if (last_char < 0x100) fputc(last_char, file); + /* 'x': we don't care what is it, but we know it isn't '\n' */ + if (s[0]) last_puts_char = 'x'; + if (!(last_char & 0x100)) { /* had trailing '\n' or '\0'? */ + last_char &= 0xff; + fputc(last_char, file); + last_puts_char = last_char; + } if (ferror(file)) { xfunc_error_retval = 4; /* It's what gnu sed exits with... */ @@ -1047,7 +1067,7 @@ strcat(pattern_space, "\n"); if (bbg.hold_space) strcat(pattern_space, bbg.hold_space); - last_char = 0x100; + last_char = '\n'; break; } @@ -1079,7 +1099,7 @@ { char *tmp = pattern_space; pattern_space = bbg.hold_space ? : xzalloc(1); - last_char = 0x100; + last_char = '\n'; bbg.hold_space = tmp; break; } From bugs at busybox.net Sat Dec 2 12:14:17 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Sat, 2 Dec 2006 12:14:17 -0800 Subject: [BusyBox 0000615]: sed convert 0x00 to 0x0a Message-ID: <05b8c49243566a0933dd20048f3452f9@bugs.busybox.net> A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=615 ====================================================================== Reported By: robang74 Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 615 Category: Other Reproducibility: always Severity: major Priority: normal Status: assigned ====================================================================== Date Submitted: 12-28-2005 01:26 PST Last Modified: 12-02-2006 12:14 PST ====================================================================== Summary: sed convert 0x00 to 0x0a Description: bash-3.00# dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out bash-3.00# echo ciao>>/tmp/test bash-3.00# dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out bash-3.00# cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 bash-3.00# wc -l /tmp/test 1 /tmp/test bash-3.00# wc -l /tmp/test2 2048 /tmp/test2 ====================================================================== ---------------------------------------------------------------------- robang74 - 12-28-05 03:18 ---------------------------------------------------------------------- sed loose the last 0x00 if it exist: bash-3.00# ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:16 /tmp/test -rw-r--r-- 1 0 0 2052 Dec 28 11:16 /tmp/test2 AFTER PATCH: / # dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out / # echo ciao>>/tmp/test / # dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out / # cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 / # wc -l /tmp/test2 1 /tmp/test2 / # wc -l /tmp/test 1 /tmp/test / # ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test2 ---------------------------------------------------------------------- landley - 01-01-06 22:47 ---------------------------------------------------------------------- Hmmm... Tricky dealing with embedded nulls when C usually considers null to be an end of string indicator. Your fix causes problems in other contexts. It was outputting a newline because there are times when that is appropriate, and this would output a null then. Something like: echo -n thingy > one echo -n again > two sed "s/i/z/" one two > three The output should be "thzngy\nagazn" and I think it would be "thzngy\0agazn". Not that I've tested it just now. I'm trying to get 1.1.0 out this friday. Not sure I'll get to this before then... ---------------------------------------------------------------------- robang74 - 01-02-06 01:25 ---------------------------------------------------------------------- patch n.2 fix the problem of newline in more files [roberto at wsraf big]$ patch -p0 < sed_2.patch patching file busybox-1.01/editors/sed.c [roberto at wsraf busybox-1.01]$ make menuconfig && make [roberto at wsraf busybox-1.01]$ echo -n thingy > one [roberto at wsraf busybox-1.01]$ echo -n again > two [roberto at wsraf busybox-1.01]$ sed "s/i/z/" one two > three [roberto at wsraf busybox-1.01]$ ./busybox sed "s/i/z/" one two > four [roberto at wsraf busybox-1.01_sed2]$ cat three thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ cat four thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ ---------------------------------------------------------------------- landley - 01-08-06 11:34 ---------------------------------------------------------------------- I tried a slightly cleaned up version of the second patch and it caused two more busybox tests to fail. To see the specific failures, cd testsuite and "./runtest -v sed". I might get around to looking at it some more today, if so I'll apply the result and close out the bug. Rob ---------------------------------------------------------------------- robang74 - 01-09-06 03:30 ---------------------------------------------------------------------- AFTER PATCH n.3 sed fails the same testa as before [roberto at wsraf testsuite]$ ./runtest sed | grep FAIL FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) BUT passes the new one [roberto at wsraf testsuite]$ ./runtest sed | grep binary PASS: sed s onto a binary input (with zeros) ---------------------------------------------------------------------- robang74 - 01-09-06 10:15 ---------------------------------------------------------------------- svn 13198 with sed_3 patch applied, defconfig [roberto at wsraf busybox_sed3]$ size busybox text data bss dec hex filename 236511 2220 28484 267215 413cf busybox svn 13201 original, defconfig [roberto at wsraf busybox]$ size busybox text data bss dec hex filename 236527 2156 28548 267231 413df busybox Between 13198 and 13201 size grows 15 bytes more tham sed_3 patch, at least on my WS but I have gcc 4.0 (good or bad, so it is). ---------------------------------------------------------------------- robang74 - 02-28-06 03:09 ---------------------------------------------------------------------- Patch n.4 applies to svn 14360, it comes in two flowers: norename: which is not include a variable rename s/lastchar/no_newline/g and the patch itself with the variable rename for improved code readibility Switching from the patch to the norename one is pretty simple (automatic): svn co ... cp -af busybox busybox.orig cat sed_4-svn14360.patch | \ sed -e "s/lastchar/no_newline/g" >sed_4-svn14360_tmp.patch patch -p0 sed_4-svn14360_norename.patch ---------------------------------------------------------------------- landley - 03-01-06 07:49 ---------------------------------------------------------------------- Did you notice the change to get_line_from_file.c you reverted, and the change to sed.c (and for that matter, sort.c) that matched that? The way you were appending the length at an arbitrary aligment would confuse platforms that don't like unaligned access, plus you hard-wired in an assumption that a pointer is 4 bytes long (not true for 64 bit platforms). ---------------------------------------------------------------------- landley - 03-01-06 07:59 ---------------------------------------------------------------------- I applied the .4-norename patch, reverted the get_line_from_file part, and made the two line fix to have get_chunk_from_file() set len. It built fine, and then I ran the sed regression test suite. Failed spectacularly. It's appending an extra blank line to most things, and in at least one case a garbage character... ---------------------------------------------------------------------- robang74 - 03-02-06 00:50 ---------------------------------------------------------------------- > Did you notice the change to get_line_from_file.c you reverted, and the > change to sed.c (and for that matter, sort.c) that matched that? The way > you were appending the length at an arbitrary aligment would confuse > platforms that don't like unaligned access, plus you hard-wired in an > assumption that a pointer is 4 bytes long (not true for 64 bit platforms). not pointers but integer, anyway to fix 64 bit platform s/-6/-2-sizeof(int)/ every platforms ignore what is wroten after \0 so appending information after the null chars is ok for me. If am wrong it necessary to report lastchar back to the caller... so change the API get_line_from_file( ..., &lastchar); > I applied the .4-norename patch, reverted the get_line_from_file part, and > made the two line fix to have get_chunk_from_file() set len. It built > fine, and then I ran the sed regression test suite. > > Failed spectacularly. It's appending an extra blank line to most things, > and in at least one case a garbage character... I know. Because you have reverted the get_chunk_from_file() part. That is the reason because that part is needed. ---------------------------------------------------------------------- robang74 - 03-02-06 01:44 ---------------------------------------------------------------------- PATCH n.5 resolve the get_line_from_file.c reverted issue: it takes get_line_from_file.c as near as possible to the original one from svn 14360 ---------------------------------------------------------------------- robang74 - 03-03-06 00:44 ---------------------------------------------------------------------- PATCH n.6 as suggested by Rob does not need to modify get_line_from_file.c ---------------------------------------------------------------------- robang74 - 03-04-06 07:11 ---------------------------------------------------------------------- patch -p0 YES cd busybox/testsuite [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | tail -n1 FAIL: sed zero support [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | wc -l 7 <--- same 6 + 1 (my test s/ciao/miao/ on 000ciao\n000) cd ../.. patch -p0 /dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed embedded NUL FAIL: sed embedded NUL g FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] After applaying patch: [roberto at GEDX0327 testsuite]$ pwd; ./sed.tests 2>/dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1_sed/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] Manual test after applaying patch dd if=/dev/zero bs=1k count=1 >/tmp/test; echo ciao>>/tmp/test; dd if=/dev/zero bs=1k count=1 >>/tmp/test; cat /tmp/test | _install/bin/busybox sed -e "s/ciao/miao/g" >/tmp/test2; wc -l /tmp/test; wc -l /tmp/test2; hexdump /tmp/test; hexdump /tmp/test2 entrati 1+0 record usciti 1+0 record entrati 1+0 record usciti 1+0 record 1 /tmp/test 1 /tmp/test2 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 6963 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 696d 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 ---------------------------------------------------------------------- robang74 - 11-30-06 04:13 ---------------------------------------------------------------------- busybox-20061130_sed.patch: apply to 20061130 snapshot version ---------------------------------------------------------------------- vda - 12-02-06 09:59 ---------------------------------------------------------------------- #!/bin/sh { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | ./busybox sed -e "s/ciao/miao/g" | hexdump -vC { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | /usr/bin/sed -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n again >z2 >znull ./busybox sed "s/i/z/" z1 z2 znull | hexdump -vC /usr/bin/sed "s/i/z/" z1 z2 znull | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | ./busybox sed -e 's/woo/bang/' | hexdump -vC echo -ne "\0bang\0woo\0" | /usr/bin/sed -e 's/woo/bang/' | hexdump -vC unpatched busybox: 00000000 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |................| 00000010 6d 69 61 6f 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |miao............| 00000020 0a 0a 0a 0a |....| 00000024 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 0a 62 61 6e 67 0a 62 61 6e 67 |.bang.bang| 0000000a 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b Patched per revision 16754: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| 0000000b 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b ---------------------------------------------------------------------- vda - 12-02-06 12:14 ---------------------------------------------------------------------- Revision 16770 fixes that issue too. Tested with: #!/bin/sh function tst() { { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | $1 -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n thingy >z1 echo -ne "thingy\0" >z3 echo -ne "\0" >zzero >znull $1 "s/i/z/" z1 z2 z3 z1 znull z1 zzero zzero znull znull z1 | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | $1 -e 's/woo/bang/' | hexdump -vC } tst "./busybox sed" >x1 tst "/usr/bin/sed" >x2 diff -u x1 x2 >x.diff || { echo Different!; sleep 2; } Can I close ticket now? Issue History Date Modified Username Field Change ====================================================================== 12-28-05 01:26 robang74 New Issue 12-28-05 01:26 robang74 Status new => assigned 12-28-05 01:26 robang74 Assigned To => BusyBox 12-28-05 01:26 robang74 Issue Monitored: robang74 12-28-05 03:18 robang74 Note Added: 0000821 12-28-05 03:18 robang74 File Added: sed.diff 01-01-06 22:47 landley Note Added: 0000835 01-02-06 01:25 robang74 Note Added: 0000836 01-02-06 01:25 robang74 File Added: sed_2.patch 01-08-06 11:34 landley Note Added: 0000873 01-09-06 03:29 robang74 Note Added: 0000886 01-09-06 03:30 robang74 Note Edited: 0000886 01-09-06 03:54 robang74 File Added: sed_3.patch 01-09-06 10:15 robang74 Note Added: 0000890 02-28-06 03:05 robang74 File Added: sed_4-svn14360.patch 02-28-06 03:05 robang74 File Added: sed_4-svn14360_norename.patch 02-28-06 03:09 robang74 Note Added: 0001148 03-01-06 07:49 landley Note Added: 0001154 03-01-06 07:59 landley Note Added: 0001155 03-02-06 00:33 robang74 Note Added: 0001156 03-02-06 00:50 robang74 Note Edited: 0001156 03-02-06 01:40 robang74 File Added: sed_5-svn14360.patch 03-02-06 01:41 robang74 File Added: sed_5-svn14360_norename.patch 03-02-06 01:43 robang74 Note Added: 0001157 03-02-06 01:44 robang74 Note Edited: 0001157 03-03-06 00:43 robang74 File Added: sed_6-svn14360.patch 03-03-06 00:43 robang74 File Added: sed_6-svn14360_norename.patch 03-03-06 00:44 robang74 Note Added: 0001160 03-04-06 07:11 robang74 Note Added: 0001161 03-04-06 07:12 robang74 File Added: busybox.14360_sed7_testsuite.patch 03-04-06 07:12 robang74 File Added: busybox.14360_sed7.patch 03-04-06 09:09 robang74 Note Added: 0001162 03-04-06 09:10 robang74 File Added: busybox.14360_sed8_testsuite.patch 03-04-06 09:10 robang74 File Added: busybox.14360_sed8.patch 03-05-06 18:57 robang74 File Added: busybox.14453_sed9.patch 03-05-06 18:58 robang74 Note Added: 0001164 03-14-06 07:39 robang74 File Added: busybox.14536_sed9.patch 11-30-06 01:05 robang74 Note Added: 0001841 11-30-06 01:07 robang74 File Added: busybox-1.2.2.1_sed.patch 11-30-06 04:12 robang74 File Added: busybox-20061130_sed.patch 11-30-06 04:13 robang74 Note Added: 0001843 12-02-06 09:59 vda Note Added: 0001853 12-02-06 10:00 vda File Added: sed_rev16754.patch 12-02-06 12:14 vda Note Added: 0001854 ====================================================================== From aldot at busybox.net Sat Dec 2 12:47:36 2006 From: aldot at busybox.net (aldot at busybox.net) Date: Sat, 2 Dec 2006 12:47:36 -0800 (PST) Subject: svn commit: trunk/busybox/coreutils Message-ID: <20061202204736.D01D048593@busybox.net> Author: aldot Date: 2006-12-02 12:47:36 -0800 (Sat, 02 Dec 2006) New Revision: 16771 Log: - commentary typo Modified: trunk/busybox/coreutils/mknod.c Changeset: Modified: trunk/busybox/coreutils/mknod.c =================================================================== --- trunk/busybox/coreutils/mknod.c 2006-12-02 20:12:12 UTC (rev 16770) +++ trunk/busybox/coreutils/mknod.c 2006-12-02 20:47:36 UTC (rev 16771) @@ -32,7 +32,7 @@ dev = 0; if ((*name != 'p') && ((argc -= 2) == 2)) { - /* Autodetect what the system supports; thexe macros should + /* Autodetect what the system supports; these macros should * optimize out to two constants. */ dev = makedev(xatoul_range(argv[2], 0, major(UINT_MAX)), xatoul_range(argv[3], 0, minor(UINT_MAX))); From bugs at busybox.net Sat Dec 2 13:02:01 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Sat, 2 Dec 2006 13:02:01 -0800 Subject: [BusyBox 0000615]: sed convert 0x00 to 0x0a Message-ID: A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=615 ====================================================================== Reported By: robang74 Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 615 Category: Other Reproducibility: always Severity: major Priority: normal Status: assigned ====================================================================== Date Submitted: 12-28-2005 01:26 PST Last Modified: 12-02-2006 13:02 PST ====================================================================== Summary: sed convert 0x00 to 0x0a Description: bash-3.00# dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out bash-3.00# echo ciao>>/tmp/test bash-3.00# dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out bash-3.00# cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 bash-3.00# wc -l /tmp/test 1 /tmp/test bash-3.00# wc -l /tmp/test2 2048 /tmp/test2 ====================================================================== ---------------------------------------------------------------------- robang74 - 12-28-05 03:18 ---------------------------------------------------------------------- sed loose the last 0x00 if it exist: bash-3.00# ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:16 /tmp/test -rw-r--r-- 1 0 0 2052 Dec 28 11:16 /tmp/test2 AFTER PATCH: / # dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out / # echo ciao>>/tmp/test / # dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out / # cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 / # wc -l /tmp/test2 1 /tmp/test2 / # wc -l /tmp/test 1 /tmp/test / # ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test2 ---------------------------------------------------------------------- landley - 01-01-06 22:47 ---------------------------------------------------------------------- Hmmm... Tricky dealing with embedded nulls when C usually considers null to be an end of string indicator. Your fix causes problems in other contexts. It was outputting a newline because there are times when that is appropriate, and this would output a null then. Something like: echo -n thingy > one echo -n again > two sed "s/i/z/" one two > three The output should be "thzngy\nagazn" and I think it would be "thzngy\0agazn". Not that I've tested it just now. I'm trying to get 1.1.0 out this friday. Not sure I'll get to this before then... ---------------------------------------------------------------------- robang74 - 01-02-06 01:25 ---------------------------------------------------------------------- patch n.2 fix the problem of newline in more files [roberto at wsraf big]$ patch -p0 < sed_2.patch patching file busybox-1.01/editors/sed.c [roberto at wsraf busybox-1.01]$ make menuconfig && make [roberto at wsraf busybox-1.01]$ echo -n thingy > one [roberto at wsraf busybox-1.01]$ echo -n again > two [roberto at wsraf busybox-1.01]$ sed "s/i/z/" one two > three [roberto at wsraf busybox-1.01]$ ./busybox sed "s/i/z/" one two > four [roberto at wsraf busybox-1.01_sed2]$ cat three thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ cat four thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ ---------------------------------------------------------------------- landley - 01-08-06 11:34 ---------------------------------------------------------------------- I tried a slightly cleaned up version of the second patch and it caused two more busybox tests to fail. To see the specific failures, cd testsuite and "./runtest -v sed". I might get around to looking at it some more today, if so I'll apply the result and close out the bug. Rob ---------------------------------------------------------------------- robang74 - 01-09-06 03:30 ---------------------------------------------------------------------- AFTER PATCH n.3 sed fails the same testa as before [roberto at wsraf testsuite]$ ./runtest sed | grep FAIL FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) BUT passes the new one [roberto at wsraf testsuite]$ ./runtest sed | grep binary PASS: sed s onto a binary input (with zeros) ---------------------------------------------------------------------- robang74 - 01-09-06 10:15 ---------------------------------------------------------------------- svn 13198 with sed_3 patch applied, defconfig [roberto at wsraf busybox_sed3]$ size busybox text data bss dec hex filename 236511 2220 28484 267215 413cf busybox svn 13201 original, defconfig [roberto at wsraf busybox]$ size busybox text data bss dec hex filename 236527 2156 28548 267231 413df busybox Between 13198 and 13201 size grows 15 bytes more tham sed_3 patch, at least on my WS but I have gcc 4.0 (good or bad, so it is). ---------------------------------------------------------------------- robang74 - 02-28-06 03:09 ---------------------------------------------------------------------- Patch n.4 applies to svn 14360, it comes in two flowers: norename: which is not include a variable rename s/lastchar/no_newline/g and the patch itself with the variable rename for improved code readibility Switching from the patch to the norename one is pretty simple (automatic): svn co ... cp -af busybox busybox.orig cat sed_4-svn14360.patch | \ sed -e "s/lastchar/no_newline/g" >sed_4-svn14360_tmp.patch patch -p0 sed_4-svn14360_norename.patch ---------------------------------------------------------------------- landley - 03-01-06 07:49 ---------------------------------------------------------------------- Did you notice the change to get_line_from_file.c you reverted, and the change to sed.c (and for that matter, sort.c) that matched that? The way you were appending the length at an arbitrary aligment would confuse platforms that don't like unaligned access, plus you hard-wired in an assumption that a pointer is 4 bytes long (not true for 64 bit platforms). ---------------------------------------------------------------------- landley - 03-01-06 07:59 ---------------------------------------------------------------------- I applied the .4-norename patch, reverted the get_line_from_file part, and made the two line fix to have get_chunk_from_file() set len. It built fine, and then I ran the sed regression test suite. Failed spectacularly. It's appending an extra blank line to most things, and in at least one case a garbage character... ---------------------------------------------------------------------- robang74 - 03-02-06 00:50 ---------------------------------------------------------------------- > Did you notice the change to get_line_from_file.c you reverted, and the > change to sed.c (and for that matter, sort.c) that matched that? The way > you were appending the length at an arbitrary aligment would confuse > platforms that don't like unaligned access, plus you hard-wired in an > assumption that a pointer is 4 bytes long (not true for 64 bit platforms). not pointers but integer, anyway to fix 64 bit platform s/-6/-2-sizeof(int)/ every platforms ignore what is wroten after \0 so appending information after the null chars is ok for me. If am wrong it necessary to report lastchar back to the caller... so change the API get_line_from_file( ..., &lastchar); > I applied the .4-norename patch, reverted the get_line_from_file part, and > made the two line fix to have get_chunk_from_file() set len. It built > fine, and then I ran the sed regression test suite. > > Failed spectacularly. It's appending an extra blank line to most things, > and in at least one case a garbage character... I know. Because you have reverted the get_chunk_from_file() part. That is the reason because that part is needed. ---------------------------------------------------------------------- robang74 - 03-02-06 01:44 ---------------------------------------------------------------------- PATCH n.5 resolve the get_line_from_file.c reverted issue: it takes get_line_from_file.c as near as possible to the original one from svn 14360 ---------------------------------------------------------------------- robang74 - 03-03-06 00:44 ---------------------------------------------------------------------- PATCH n.6 as suggested by Rob does not need to modify get_line_from_file.c ---------------------------------------------------------------------- robang74 - 03-04-06 07:11 ---------------------------------------------------------------------- patch -p0 YES cd busybox/testsuite [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | tail -n1 FAIL: sed zero support [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | wc -l 7 <--- same 6 + 1 (my test s/ciao/miao/ on 000ciao\n000) cd ../.. patch -p0 /dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed embedded NUL FAIL: sed embedded NUL g FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] After applaying patch: [roberto at GEDX0327 testsuite]$ pwd; ./sed.tests 2>/dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1_sed/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] Manual test after applaying patch dd if=/dev/zero bs=1k count=1 >/tmp/test; echo ciao>>/tmp/test; dd if=/dev/zero bs=1k count=1 >>/tmp/test; cat /tmp/test | _install/bin/busybox sed -e "s/ciao/miao/g" >/tmp/test2; wc -l /tmp/test; wc -l /tmp/test2; hexdump /tmp/test; hexdump /tmp/test2 entrati 1+0 record usciti 1+0 record entrati 1+0 record usciti 1+0 record 1 /tmp/test 1 /tmp/test2 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 6963 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 696d 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 ---------------------------------------------------------------------- robang74 - 11-30-06 04:13 ---------------------------------------------------------------------- busybox-20061130_sed.patch: apply to 20061130 snapshot version ---------------------------------------------------------------------- vda - 12-02-06 09:59 ---------------------------------------------------------------------- #!/bin/sh { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | ./busybox sed -e "s/ciao/miao/g" | hexdump -vC { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | /usr/bin/sed -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n again >z2 >znull ./busybox sed "s/i/z/" z1 z2 znull | hexdump -vC /usr/bin/sed "s/i/z/" z1 z2 znull | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | ./busybox sed -e 's/woo/bang/' | hexdump -vC echo -ne "\0bang\0woo\0" | /usr/bin/sed -e 's/woo/bang/' | hexdump -vC unpatched busybox: 00000000 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |................| 00000010 6d 69 61 6f 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |miao............| 00000020 0a 0a 0a 0a |....| 00000024 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 0a 62 61 6e 67 0a 62 61 6e 67 |.bang.bang| 0000000a 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b Patched per revision 16754: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| 0000000b 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b ---------------------------------------------------------------------- vda - 12-02-06 12:14 ---------------------------------------------------------------------- Revision 16770 fixes that issue too. Tested with: #!/bin/sh function tst() { { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | $1 -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n thingy >z1 echo -ne "thingy\0" >z3 echo -ne "\0" >zzero >znull $1 "s/i/z/" z1 z2 z3 z1 znull z1 zzero zzero znull znull z1 | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | $1 -e 's/woo/bang/' | hexdump -vC } tst "./busybox sed" >x1 tst "/usr/bin/sed" >x2 diff -u x1 x2 >x.diff || { echo Different!; sleep 2; } Can I close ticket now? ---------------------------------------------------------------------- robang74 - 12-02-06 13:02 ---------------------------------------------------------------------- busybox-20061201_sed.patch [roberto at nbraf testsuite]$ ./sed.tests | egrep "^FAIL" FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) sed: unsupported command o FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label [roberto at nbraf busybox]$ echo -n thingy >z1 [roberto at nbraf busybox]$ echo -n again >z2 [roberto at nbraf busybox]$ ./busybox sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c [roberto at nbraf busybox]$ sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c Issue History Date Modified Username Field Change ====================================================================== 12-28-05 01:26 robang74 New Issue 12-28-05 01:26 robang74 Status new => assigned 12-28-05 01:26 robang74 Assigned To => BusyBox 12-28-05 01:26 robang74 Issue Monitored: robang74 12-28-05 03:18 robang74 Note Added: 0000821 12-28-05 03:18 robang74 File Added: sed.diff 01-01-06 22:47 landley Note Added: 0000835 01-02-06 01:25 robang74 Note Added: 0000836 01-02-06 01:25 robang74 File Added: sed_2.patch 01-08-06 11:34 landley Note Added: 0000873 01-09-06 03:29 robang74 Note Added: 0000886 01-09-06 03:30 robang74 Note Edited: 0000886 01-09-06 03:54 robang74 File Added: sed_3.patch 01-09-06 10:15 robang74 Note Added: 0000890 02-28-06 03:05 robang74 File Added: sed_4-svn14360.patch 02-28-06 03:05 robang74 File Added: sed_4-svn14360_norename.patch 02-28-06 03:09 robang74 Note Added: 0001148 03-01-06 07:49 landley Note Added: 0001154 03-01-06 07:59 landley Note Added: 0001155 03-02-06 00:33 robang74 Note Added: 0001156 03-02-06 00:50 robang74 Note Edited: 0001156 03-02-06 01:40 robang74 File Added: sed_5-svn14360.patch 03-02-06 01:41 robang74 File Added: sed_5-svn14360_norename.patch 03-02-06 01:43 robang74 Note Added: 0001157 03-02-06 01:44 robang74 Note Edited: 0001157 03-03-06 00:43 robang74 File Added: sed_6-svn14360.patch 03-03-06 00:43 robang74 File Added: sed_6-svn14360_norename.patch 03-03-06 00:44 robang74 Note Added: 0001160 03-04-06 07:11 robang74 Note Added: 0001161 03-04-06 07:12 robang74 File Added: busybox.14360_sed7_testsuite.patch 03-04-06 07:12 robang74 File Added: busybox.14360_sed7.patch 03-04-06 09:09 robang74 Note Added: 0001162 03-04-06 09:10 robang74 File Added: busybox.14360_sed8_testsuite.patch 03-04-06 09:10 robang74 File Added: busybox.14360_sed8.patch 03-05-06 18:57 robang74 File Added: busybox.14453_sed9.patch 03-05-06 18:58 robang74 Note Added: 0001164 03-14-06 07:39 robang74 File Added: busybox.14536_sed9.patch 11-30-06 01:05 robang74 Note Added: 0001841 11-30-06 01:07 robang74 File Added: busybox-1.2.2.1_sed.patch 11-30-06 04:12 robang74 File Added: busybox-20061130_sed.patch 11-30-06 04:13 robang74 Note Added: 0001843 12-02-06 09:59 vda Note Added: 0001853 12-02-06 10:00 vda File Added: sed_rev16754.patch 12-02-06 12:14 vda Note Added: 0001854 12-02-06 13:02 robang74 Note Added: 0001855 ====================================================================== From bugs at busybox.net Sat Dec 2 13:40:53 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Sat, 2 Dec 2006 13:40:53 -0800 Subject: [BusyBox 0000615]: sed convert 0x00 to 0x0a Message-ID: <53084a59f16e38cbfc1af6df4bbc248f@bugs.busybox.net> A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=615 ====================================================================== Reported By: robang74 Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 615 Category: Other Reproducibility: always Severity: major Priority: normal Status: assigned ====================================================================== Date Submitted: 12-28-2005 01:26 PST Last Modified: 12-02-2006 13:40 PST ====================================================================== Summary: sed convert 0x00 to 0x0a Description: bash-3.00# dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out bash-3.00# echo ciao>>/tmp/test bash-3.00# dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out bash-3.00# cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 bash-3.00# wc -l /tmp/test 1 /tmp/test bash-3.00# wc -l /tmp/test2 2048 /tmp/test2 ====================================================================== ---------------------------------------------------------------------- robang74 - 12-28-05 03:18 ---------------------------------------------------------------------- sed loose the last 0x00 if it exist: bash-3.00# ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:16 /tmp/test -rw-r--r-- 1 0 0 2052 Dec 28 11:16 /tmp/test2 AFTER PATCH: / # dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out / # echo ciao>>/tmp/test / # dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out / # cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 / # wc -l /tmp/test2 1 /tmp/test2 / # wc -l /tmp/test 1 /tmp/test / # ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test2 ---------------------------------------------------------------------- landley - 01-01-06 22:47 ---------------------------------------------------------------------- Hmmm... Tricky dealing with embedded nulls when C usually considers null to be an end of string indicator. Your fix causes problems in other contexts. It was outputting a newline because there are times when that is appropriate, and this would output a null then. Something like: echo -n thingy > one echo -n again > two sed "s/i/z/" one two > three The output should be "thzngy\nagazn" and I think it would be "thzngy\0agazn". Not that I've tested it just now. I'm trying to get 1.1.0 out this friday. Not sure I'll get to this before then... ---------------------------------------------------------------------- robang74 - 01-02-06 01:25 ---------------------------------------------------------------------- patch n.2 fix the problem of newline in more files [roberto at wsraf big]$ patch -p0 < sed_2.patch patching file busybox-1.01/editors/sed.c [roberto at wsraf busybox-1.01]$ make menuconfig && make [roberto at wsraf busybox-1.01]$ echo -n thingy > one [roberto at wsraf busybox-1.01]$ echo -n again > two [roberto at wsraf busybox-1.01]$ sed "s/i/z/" one two > three [roberto at wsraf busybox-1.01]$ ./busybox sed "s/i/z/" one two > four [roberto at wsraf busybox-1.01_sed2]$ cat three thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ cat four thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ ---------------------------------------------------------------------- landley - 01-08-06 11:34 ---------------------------------------------------------------------- I tried a slightly cleaned up version of the second patch and it caused two more busybox tests to fail. To see the specific failures, cd testsuite and "./runtest -v sed". I might get around to looking at it some more today, if so I'll apply the result and close out the bug. Rob ---------------------------------------------------------------------- robang74 - 01-09-06 03:30 ---------------------------------------------------------------------- AFTER PATCH n.3 sed fails the same testa as before [roberto at wsraf testsuite]$ ./runtest sed | grep FAIL FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) BUT passes the new one [roberto at wsraf testsuite]$ ./runtest sed | grep binary PASS: sed s onto a binary input (with zeros) ---------------------------------------------------------------------- robang74 - 01-09-06 10:15 ---------------------------------------------------------------------- svn 13198 with sed_3 patch applied, defconfig [roberto at wsraf busybox_sed3]$ size busybox text data bss dec hex filename 236511 2220 28484 267215 413cf busybox svn 13201 original, defconfig [roberto at wsraf busybox]$ size busybox text data bss dec hex filename 236527 2156 28548 267231 413df busybox Between 13198 and 13201 size grows 15 bytes more tham sed_3 patch, at least on my WS but I have gcc 4.0 (good or bad, so it is). ---------------------------------------------------------------------- robang74 - 02-28-06 03:09 ---------------------------------------------------------------------- Patch n.4 applies to svn 14360, it comes in two flowers: norename: which is not include a variable rename s/lastchar/no_newline/g and the patch itself with the variable rename for improved code readibility Switching from the patch to the norename one is pretty simple (automatic): svn co ... cp -af busybox busybox.orig cat sed_4-svn14360.patch | \ sed -e "s/lastchar/no_newline/g" >sed_4-svn14360_tmp.patch patch -p0 sed_4-svn14360_norename.patch ---------------------------------------------------------------------- landley - 03-01-06 07:49 ---------------------------------------------------------------------- Did you notice the change to get_line_from_file.c you reverted, and the change to sed.c (and for that matter, sort.c) that matched that? The way you were appending the length at an arbitrary aligment would confuse platforms that don't like unaligned access, plus you hard-wired in an assumption that a pointer is 4 bytes long (not true for 64 bit platforms). ---------------------------------------------------------------------- landley - 03-01-06 07:59 ---------------------------------------------------------------------- I applied the .4-norename patch, reverted the get_line_from_file part, and made the two line fix to have get_chunk_from_file() set len. It built fine, and then I ran the sed regression test suite. Failed spectacularly. It's appending an extra blank line to most things, and in at least one case a garbage character... ---------------------------------------------------------------------- robang74 - 03-02-06 00:50 ---------------------------------------------------------------------- > Did you notice the change to get_line_from_file.c you reverted, and the > change to sed.c (and for that matter, sort.c) that matched that? The way > you were appending the length at an arbitrary aligment would confuse > platforms that don't like unaligned access, plus you hard-wired in an > assumption that a pointer is 4 bytes long (not true for 64 bit platforms). not pointers but integer, anyway to fix 64 bit platform s/-6/-2-sizeof(int)/ every platforms ignore what is wroten after \0 so appending information after the null chars is ok for me. If am wrong it necessary to report lastchar back to the caller... so change the API get_line_from_file( ..., &lastchar); > I applied the .4-norename patch, reverted the get_line_from_file part, and > made the two line fix to have get_chunk_from_file() set len. It built > fine, and then I ran the sed regression test suite. > > Failed spectacularly. It's appending an extra blank line to most things, > and in at least one case a garbage character... I know. Because you have reverted the get_chunk_from_file() part. That is the reason because that part is needed. ---------------------------------------------------------------------- robang74 - 03-02-06 01:44 ---------------------------------------------------------------------- PATCH n.5 resolve the get_line_from_file.c reverted issue: it takes get_line_from_file.c as near as possible to the original one from svn 14360 ---------------------------------------------------------------------- robang74 - 03-03-06 00:44 ---------------------------------------------------------------------- PATCH n.6 as suggested by Rob does not need to modify get_line_from_file.c ---------------------------------------------------------------------- robang74 - 03-04-06 07:11 ---------------------------------------------------------------------- patch -p0 YES cd busybox/testsuite [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | tail -n1 FAIL: sed zero support [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | wc -l 7 <--- same 6 + 1 (my test s/ciao/miao/ on 000ciao\n000) cd ../.. patch -p0 /dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed embedded NUL FAIL: sed embedded NUL g FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] After applaying patch: [roberto at GEDX0327 testsuite]$ pwd; ./sed.tests 2>/dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1_sed/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] Manual test after applaying patch dd if=/dev/zero bs=1k count=1 >/tmp/test; echo ciao>>/tmp/test; dd if=/dev/zero bs=1k count=1 >>/tmp/test; cat /tmp/test | _install/bin/busybox sed -e "s/ciao/miao/g" >/tmp/test2; wc -l /tmp/test; wc -l /tmp/test2; hexdump /tmp/test; hexdump /tmp/test2 entrati 1+0 record usciti 1+0 record entrati 1+0 record usciti 1+0 record 1 /tmp/test 1 /tmp/test2 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 6963 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000400 696d 6f61 000a 0000 0000 0000 0000 0000 0000410 0000 0000 0000 0000 0000 0000 0000 0000 * 0000800 0000 0000 0000 0000805 ---------------------------------------------------------------------- robang74 - 11-30-06 04:13 ---------------------------------------------------------------------- busybox-20061130_sed.patch: apply to 20061130 snapshot version ---------------------------------------------------------------------- vda - 12-02-06 09:59 ---------------------------------------------------------------------- #!/bin/sh { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | ./busybox sed -e "s/ciao/miao/g" | hexdump -vC { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | /usr/bin/sed -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n again >z2 >znull ./busybox sed "s/i/z/" z1 z2 znull | hexdump -vC /usr/bin/sed "s/i/z/" z1 z2 znull | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | ./busybox sed -e 's/woo/bang/' | hexdump -vC echo -ne "\0bang\0woo\0" | /usr/bin/sed -e 's/woo/bang/' | hexdump -vC unpatched busybox: 00000000 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |................| 00000010 6d 69 61 6f 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a |miao............| 00000020 0a 0a 0a 0a |....| 00000024 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 0a 62 61 6e 67 0a 62 61 6e 67 |.bang.bang| 0000000a 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b Patched per revision 16754: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 6d 69 61 6f 0a 00 00 00 00 00 00 00 00 00 00 00 |miao............| 00000020 00 00 00 00 00 |.....| 00000025 ============ 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| 0000000b 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c ============ 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b ---------------------------------------------------------------------- vda - 12-02-06 12:14 ---------------------------------------------------------------------- Revision 16770 fixes that issue too. Tested with: #!/bin/sh function tst() { { dd if=/dev/zero bs=16 count=1 2>/dev/null echo ciao dd if=/dev/zero bs=16 count=1 2>/dev/null } | $1 -e "s/ciao/miao/g" | hexdump -vC echo ============ echo -n thingy >z1 echo -n thingy >z1 echo -ne "thingy\0" >z3 echo -ne "\0" >zzero >znull $1 "s/i/z/" z1 z2 z3 z1 znull z1 zzero zzero znull znull z1 | hexdump -vC echo ============ echo -ne "\0bang\0woo\0" | $1 -e 's/woo/bang/' | hexdump -vC } tst "./busybox sed" >x1 tst "/usr/bin/sed" >x2 diff -u x1 x2 >x.diff || { echo Different!; sleep 2; } Can I close ticket now? ---------------------------------------------------------------------- robang74 - 12-02-06 13:18 ---------------------------------------------------------------------- busybox-20061201_sed.patch [roberto at nbraf testsuite]$ ./sed.tests | egrep "^FAIL" FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) sed: unsupported command o FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label [roberto at nbraf busybox]$ echo -n thingy >z1 [roberto at nbraf busybox]$ echo -n again >z2 [roberto at nbraf busybox]$ ./busybox sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c [roberto at nbraf busybox]$ sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c [roberto at nbraf busybox-20061201_sed]$ echo -ne "\0bang\0woo\0" | ./busybox sed -e 's/woo/bang/' | hexdump -vC 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b [roberto at nbraf busybox-20061201_sed]$ echo -ne "\0bang\0woo\0" | sed -e 's/woo/bang/' | hexdump -vC 00000000 00 62 61 6e 67 00 62 61 6e 67 00 |.bang.bang.| 0000000b ---------------------------------------------------------------------- robang74 - 12-02-06 13:40 ---------------------------------------------------------------------- [roberto at nbraf busybox]$ tar xvjf busybox-20061201.tar.bz2 [roberto at nbraf busybox]$ cd busybox [roberto at nbraf busybox]$ patch -p1 < ../../sed_rev16754.patch [roberto at nbraf busybox]$ echo -n thingy >z1 [roberto at nbraf busybox]$ echo -n again >z2 [roberto at nbraf busybox]$ ./busybox sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 61 67 61 7a 6e |thzngyagazn| 0000000b [roberto at nbraf busybox]$ sed "s/i/z/" z1 z2 | hexdump -vC 00000000 74 68 7a 6e 67 79 0a 61 67 61 7a 6e |thzngy.agazn| 0000000c Issue History Date Modified Username Field Change ====================================================================== 12-28-05 01:26 robang74 New Issue 12-28-05 01:26 robang74 Status new => assigned 12-28-05 01:26 robang74 Assigned To => BusyBox 12-28-05 01:26 robang74 Issue Monitored: robang74 12-28-05 03:18 robang74 Note Added: 0000821 12-28-05 03:18 robang74 File Added: sed.diff 01-01-06 22:47 landley Note Added: 0000835 01-02-06 01:25 robang74 Note Added: 0000836 01-02-06 01:25 robang74 File Added: sed_2.patch 01-08-06 11:34 landley Note Added: 0000873 01-09-06 03:29 robang74 Note Added: 0000886 01-09-06 03:30 robang74 Note Edited: 0000886 01-09-06 03:54 robang74 File Added: sed_3.patch 01-09-06 10:15 robang74 Note Added: 0000890 02-28-06 03:05 robang74 File Added: sed_4-svn14360.patch 02-28-06 03:05 robang74 File Added: sed_4-svn14360_norename.patch 02-28-06 03:09 robang74 Note Added: 0001148 03-01-06 07:49 landley Note Added: 0001154 03-01-06 07:59 landley Note Added: 0001155 03-02-06 00:33 robang74 Note Added: 0001156 03-02-06 00:50 robang74 Note Edited: 0001156 03-02-06 01:40 robang74 File Added: sed_5-svn14360.patch 03-02-06 01:41 robang74 File Added: sed_5-svn14360_norename.patch 03-02-06 01:43 robang74 Note Added: 0001157 03-02-06 01:44 robang74 Note Edited: 0001157 03-03-06 00:43 robang74 File Added: sed_6-svn14360.patch 03-03-06 00:43 robang74 File Added: sed_6-svn14360_norename.patch 03-03-06 00:44 robang74 Note Added: 0001160 03-04-06 07:11 robang74 Note Added: 0001161 03-04-06 07:12 robang74 File Added: busybox.14360_sed7_testsuite.patch 03-04-06 07:12 robang74 File Added: busybox.14360_sed7.patch 03-04-06 09:09 robang74 Note Added: 0001162 03-04-06 09:10 robang74 File Added: busybox.14360_sed8_testsuite.patch 03-04-06 09:10 robang74 File Added: busybox.14360_sed8.patch 03-05-06 18:57 robang74 File Added: busybox.14453_sed9.patch 03-05-06 18:58 robang74 Note Added: 0001164 03-14-06 07:39 robang74 File Added: busybox.14536_sed9.patch 11-30-06 01:05 robang74 Note Added: 0001841 11-30-06 01:07 robang74 File Added: busybox-1.2.2.1_sed.patch 11-30-06 04:12 robang74 File Added: busybox-20061130_sed.patch 11-30-06 04:13 robang74 Note Added: 0001843 12-02-06 09:59 vda Note Added: 0001853 12-02-06 10:00 vda File Added: sed_rev16754.patch 12-02-06 12:14 vda Note Added: 0001854 12-02-06 13:02 robang74 Note Added: 0001855 12-02-06 13:02 robang74 File Added: busybox-20061201_sed.patch 12-02-06 13:18 robang74 Note Edited: 0001855 12-02-06 13:40 robang74 Note Added: 0001856 ====================================================================== From bugs at busybox.net Sat Dec 2 13:50:49 2006 From: bugs at busybox.net (bugs at busybox.net) Date: Sat, 2 Dec 2006 13:50:49 -0800 Subject: [BusyBox 0000615]: sed convert 0x00 to 0x0a Message-ID: <093ee7b1f7c5f4345d701943833d911a@bugs.busybox.net> A NOTE has been added to this issue. ====================================================================== http://busybox.net/bugs/view.php?id=615 ====================================================================== Reported By: robang74 Assigned To: BusyBox ====================================================================== Project: BusyBox Issue ID: 615 Category: Other Reproducibility: always Severity: major Priority: normal Status: assigned ====================================================================== Date Submitted: 12-28-2005 01:26 PST Last Modified: 12-02-2006 13:50 PST ====================================================================== Summary: sed convert 0x00 to 0x0a Description: bash-3.00# dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out bash-3.00# echo ciao>>/tmp/test bash-3.00# dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out bash-3.00# cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 bash-3.00# wc -l /tmp/test 1 /tmp/test bash-3.00# wc -l /tmp/test2 2048 /tmp/test2 ====================================================================== ---------------------------------------------------------------------- robang74 - 12-28-05 03:18 ---------------------------------------------------------------------- sed loose the last 0x00 if it exist: bash-3.00# ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:16 /tmp/test -rw-r--r-- 1 0 0 2052 Dec 28 11:16 /tmp/test2 AFTER PATCH: / # dd if=/dev/zero bs=1k count=1 >/tmp/test 1+0 records in 1+0 records out / # echo ciao>>/tmp/test / # dd if=/dev/zero bs=1k count=1 >>/tmp/test 1+0 records in 1+0 records out / # cat /tmp/test | sed -e "s/ciao/miao/g" >/tmp/test2 / # wc -l /tmp/test2 1 /tmp/test2 / # wc -l /tmp/test 1 /tmp/test / # ls -al /tmp/test* -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test -rw-r--r-- 1 0 0 2053 Dec 28 11:17 /tmp/test2 ---------------------------------------------------------------------- landley - 01-01-06 22:47 ---------------------------------------------------------------------- Hmmm... Tricky dealing with embedded nulls when C usually considers null to be an end of string indicator. Your fix causes problems in other contexts. It was outputting a newline because there are times when that is appropriate, and this would output a null then. Something like: echo -n thingy > one echo -n again > two sed "s/i/z/" one two > three The output should be "thzngy\nagazn" and I think it would be "thzngy\0agazn". Not that I've tested it just now. I'm trying to get 1.1.0 out this friday. Not sure I'll get to this before then... ---------------------------------------------------------------------- robang74 - 01-02-06 01:25 ---------------------------------------------------------------------- patch n.2 fix the problem of newline in more files [roberto at wsraf big]$ patch -p0 < sed_2.patch patching file busybox-1.01/editors/sed.c [roberto at wsraf busybox-1.01]$ make menuconfig && make [roberto at wsraf busybox-1.01]$ echo -n thingy > one [roberto at wsraf busybox-1.01]$ echo -n again > two [roberto at wsraf busybox-1.01]$ sed "s/i/z/" one two > three [roberto at wsraf busybox-1.01]$ ./busybox sed "s/i/z/" one two > four [roberto at wsraf busybox-1.01_sed2]$ cat three thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ cat four thzngy agazn[roberto at wsraf busybox-1.01_sed2]$ ---------------------------------------------------------------------- landley - 01-08-06 11:34 ---------------------------------------------------------------------- I tried a slightly cleaned up version of the second patch and it caused two more busybox tests to fail. To see the specific failures, cd testsuite and "./runtest -v sed". I might get around to looking at it some more today, if so I'll apply the result and close out the bug. Rob ---------------------------------------------------------------------- robang74 - 01-09-06 03:30 ---------------------------------------------------------------------- AFTER PATCH n.3 sed fails the same testa as before [roberto at wsraf testsuite]$ ./runtest sed | grep FAIL FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) BUT passes the new one [roberto at wsraf testsuite]$ ./runtest sed | grep binary PASS: sed s onto a binary input (with zeros) ---------------------------------------------------------------------- robang74 - 01-09-06 10:15 ---------------------------------------------------------------------- svn 13198 with sed_3 patch applied, defconfig [roberto at wsraf busybox_sed3]$ size busybox text data bss dec hex filename 236511 2220 28484 267215 413cf busybox svn 13201 original, defconfig [roberto at wsraf busybox]$ size busybox text data bss dec hex filename 236527 2156 28548 267231 413df busybox Between 13198 and 13201 size grows 15 bytes more tham sed_3 patch, at least on my WS but I have gcc 4.0 (good or bad, so it is). ---------------------------------------------------------------------- robang74 - 02-28-06 03:09 ---------------------------------------------------------------------- Patch n.4 applies to svn 14360, it comes in two flowers: norename: which is not include a variable rename s/lastchar/no_newline/g and the patch itself with the variable rename for improved code readibility Switching from the patch to the norename one is pretty simple (automatic): svn co ... cp -af busybox busybox.orig cat sed_4-svn14360.patch | \ sed -e "s/lastchar/no_newline/g" >sed_4-svn14360_tmp.patch patch -p0 sed_4-svn14360_norename.patch ---------------------------------------------------------------------- landley - 03-01-06 07:49 ---------------------------------------------------------------------- Did you notice the change to get_line_from_file.c you reverted, and the change to sed.c (and for that matter, sort.c) that matched that? The way you were appending the length at an arbitrary aligment would confuse platforms that don't like unaligned access, plus you hard-wired in an assumption that a pointer is 4 bytes long (not true for 64 bit platforms). ---------------------------------------------------------------------- landley - 03-01-06 07:59 ---------------------------------------------------------------------- I applied the .4-norename patch, reverted the get_line_from_file part, and made the two line fix to have get_chunk_from_file() set len. It built fine, and then I ran the sed regression test suite. Failed spectacularly. It's appending an extra blank line to most things, and in at least one case a garbage character... ---------------------------------------------------------------------- robang74 - 03-02-06 00:50 ---------------------------------------------------------------------- > Did you notice the change to get_line_from_file.c you reverted, and the > change to sed.c (and for that matter, sort.c) that matched that? The way > you were appending the length at an arbitrary aligment would confuse > platforms that don't like unaligned access, plus you hard-wired in an > assumption that a pointer is 4 bytes long (not true for 64 bit platforms). not pointers but integer, anyway to fix 64 bit platform s/-6/-2-sizeof(int)/ every platforms ignore what is wroten after \0 so appending information after the null chars is ok for me. If am wrong it necessary to report lastchar back to the caller... so change the API get_line_from_file( ..., &lastchar); > I applied the .4-norename patch, reverted the get_line_from_file part, and > made the two line fix to have get_chunk_from_file() set len. It built > fine, and then I ran the sed regression test suite. > > Failed spectacularly. It's appending an extra blank line to most things, > and in at least one case a garbage character... I know. Because you have reverted the get_chunk_from_file() part. That is the reason because that part is needed. ---------------------------------------------------------------------- robang74 - 03-02-06 01:44 ---------------------------------------------------------------------- PATCH n.5 resolve the get_line_from_file.c reverted issue: it takes get_line_from_file.c as near as possible to the original one from svn 14360 ---------------------------------------------------------------------- robang74 - 03-03-06 00:44 ---------------------------------------------------------------------- PATCH n.6 as suggested by Rob does not need to modify get_line_from_file.c ---------------------------------------------------------------------- robang74 - 03-04-06 07:11 ---------------------------------------------------------------------- patch -p0 YES cd busybox/testsuite [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | tail -n1 FAIL: sed zero support [roberto at nbraf testsuite]$ ./sed.tests | grep FAIL | wc -l 7 <--- same 6 + 1 (my test s/ciao/miao/ on 000ciao\n000) cd ../.. patch -p0 /dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed embedded NUL FAIL: sed embedded NUL g FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] After applaying patch: [roberto at GEDX0327 testsuite]$ pwd; ./sed.tests 2>/dev/null | grep FAIL /home/roberto/busybox/busybox-1.2.2.1_sed/testsuite FAIL: sed s//g (exhaustive) FAIL: sed n (flushes pattern space, terminates early) FAIL: sed N (doesn't flush pattern space when terminating) FAIL: sed NUL in command FAIL: sed append autoinserts newline FAIL: sed clusternewline FAIL: sed nonexistent label PASS: sed -i with no arg [GNUFAIL] Manual test after applaying patch dd if=/dev/zero bs=1k count=1 >/tmp/test; echo ciao>>/tmp/test; dd if=/dev/zero bs=1k count=1 >>/tmp/test; cat /tmp/test | _install/bin/busybox sed -e "s/ciao/miao/g" >/tmp/test2; wc -l /tmp/test; wc -l /tmp/test2; hexdump /tmp/test; hexdump /tmp/test2 entrati 1+0 record usciti 1+0 record entrati 1+0 record usciti 1+0 record 1 /tmp/test 1 /tmp/test2 0000000 0000 0000 0000 0000 0000 0000 0000 0